Draft terms of reference - United Nations Evaluation · PDF file2 Source: Project Document Rev. 2, and Advanced DGEF Database Information System (ADDIS) ... Thus, the area is considered

TOR: Terminal Evaluation for the UNEP-GEF Project “Demonstration of Sustainable Management of Coral Reef Resources in the Coastal Waters of Ninh Hai District, Ninh Thuan Province, Viet Nam”

Page 1 of 31

TERMS OF REFERENCE1

Terminal Evaluation of the UNEP GEF project

Demonstration of Sustainable Management of Coral Reef Resources in the Coastal Waters of Ninh Hai District, Ninh Thuan Province, Viet Nam (GEF ID. 3187)

I. PROJECT BACKGROUND AND OVERVIEW

1. Project Background and General Information

Table 1. Project Summary2

GEF project ID: 3187 IMIS number: GFL/2328-2731-4B39

Focal Area(s): Multi-focal Areas GEF OP #: 8

GEF Strategic Priority/Objective:

IW-1/SO-2/SP-1 GEF approval date:

5 June 2008

UNEP approval date: xxx Date of first

Disbursement: 3 August 2010

Actual start date: 6th

July 2010 Planned duration: 35 months

Intended completion date: 31

st December 2012 Actual or Expected

completion date: December 2014

Project Type: Medium-Sized Project (MSP)

GEF Allocation: US$ 406,900

Expected MSP/FSP Co-financing:

US $935,186 Total Cost:

US$ 1,342,086

Mid-term review/eval. (planned date):

December 2011 Terminal Evaluation:

August 2014

Mid-term review/eval. (actual date):

Jan 2013 (mid-term review meeting)

No. of revisions: 2

Date of last Steering Committee meeting:

30 January 2013 Date of last Revision:

11 April 2013

Disbursement as of 30 June 2013:

US$ 318,488 Actual expenditures entered in IMIS as of 30 June 2013:

US$ $ 279,091.24

Total co-financing realized as of 30 June 2013:

US$ 806,500.74 Leveraged financing: $ 0

1. This medium-sized project was developed in accordance with a regional agreement made by the Intergovernmental Steering Committee of the UNEP/GEF project entitled “Reversing Environmental Degradation Trends in the South China Sea and Gulf of Thailand” (SCS Project) launched in 2002. The SCS Project was developed and implemented on the basis of a Transboundary Diagnostic Analysis (TDA) and a draft Strategic Action Programme (SAP) for the South China Sea that was prepared between 1996 and 1999. The overall goals of the SCS Project are: “to create an environment at the regional level, in which collaboration and partnership in addressing environmental problems of the South China Sea, between all stakeholders, and at all levels is fostered and encouraged; and to enhance the capacity of the participating governments to integrate environmental consideration into national development planning.” The three components of the SCS Project are: i) Habitat Degradation and Loss, ii) Over Exploitation of Fisheries in the Gulf of Thailand, and iii) Land-based Pollution.

2. The component of the SCS Project - “Habitat Degradation and Loss” addresses four priority habitats, namely: mangroves, coral reefs, seagrass, and wetlands. These habitats were identified as being the highest priority in the region due to ongoing rates of degradation and their regional and global significance. The seven (7) participating countries in the SCS Project (i.e. Cambodia, China, Indonesia, Malaysia, Philippines, Thailand and Vietnam) agreed

1 TOR version of Sep-13 2 Source: Project Document Rev. 2, and Advanced DGEF Database Information System (ADDIS)


Page 2 of 31

that these should be the priority areas during initial SAP implementation. A total of 136 potential demonstration sites (26 mangrove sites, 43 coral reef sites, 26 seagrass sites, and 41 wetland sites) were thereafter nominated by the governments of these participating countries.

The demonstration site in Ninh Hai District, in the coastal zone of south-central Vietnam, received a high-ranking in the prioritisation analysis of potential demonstration sites under the framework of the SCS Project, because of its diverse fringing coral reefs and seagrass beds, which function as a nationally and regionally important spawning and nursery area for sea turtles, as well as the reversibility of current threats, and the level of direct stakeholder involvement.

2. Project Rationale

3. Viet Nam, with more than 3,200 km of mainland and island coasts facing the South China Sea and Gulf of Thailand, is an important reservoir of tropical marine biodiversity. Coral reefs, mangroves and seagrass are important habitats in coastal waters of Vietnam, playing important roles as reservoirs of biodiversity, for fisheries and tourist development in the coastal zone.

4. Despite the importance of coastal and marine environments in Vietnam, coastal habitats have not been properly managed. Escalating degradation of these habitats has far reaching environmental and socio-economic consequences. Like most countries in the Southeast Asian region, increasing human population threatens the ecological integrity of coastal and marine areas in Vietnam. Many coral reef areas have been degraded through over-exploitation, destructive fishing practices, coastal development, tourism, and pollution. This degradation and loss of coastal habitats in Vietnam has local, national, regional and global implications, and is therefore the focus of concerted efforts by inter-governmental, government and non-governmental organizations.

5. Approximately 1,070 ha of coral reefs and 40 ha of seagrass beds are found in Ninh Hai coastal waters. Coral reefs are distributed along 30 km of the district’s shoreline, and sustain rich biodiversity. The site is located near the centre of a major “upwelling” area on Vietnam’s coast. The cooler waters from upwelling reduce the threat of coral bleaching and associated biodiversity degradation caused by high sea temperatures during summer. The area was previously designated as Nui Chua Nature Reserve (1997-2003), mainly focusing on the conservation of dry forests and primates on land. In 2003, the status of the nature reserve was upgraded to the Nui Chua National Park; and its conservation interest was extended to marine components. Although the conservation of coastal and marine habitats was not the major interest of the nature reserve until 2003, previous conservation activities on land provided a certain degree of protection to the marine environment, as the landscape and water courses are mostly undisturbed, and future land-based development threats are restricted.

6. Ninh Hai is one of the few remaining sites in Vietnam where coastal and marine habitats are still in good condition. The marine waters comprise a diverse array of habitats such as coral reefs and seagrass beds and the beaches are one of the most important mainland nesting sites of global/regional endangered sea turtles such as Chelonia mydas, Eretmochelys imbricata and Caretta caretta. Some other species including the giant clams Tridacna squamosa and Tridacna crocea, and top shell Trochus niloticus are also listed at nationally and regionally endangered level. The reefs in the area are home for regional transient fish species such as mackerel Scomberomorus sp. and tuna Auxis sp. and are considered as spawning and/or feeding grounds of these migratory species which annually migrate from March to August. The reefs support a large number of new distribution records for Vietnam, some of which are considered to be globally uncommon or rare. These coral communities show considerable differences in species composition and community structure from those of further north and south, thus provide a high degree of complementarity in development of an integrated, representative national and regional Marine Protected Areas (MPA) network. The reefs also support coral populations of regional significance in terms of maintenance and replenishment, notably the species-rich Montipora assemblages. Thus, the area is considered as a potential source in transporting larvae to other areas in the South China Sea.

7. The coastal and marine habitats in Ninh Hai, however, face various anthropogenic threats such as: over-fishing; destructive and illegal harvesting of coastal and marine resources; collection of live corals and mining; and siltation. These threats had not been effectively addressed prior to the project due to the lack of an integrated management plan for the site, mainly as a result of limited financial and human resources. There was, therefore, a strong need for immediate interventions to address such threats and their root-causes.

8. As a demonstration project designed and implemented in accordance with the over-arching strategy of the UNEP/GEF SCS Project and the Strategic Action Programme, the benefits and lessons learned from the proposed project will be replicated and disseminated through the Framework for Regional Coordination, Dissemination of Experiences, and Personnel Exchange between Sites.


Page 3 of 31

9. Furthermore, it is intended to exchange lessons learned from Ninh Hai demonstration project with other SCS demonstration sites in different habitats in Vietnam such as a wetland and mangrove demonstration site in Red River Mouth. Sites such as Cu Lao Cham MPA and Con Dao National Park, which have practiced management with the support of DANIDA and UNDP/GEF and Viet Nam Government, would be the additional candidates for exchanging information and experience. Further, the replication opportunity would be enhanced among the Vietnam Marine Protected Area network mechanism.

3. Project Objectives and Components

10. The overarching goal of the Ninh Hai Coral Reef Project is “to reduce environmental stress on the trans-boundary water body of the South China Sea and Gulf of Thailand, through the further elaboration of the draft Strategic Action Programme and the implementation of a network of demonstration activities at sites of regional and global significance”.

11. According to the project document, its main objective was “to demonstrate measures to reduce the stress on regionally significant coral reef and seagrass habitats connected to the South China Sea through conserving critical marine biodiversity, preventing ecosystem degradation and promoting sustainable utilisation of marine and coastal resources of Ninh Hai waters”.

12. The project was expected to establish Ninh Hai as a marine protected area (MPA) for the effective management of 1,070 ha of coral reef including a total 40 ha of seagrass, through an integrated management plan that is based on cross-sectorial and participatory management and demarcation. The project would also address the implementation of policy and institutional reforms, as a contribution to the implementation of the inter-governmentally approved draft SAP for the South China Sea.

13. To achieve these objectives and expected results, the project comprised 3 components:

Component 1: improve area management through cross-sectorial and participatory approaches through establishment of institutional arrangements for cross-sectorial management, development and adoption of an Integrated Management Plan including zoning plan, establishment of demarcation and enforcement of relevant regulations, rehabilitation of some damaged environment, and establishment of monitoring system of coral reefs and seagrass beds habitats and annual monitoring;

Component 2: implement pilot projects on diverse sustainable income-generation options and develop the Sustainable Financial Strategy; and

Component 3: focus on building capacity and awareness at national, local, and community levels.

14. The benefits expected to be derived from the project include: ecosystem benefits i.e. protection of coral reef and associated habitats; transboundary benefits i.e. conservation of spawning and nursery grounds for fish and other marine animals of transboundary significance; and local benefits i.e. increased alternative opportunities for sustainable income-generation. The table below provides a summary of the project’s outputs and activities to achieve the expected outcomes.

Table 2. Project Components

Expected Outcome Outputs Activities

Component 1: Introducing cross-sectoral and participatory management approach

Outcome 1: Improved area management through cross-sectorial and participatory approaches

Output 1.1: Institutional arrangements for cross-sectoral management

Activity 1.1.1. Formal establishment of the Management Board (Project Steering Committee - PSC) and organization of annual meeting of the PSC

Activity 1.1.2. Formation of Management Advisory Group (AG) and organization of annual meeting

Activity 1.1.3. Mid-term and final meetings to evaluate implementation and success of the project of the Management Advisory Group (final meeting)

Output 1.2: Development of Integrated Management Plan with involvement of local communities

Activity 1.2.1: Surveys on biodiversity, fisheries production and socio-economic conditions

Activity 1.2.2. Preparation and adoption of Integrated Management Plan

Output 1.3: Demarcation, enforcement and surveillance at the project site with involvement of local

Activity 1.3.1. Management plan of sub demo site involving local community, private sectors

Activity 1.3.2. Zoning/Demarcation


Page 4 of 31

Expected Outcome Outputs Activities

communities Activity 1.3.3. Participatory preparation of regular

Activity 1.3.4. Enforcement & surveillance with involvement of local community (patrolling, petrol, allowance, boat…)

Output 1.4: Rehabilitation of some damaged environment implemented with community participation

Activity 1.4.1. Training professional skills for transplantation of coral reefs

Activity 1.4.2. Rehabilitation trials at a damaged reef

Output 1.5: Establishment of coral reefs and seagrass beds habitats monitoring system; annual monitoring and re-assessment of ecological change in the end of the project

Activity 1.5.1. Scientific and community-based monitoring of key indicators at selected reefs

Activity 1.5.2. Monitoring on Fisheries production

Activity 1.5.3. Monitoring on socio-economic (including water quality)

Activity 1.5.4. Re-assessment of ecological change of biodiversity and resources

Component 2: Pilot Projects on sustainable Income-Generation Options

Outcome 2: Pressure to coral reef ecosystems derived from unsustainable livelihoods is reduced

Output 2.1: Demonstration of pilot projects on diverse sustainable income-generation options

Activity 2.1 1. Development of sustainable fisheries & aquaculture

Activity 2.1.2. Development of local made souvenir products

Activity 2.1.3. Development of local based-ecotourism models

Activity 2.1.4. Establishment of mechanism to maintain of local based-ecotourism

Activity 2.1.5. Ninh Thuan Provincial budget support annually for income generation options

Activity 2.1.6. Developing alternative income generation for communities in buffer zone of NCNP

Activity 2.1.7. Alternative livelihood training for community

Component 3: Capacity building and Awareness Raising

Outcome 3: Knowledge and skills for the management of coral reef habitats are increased

Output 3.1: Preparing and dissemination of public awareness raising materials

Activity 3.1.1. General public awareness raising materials prepared and dissemination

Activity 3.1.2. Seminar for awareness enhancement

Output 3.2. Training workshop on sustainable use of coral reefs and seagrass beds resources targeting policy-makers, government officials and community representatives

Activity 3.2.1. Training workshop for policy-makers

Activity 3.2.2. Training workshop for government officials

Activity 3.2.3. Training workshop for community representatives

Output 3.3. Training workshops on professional skills on coral reefs and seagrass beds management

Activity 3.3.1. Training professional skill of diving for Demo-site staff

Activity 3.3.2. Training professional skills of monitoring of coral reefs and sea grass beds.

Activity 3.3.3. Skill-up training for staffs of project and partner organizations

Output 3.4: Exchange of information and experience with other relevant habitats management sites

Activity 3.4.1. Establishment and maintaining the website of the project

Activity 3.4.2. National Conference on MPAs/sustainable coastal management

Activity 3.4.4. Information exchange with UNEP/GEF/SCS projects

Source: Project Document

4. Executing Arrangements

15. The Institute of Oceanography (IO) was the project’s Executing Agency (EA), with an initial responsibility of project execution under the supervision and monitoring of the Implementing Agency - UNEP. The IO provided operational guidance for: project design and execution; progress and financial reports to UNEP; project management groups and stakeholders including national and local authorities; Project Steering Committee (PSC) and Advisory Group (AG); and the Demonstration Site Coordinating Unit (DSCU). The IO, as the Specialized Executive Agency of Coral Reefs (SEA-CR) sub-component of the SCS Project, was also responsible for ensuring close communication and collaboration between the Ninh Hai project and key elements of the SCS Project.


Page 5 of 31

16. The Project Steering Committee (PSC) was established at the provincial level in order to ensure the cross-sectoral institutional arrangement and participatory management. The PSC was comprised of the vice-chairman of Ninh Thuan Provincial People’s Committee (PPC), provisional members from Institute of Oceanography (IO), National Focal Point of Coral Reefs, and relevant local government agencies (Department of Science and Technology, Department of Natural Resources and Environment, Department of Trade and Tourism, Department of Rural Development and Agriculture, Department of Fisheries, Department of Police, Nui Chua National Park and Sub-department of Fisheries Protection).

17. Demonstration Site Co-ordinating Unit (DSCU) will be established at Ninh Thuan province and hosted by Ninh Thuan Department of Science and Technology (DOST). DSCU will be responsible for day-to-day project operation and reports to PSC. DSCU execute project activities, correspondences with relevant project stakeholders. It is anticipated that experts from both IO and Ninh Thuan DOST will join the DSCU either as project staff. It is beneficial for the project to bring both institutes to actively commit themselves for the project management. It is expected that DOST will support the implementation of the project not only by providing an office space and administrative supports, but also by strengthening collaboration and partnership among all stakeholders through its leadership.

18. An Advisory Group (AG) including several technical and scientific experts will be established and provided necessary scientific and technical advice to the DSCU.

19. Stakeholders identified during the project preparation include those from government sectors, private sectors, local communities, NGOs and research institutes as the following.

20. In Viet Nam, decision making and authorization with regard to the planning and management of socio-economic activities of district is at provincial level. Therefore, the Ninh Thuan Provincial People’s Committee (PPC), the highest decision-making body in the province, is one of the essential stakeholders for designing and implementing the proposed project. The Department of Science and Technology is authorized by PPC as a focal point of the proposed demonstration site project and is responsible for the management of environment and natural resources. In addition, other provincial departments will be involved with diverse responsibilities such as funding, policy making, project implementation, as well as enforcement of law and regulation. These include the Department of Fisheries (DOF), Sub-Department of Fisheries Protection, Department of Agriculture and Rural Development (DOARD), Department of Trade and Tourism (DOTT), Border Army and Marine Police.

21. The focal point at a district level is the Ninh Hai district People Committee with its Department of Economy and Department of Agriculture and Rural Development. The office of Fisheries and Agriculture Promotion, an auxiliary organizations at the district level, receives funds from the provincial government for providing training, transferring techniques, experimenting management models to improve community living standards, will be also involved.

22. The most important agency for natural resources and environmental management in the district is the Office of Nui Chua National Park with 32 staffs. The Park office manages 37,023 ha of land including 29,673 ha of natural forest and 7,350 ha of the buffer zone, and marine component is 7,352 ha. During the project preparation meetings at provincial and district levels, it was noted that the park office will play a significant role for the project implementation and sustainability of project outcomes beyond the life of the project. The park office receives funds from national and provincial governments for activities such as reforestation at the park and community development in the buffer zone.

23. Some private sectors such as Hoan Cau Resort and EMESCO Company will be involved in the development of eco-tourism model. With the anticipation of provincial development plan which will directed for tourism development, the numbers of tourism operators will be increased near future thus it is critical to encourage private sectors’ involvement into the project to obtain their supports for community involvement in the eco-tourism activities to maximize the benefits to the society and to minimize the negative environmental impacts in MPA.

24. Some communities whose livelihoods are rely on their surrounding environment and natural resources are the key stakeholders of the project as their socio-economic activities have great impacts on marine and coastal ecosystems. The degradation of those ecosystems will affect negatively to their lives from the consequence of unsustainable resources use. Representatives from local communities at target sites will be selected from different socio-economic groups as woman, farmer and youth associations for their involvement of the project implementation.

25. World Wildlife Fund (WWF)-Indochina have been playing significant roles in conservation and awareness enhancement under the framework of national program on marine conservation and sea turtle conservation. Other local and international NGOs will also be invited as appropriate, in particular, education, training and information and personnel exchanges. Especially, IUCN has executed successfully Hon Mun Marine Protected Areas under the support


Page 6 of 31

of WB/GEF. The Project Management Board will invite this organisation for technical advice and experience exchange during implementation.

26. Research and academic institutions will be key organization for collection and analysis of scientific information which will be the baseline for developing integrated management and zoning plan. Institute of Oceanography (IO), for example, has two decades of research experience in the proposed project area that their technical capacity will best be utilized for the project.

27. Vietnam Ministry of Natural Resources and Environment is the Focal Ministry of UNEP GEF/SCS project. The National Project Steering Committee of the SCS Project was established. Along with other concerned ministries, it has played key role for the preparation and development of this proposed project. The SCS’s National PSC has already decided to include the chairperson of the Project Steering Committee of Ninh Hai MSP project as a new member of National PSC. The Focal Ministry and National PSC assumes supervisory responsibility for Ninh Hai MSP project. Furthermore, MPA network operated by Former Ministry of Fisheries (recently Ministry of Agriculture and Rural Development) should be a key leading agency in developing policy, capacity building and technical advice for establishing and managing MPA at Ninh Hai.

5. Project Cost and Financing

28. This was a medium-sized project with a total cost of USD 935,186 of which USD 406,900 was GEF financing, and the balance was co-financing as described below. Cash co-financing was managed by the third party organisations in accordance with financial rules of the Viet Nam Government, supervised by the Executing Agency (Institute of Oceanography ) and reported periodically to UNEP.

Cost to the GEF Trust Fund: 406,900

Third party co-finance (cash): 417,591

Third party co-finance (in-Kind): 110,695

Total cost of the project: 935,186

29. The table below presents a summary of the proposed project cost by component.

Table 3. Planned Project Cost

Project Components Expected Outcomes Expected Outputs GEF Financing Co-financing Total($)

($) % ($) %

Improving area management through cross-sectorial and participatory approaches

Management framework established and the management capacity improved

Institutional arrangement for cross-sectorial management, integrated management plan, demarcation, enforcement and surveillance of regulations and legislation, rehabilitation of damaged habitats through the participatory approach, habitat monitoring system

218,565

43

292,842

57 511,407

Pilot project on alternative measures for sustainable income-generation for the local community

Unsustainable exploitation of natural resources reduced Sustainable livelihoods of local people developed

Alternative measures for sustainable income-generation ex. eco-tourism, aquaculture

87,200

30 201,824 70 289,024

Capacity Building and awareness raising

Knowledge and skills for the management of coral reef habitats increased

Awareness raising materials, training workshop on sustainable use and professional skills, exchange of information and experience

60,445

92 5,395

8 65,840

Project management 40,690 58 28,225 42 68,915

Total Project Costs 406,900 528,286 935,186

30. The table below shows a breakdown of the confirmed co-financing that was made towards the project:


Page 7 of 31

Table 4. Planned Sources of Co-financing

Name of co-financier (source) Classification Type Amount ($) %*

Nui Chua National Park Provincial government Cash 376,009 71%

Ninh Thuan Department of Science & Technology Provincial government Cash 19,430 4%

Ninh Thuan Sub-department of Fisheries Protection Provincial government Cash 22,152 4%

Various national and provincial agencies and local communities Government and NGOs In-kind 110,695 21%

Total Co-financing 528,286 100%

31. Project revisions were made to reflect changes in the project budget (such as unspent balances from the GEF trust fund expenditures), to extend the project closing date to allow completion of all project activities, and to rephrase the total unspent funds arising from the years 2011 and 2012 to subsequent years in the light of budgeted requirements. Revisions were also done to reallocate funds between budget lines according to the needs of the revised work plan.

6. Implementation Issues

32. There have been no other prior evaluations conducted on this project due to the fact that budgeted funds allocated for a mid-term evaluation were relatively low; there was however, a mid-term review meeting that was undertaken during one of the Project Steering Committee (PSC) meetings which subsequently resulted in a revision of the project design. Project revisions and extensions were generally experienced on account of administrative and funding issues e.g. re-phasing funds and reallocating funds between budget lines according to the needs of the revised work plan. The project was also extended to allow for the completion of all project activities – from an original duration of 48 months to 54 months. There was a change in the project’s UNEP Task Manager (TM) mid-way through the project; the former TM served the project in year 1 and 2 and was replaced by a new TM who presently heads the UNEP Coral Reef Unit based in Bangkok.

II. TERMS OF REFERENCE FOR THE EVALUATION

1. Objective and Scope of the Evaluation

33. In line with the UNEP Evaluation Policy3 and the UNEP Evaluation Manual

4, the Terminal Evaluation of the

Project “Demonstration of Sustainable Management of Coral Reef Resources in the Coastal Waters of Ninh Hai District, Ninh Thuan Province, Viet Nam” is undertaken at completion of the project to assess project performance (in terms of relevance, effectiveness and efficiency), and determine outcomes and impacts (actual and potential) stemming from the project, including their sustainability.

34. The evaluation has two primary purposes: (i) to provide evidence of results to meet accountability requirements, and (ii) to promote learning, feedback, and knowledge sharing through results and lessons learned among UNEP, governments, international and national executing agencies. Therefore, the evaluation will identify lessons of operational relevance for future project formulation and implementation.

35. The evaluation will focus on the following sets of key questions, based on the project’s intended outcome and results, which may be expanded by the consultant as deemed appropriate:

a. How successful has the project been in demonstrating methods of reducing environmental stress on regionally significant coral reef resources and seagrass habitats connected to the South China Sea through promoting sustainable utilisation of marine and coastal resources of Ninh Hai waters.?

b. Has the demonstration project sufficiently exhibited (i) ecosystem benefits: protection of coral reef and associated habitats; (ii) transboundary benefits: conservation of spawning and nursery grounds for fish and other marine animals of transboundary significance; and (iii) local benefits: increased alternative opportunities for sustainable income-generation?

3 http://www.unep.org/eou/StandardsPolicyandPractices/UNEPEvaluationPolicy/tabid/3050/language/en-US/Default.aspx

4 http://www.unep.org/eou/StandardsPolicyandPractices/UNEPEvaluationManual/tabid/2314/language/en-US/Default.aspx


Page 8 of 31

c. Has the project been effective in establishing Ninh Hai as a Marine Protected Area (MPA) and

implementing an integrated management plan that is based on cross-sectorial and participatory approaches?

d. What is the likelihood that the MPA established in Ninh Hai will contribute to national and regional biodiversity conservation goals, and act as a model for the development of a network of similar community-based MPAs in Viet Nam and elsewhere?

e. To what extent did the project succeed in demonstrating alternative livelihood options that are designed and implemented to improve the economic situation of the coastal communities while reducing pressure on the ecosystem of Ninh Hai waters?

f. How successful has the project been in strengthening local capacities in coastal and marine conservation, biodiversity and socio-economic monitoring, and management of coral reef habitats?

g. What were the most effective strategies used by the project and what were the key drivers and assumptions required to influence the achievement of project’s objectives and results?

2. Key Evaluation principles

36. Evaluation findings and judgements should be based on sound evidence and analysis, clearly documented in the evaluation report. Information will be triangulated (i.e. verified from different sources) to the extent possible, and when verification is not possible, the single source will be mentioned. Analysis leading to evaluative judgements should always be clearly spelled out.

37. The evaluation will assess the project with respect to a minimum set of evaluation criteria grouped into six categories: (1) Strategic Relevance; (2) Attainment of objectives and planned result, which comprises the assessment of outputs achieved, effectiveness and likelihood of impact; (3) Sustainability and replication; (4) Efficiency; (5) Factors and processes affecting project performance, including preparation and readiness, implementation and management, stakeholder participation and public awareness, country ownership and driven-ness, financial planning and management, UNEP supervision and backstopping, and project monitoring and evaluation; and (6) Complementarity with the UNEP strategies and programmes. The evaluation consultants can propose other evaluation criteria as deemed appropriate.

38. Ratings. All evaluation criteria will be rated on a six-point scale. However, complementarity of the project with the UNEP strategies and programmes, including other relevant partners’ strategies and programmes is not rated. Annex 2 provides guidance on how the different criteria should be rated and how ratings should be aggregated for the different evaluation categories.

39. In attempting to attribute any outcomes and impacts to the project, the evaluator should consider the difference between “what has happened with” and “what would have happened without” the project. This implies that there should be consideration of the baseline conditions and trends in relation to the intended project outcomes and impacts. This also means that there should be plausible evidence to attribute such outcomes and impacts to the actions of the project. Sometimes, adequate information on baseline conditions and trends is lacking. In such cases this should be clearly highlighted by the evaluator, along with any simplifying assumptions that were taken to enable the evaluator make informed judgements about project performance.

40. The “Why?” question should be at the front of the evaluator’s mind all through the evaluation exercise. This means that the consultant needs to go beyond the assessment of “what” the project performance was, and make a serious effort to provide a deeper understanding of “why” the performance was as it was, i.e. of processes affecting attainment of project results (criteria under category F – see below). This should provide the basis for the lessons that can be drawn from the project. In fact, the usefulness of the evaluation will be determined to a large extent by the capacity of the consultant to explain “why things happened” as they happened and are likely to evolve in this or that direction - which goes well beyond the mere review of “where things stand” at the time of evaluation.

3. Overall Approach and Methods

41. The Terminal Evaluation for this project will be conducted by an independent consultant under the overall responsibility and management of the UNEP Evaluation Office (EO) in Nairobi, and in consultation with the UNEP Coral Reef Unit (Bangkok, Thailand).

42. It will be an in-depth evaluation using a participatory approach whereby key stakeholders are kept informed and consulted throughout the evaluation process. Both quantitative and qualitative evaluation methods will be used


Page 9 of 31

to determine project achievements against the expected outputs, outcomes and impacts. Site visits will be an integral component of the evaluation exercise and will include travel to the project sites in Ninh Hai, Viet Nam. The consultant will liaise with the UNEP EO, the Task Manager and the Executing Agency on any logistic and/or methodological issues to properly conduct the evaluation as independently as possible. The EA shall provide the consultant with logistical support for in-country travel to the project sites.

43. The findings of the evaluation will be based on the following:

(a) A desk review of:

Relevant background documentation, inter alia UNEP Medium-term Strategy 2010-2013 and Programmes of Work, relevant policies and legislation, including project background information available online;

Project design documents and their revisions,

Project logical framework;

Annual work plans and budgets or equivalent;

Project reports such as progress implementation reports, financial reports, project reviews;

Minutes of relevant meetings, as well as relevant correspondences, workshop reports, ;

Documentation related to project outputs and outcomes, relevant materials published on the project, web-site(s), monitoring/assessment tools, public awareness materials, etc.; and

Any other relevant material on the project design and its implementation.

(b) Interviews5 with, but not limited to:

The head of the UNEP Coral Reef Unit, Freshwater and Marine Ecosystems Branch (Task Manager);

Relevant staff in the UNEP/ GEF Coordination Office (International Waters) and Division of Environmental Policy Implementation (DEPI);

Relevant staff in the Institute of Oceanography (Executing Agency);

Former SCS project staff;

Representatives from the project’s Advisory Group, Project Management Board, and Project Steering Committee;

Representatives from key project partners in government, private sector, local communities and research institutions involved in the project; and

The project’s Fund Management Officer.

44. The SCS project implementation phase6 stands to benefit from the findings of this evaluation; attention should

be drawn to the formulation of lessons based on good practice and successes that have the potential for wider application and use – describing the context from which they are derived and specifying the contexts in which they may be useful. Of particular interest, are lessons on the project’s catalytic role (nationally, regionally), stakeholder engagement and project ownership.

4. Evaluation criteria

A. Strategic relevance

45. The evaluation will assess, in retrospect, to what extent the project’s objectives and implementation strategies were consistent with national and regional environmental issues and needs; ii) the UNEP mandate and policies at the time of design and implementation; and iii) the relevant GEF focal areas, strategic priorities and operational programme(s).

46. It will also assess whether the project was aligned with UNEP’s Medium-term Strategy (MTS) 2010-2013 and Programmes of Work (PoW) 2010-2011 and 2012-2013. UNEP MTS 2010-2013 specifies desired results in six thematic

5 Face-to-face or through any other appropriate means of communication 6 Still in the conceptualization stage


Page 10 of 31

focal areas. The desired results are termed Expected Accomplishments. This project was located under the Ecosystems Management Sub-programme. The evaluation should comment on whether the project makes a tangible contribution to the Expected Accomplishments specified in the UNEP MTS 2010-13. The magnitude and extent of any contributions and the causal linkages should be fully described.

47. The evaluation will also assess whether the project objectives were realistic, given the time and budget allocated to the project, the baseline situation and the institutional context in which the project was to operate.

B. Achievement of Outputs

48. The evaluation will assess the project’s success in producing the programmed activities and outputs presented in Table 2 above, both in quantity and quality, as well as their usefulness and timeliness.

49. Briefly explain the reasons behind the success (or failure) of the project in achieving its different outputs and meeting expected quality standards, cross-referencing as needed to more detailed explanations provided under Section F (which covers the processes affecting attainment of project results).

C. Effectiveness: Attainment of Objectives and Planned Results

50. The evaluation will assess the extent to which the project’s objectives were effectively achieved or are expected to be achieved.

51. The evaluation will reconstruct the Theory of Change (ToC) of the project based on a review of project documentation and stakeholder interviews. The ToC of a project depicts the causal pathways from project outputs (goods and services delivered by the project) over outcomes (changes resulting from the use made by key stakeholders of project outputs) towards impact (changes in environmental benefits and living conditions). The ToC will also depict any intermediate changes required between project outcomes and impact, called “intermediate states”. The ToC further defines the external factors that influence change along the pathways, whether one result can lead to the next. These external factors are either drivers (when the project has a certain level of control) or assumptions (when the project has no control).

52. The assessment of effectiveness will be structured in three sub-sections:

(a) Evaluation of the achievement of outcomes as defined in the reconstructed ToC. These are the first-level outcomes expected to be achieved as an immediate result of project outputs. For this project, the

main questions will be to what extent the project has demonstrated measures to reduce the stress on regionally significant coral reef and seagrass habitats connected to the South China Sea through conserving critical marine biodiversity, preventing ecosystem degradation and promoting sustainable utilisation of marine and coastal resources of Ninh Hai waters.

(b) Assessment of the likelihood of impact using a Review of Outcomes to Impacts (ROtI) approach as summarized in Annex 6 of the TORs. The evaluation will appreciate to what extent the project has to

date contributed, and is likely in the future to further contribute to reducing environmental stress on transboundary water body of the South China Sea and Gulf of Thailand.

(c) Evaluation of the achievement of the formal project overall objective, overall purpose, goals and component outcomes using the project’s own results statements as presented in the Project Document (including revisions). This sub-section will refer back where applicable to the preceding sub-sections (a) and (b) above to avoid repetition in the report. To measure achievement, the evaluation will use as much as appropriate the indicators for achievement proposed in the Logical Framework (Logframe) of the project, adding other relevant indicators as appropriate. Briefly explain what factors affected the project’s success in achieving its objectives, cross-referencing as needed to more detailed explanations provided under Section F.

D. Sustainability and replication

53. Sustainability is understood as the probability of continued long-term project-derived results and impacts after the external project funding and assistance ends. The evaluation will identify and assess the key conditions or factors that are likely to undermine or contribute to the persistence of benefits. Some of these factors might be direct results of the project while others will include contextual circumstances or developments that are not under control of the project but that may condition sustainability of benefits. The evaluation should ascertain to what extent follow-up work has been initiated and how project results will be sustained and enhanced over time. The reconstructed ToC


Page 11 of 31

will assist in the evaluation of sustainability, as the drivers and assumptions required to achieve higher-level results are often similar to the factors affecting sustainability of these changes.

54. Four aspects of sustainability will be addressed:

(a) Socio-political sustainability. Are there any social or political factors that may influence positively or negatively the sustenance of project results and progress towards impacts? Is the level of ownership by the main stakeholders sufficient to allow for the project results to be sustained? Are there sufficient government and stakeholder awareness, interests, commitment and incentives to execute, enforce and pursue the programmes, plans, agreements, monitoring systems etc. prepared or agreed upon under the project?

(b) Financial resources. To what extent are the continuation of project results and the eventual impact of the project dependent on financial resources? What is the likelihood that adequate financial resources

7

will be or will become available to implement the programmes, plans, agreements, monitoring systems etc. prepared and agreed upon under the project? Are there any financial risks that may jeopardize sustenance of project results and onward progress towards impact? How financially sustainable are the community-based projects?

(c) Institutional framework. To what extent is the sustenance of the results and onward progress towards impact dependent on issues relating to institutional frameworks and governance? How robust are the institutional achievements such as governance structures and processes, policies, sub-regional agreements, legal and accountability frameworks etc. required to sustaining project results and to lead those to impact on human behaviour and environmental resources?

(d) Environmental sustainability. Are there any environmental factors, positive or negative, that can influence the future flow of project benefits? Are there any project outputs or higher level results that are likely to affect the environment, which, in turn, might affect sustainability of project benefits? Are there any foreseeable negative environmental impacts that may occur as the project results are being up-scaled?

55. Catalytic role and replication. The catalytic role of UNEP interventions is embodied in their approach of supporting the creation of an enabling environment and of investing in pilot activities which are innovative and showing how new approaches can work. UNEP also aims to support activities that upscale new approaches to a national, regional or global level, with a view to achieve sustainable global environmental benefits. The evaluation will assess the catalytic role played by this project, namely to what extent the project has:

(a) catalysed behavioural changes in terms of use and application by the relevant stakeholders, at a national (and possibly sub-regional) level, of: (i) cross-sectorial and participatory approaches to rehabilitation and sustainable management of coral reef ecosystems; (ii) increased local institutional capacity to implement existing legislation relevant to coral reef and sea grass habitats; (iii) community-based projects supporting local livelihoods through sustainable income generation activities; and (iv) built technical capacity in coral reef ecosystem management and monitoring -attained through training workshops.

(b) provided incentives (social, economic, market based, competencies etc.) to contribute to catalysing changes in stakeholder behaviour;

(c) contributed to institutional changes, for instance institutional uptake of project-demonstrated ICM and mangrove ecosystem management approaches;

(d) contributed to policy changes (on paper and in implementation of policy);

(e) contributed to sustained follow-on financing (catalytic financing) from governments, private sector, donors etc.;

(f) created opportunities for particular individuals or institutions (“champions”) to catalyse change (without which the project would not have achieved all of its results).

7 Those resources can be from multiple sources, such as the public and private sectors, income generating activities, other development projects etc.


Page 12 of 31

56. Replication, in the context of UNEP projects, is defined as lessons and experiences coming out of the project that are replicated (experiences are repeated and lessons applied in different geographic areas) or scaled up (experiences are repeated and lessons applied in the same geographic area but on a much larger scale and funded by other sources). The evaluation will assess the approach adopted by the project to promote replication effects and appreciate to what extent actual replication has already occurred or is likely to occur in the near future. What are the factors that may influence replication and scaling up of project experiences and lessons?

E. Efficiency

57. The evaluation will assess the cost-effectiveness and timeliness of project execution. It will describe any cost- or time-saving measures put in place in attempting to bring the project as far as possible in achieving its results within its programmed budget and (extended) time. It will also analyse how delays have affected project execution, costs and effectiveness. Wherever possible, costs and time over results ratios of the project will be compared with that of other similar interventions.

58. The evaluation will give special attention to efforts by the project teams to make use of/build upon pre-existing institutions, agreements and partnerships, data sources, synergies and complementarities with other similar initiatives, programmes and projects etc. to increase project efficiency.

F. Factors and processes affecting project performance

59. Preparation and readiness. This criterion focusses on the quality of project design and preparation. Were project stakeholders adequately identified? Were the project’s objectives and components clear, practicable and feasible within its timeframe? Were the capacities of the Executing Agency properly considered when the project was designed? Was the project document clear and realistic to enable effective and efficient implementation? Were the partnership arrangements properly identified and the roles and responsibilities negotiated prior to project implementation? Were counterpart resources (funding, staff, and facilities) and enabling legislation assured? Were adequate project management arrangements in place? Were lessons from other relevant projects properly incorporated in the project design? What factors influenced the quality-at-entry of the project design, choice of partners, allocation of financial resources etc.?

60. Project implementation and management. This includes an analysis of implementation approaches used by the project, its management framework, the project’s adaptation to changing conditions (adaptive management), the performance of the implementation arrangements and partnerships, relevance of changes in project design, and overall performance of project management. The evaluation will:

(a) Ascertain to what extent the project implementation mechanisms outlined in the project document have been followed and were effective in delivering project milestones, outputs and outcomes. Were pertinent adaptations made to the approaches originally proposed?

(b) Evaluate the effectiveness and efficiency of project management and how well the management was able to adapt to changes during the life of the project.

(c) Assess the role and performance of the project management and administration, including the project executing arrangements at all levels.

(d) Assess the extent to which project management responded to direction and guidance provided by the project’s Advisory Group and UNEP supervision recommendations.

(e) Identify administrative, operational, technical, institutional and/or political problems and constraints that influenced the effective implementation of the project, and how the project partners tried to overcome these problems. How did the relationship between the project management team and the collaborating partners develop?

61. Stakeholder8 participation and public awareness. The TOC analysis should assist the evaluator in identifying

the key stakeholders and their respective roles, capabilities and motivations in each step of the causal pathway from activities to achievement of outputs and outcomes to impact. The assessment will look at three related and often overlapping processes: (1) information dissemination to and between stakeholders, (2) consultation with and

8 The term stakeholder should be considered in the broadest sense, encompassing are the individuals, groups, institutions, or other bodies that have an interest or stake in the outcome of the project. The term also applies to those potentially adversely affected by the project.


Page 13 of 31

between stakeholders, and (3) active engagement of stakeholders in project decision making and activities. The evaluation will specifically assess:

(a) The approach(es) used to identify and engage stakeholders (within and outside UNEP) in project design and implementation. What were the strengths and weaknesses of these approaches with respect to the project’s objectives and the stakeholders’ motivations and capacities? What was the achieved degree and effectiveness of collaboration and interactions between the various project partners and stakeholders during design and implementation of the project?

(b) The degree and effectiveness of any public awareness activities that were undertaken during the course of implementation of the project; or that are built into the assessment methods so that public awareness can be raised at the time the assessments will be conducted;

(c) How the results of the project promote participation of stakeholders, including users, in decision making.

62. Country ownership and driven-ness. This is the relevance of the project to national development and environmental agendas, recipient country commitment, and regional and international agreements. The evaluation will assess:

(a) The level of country ownership. How well did the project stimulate country ownership of the management of regionally significant coral reef and seagrass habitats connected to the South China Sea, national or regional level policies to reduce environmental stress on the trans-boundary water body of the South China Sea and Gulf of Thailand, including commitments and partnerships for implementation?

(b) Specifically, the evaluator should assess how far the government has assumed responsibility for the project and provided adequate support to project execution, including the degree of cooperation received from the various government and private institutions involved in the project.

(c) Whether the demonstration project was effective in catalysing action to improve decision-making on conserving critical marine biodiversity, preventing ecosystem degradation and promoting sustainable utilisation of marine and coastal resources of Ninh Hai waters.

63. Financial planning and management. Evaluation of financial planning requires assessment of the quality and effectiveness of financial planning and control of financial resources throughout the project’s lifetime. The assessment will look at actual project costs by activities compared to budget (variances), financial management (including disbursement issues), and co-financing. The evaluation will, to the extent possible:

(a) Verify the application of proper standards (clarity, transparency, audit etc.) and timeliness of financial planning, management and reporting to ensure that sufficient and timely financial resources were available to the project and its partners;

(b) The evaluation will seek to provide a breakdown of final actual costs for the different project components (see template in Annex 3).

(c) Describe the resources the project has leveraged9 since inception and indicate how these resources are

contributing to the project’s ultimate objective.

(d) If applicable, analyse the effects on project performance of any irregularities in procurement, use of financial resources and human resource management, and the measures taken by UNEP to prevent such irregularities in the future. Appreciate whether the measures taken were adequate.

64. Supervision, guidance and technical backstopping. The purpose of supervision is to verify the quality and timeliness of project execution in terms of finances, administration and achievement of outputs, in order to identify and recommend ways to deal with problems which arise during project execution. Such problems may be related to project management but may also involve technical/institutional substantive issues in which UNEP has a major contribution to make. The evaluator should assess the effectiveness of supervision, administrative, financial and technical support provided by UNEP including:

9 Leveraged resources are additional resources—beyond those committed to the project itself at the time of approval—that are mobilized later as a direct result of the project. Leveraged resources can be financial or in-kind and they may be from other donors, NGO’s, foundations, governments, communities or the private sector.


Page 14 of 31

(a) The adequacy of project supervision plans, inputs and processes;

(b) The realism and candour of project reporting and the emphasis given to outcome monitoring (results-based project management);

(c) How well did the guidance and backstopping mechanisms work? What were the strengths in guidance and backstopping and what were the limiting factors?

65. Monitoring and evaluation. The evaluation will include an assessment of the quality, application and effectiveness of project monitoring and evaluation plans and tools, including an assessment of risk management based on the assumptions and risks identified in the project document. The evaluation will appreciate how information generated by the M&E system during project implementation was used to adapt and improve project execution, achievement of outcomes and ensuring sustainability. M&E is assessed on three levels:

(a) M&E Design. The evaluator should use the following questions to help assess the M&E design aspects:

Did the project have a sound M&E plan to monitor results and track progress towards achieving project objectives?

How well was the project logical framework (original and updates) designed as a planning and monitoring instrument?

SMART10

-ness of indicators: Are there specific indicators in the Logframe for each of the project objectives? Are the indicators measurable, attainable (realistic) and relevant to the objectives? Are the indicators time-bound?

Adequacy of baseline information: To what extent has baseline information on performance indicators been collected and presented in a clear manner? Was the methodology for the baseline data collection explicit and reliable? For instance, was there adequate baseline information on pre-existing accessible information on environmental status and trends in the sub-region? Was there sufficient information about the capacity of participating governments and collaborating partners etc. to determine their administrative and technical support needs?

Arrangements for monitoring: Have the responsibilities for M&E activities been clearly defined? Were the data sources and data collection instruments appropriate? Was the time frame for various M&E activities specified? Was the frequency of various monitoring activities specified and adequate? To what extent were project users involved in monitoring?

Arrangements for evaluation: Have specific targets been specified for project outputs? Has the desired level of achievement been specified for all indicators of objectives and outcomes? Were there adequate provisions binding project partners to fully collaborate in evaluations?

Budgeting and funding for M&E activities: Determine whether support for M&E was budgeted adequately and was funded in a timely fashion during implementation.

(b) M&E Plan Implementation. The evaluation will verify that:

the M&E system was operational and facilitated timely tracking of results and progress towards projects objectives throughout the project implementation period;

Progress reports & financial reports were complete and accurate;

the information provided by the M&E system was used during the project to improve project performance and to adapt to changing needs;

the M&E system had in place proper training, instruments and resources for parties responsible for M&E.

G. Complementarity with UNEP policies and strategies

66. Alignment with the Bali Strategic Plan (BSP)11

. The linkage between the project’s outcomes and achievements

to the objectives of the UNEP BSP should be briefly discussed (as applicable).

10 SMART stands for: specific, measurable, achievable, relevant and time-bound.


Page 15 of 31

67. Gender. Ascertain to what extent project design, implementation and monitoring have taken into consideration: (i) possible gender inequalities in access to and the control over resources; (ii) specific vulnerabilities of women and children to economic, social and environmental crises; and (iii) the role of women in mitigating or adapting to environmental changes and engaging in coral reef ecosystem and sea grass habitat management. Appreciate whether the intervention is likely to have any lasting differential impacts on gender equality and the relationship between women and the environment. To what extent do unresolved gender inequalities affect sustainability of project benefits?

68. South-South Cooperation. This is regarded as the exchange of resources, technology, and knowledge between developing countries and countries with economies in transition. Briefly describe any aspects of the project that could be considered as examples of South-South Cooperation. How did the project promote and benefit from the exchange of resources, technology, and knowledge?

5. The Consultant

69. For this evaluation, the Consultant should have extensive experience in the evaluation of environmental projects/programmes. S/he should have at least 5 years of work experience and practical knowledge in the general area of transboundary ecosystems management, and more specifically in marine and coastal biodiversity. Professional experience in marine protected areas, coral reefs, sea grass habitats, ecosystem management tools, related policies and implementation mechanisms is advantageous.

70. In addition to these professional skills, a university degree in environmental sciences, particularly marine and coastal zone management or relevant discipline, is required. Fluency in oral and written English is required

12.

71. Good interpersonal and communication skills are required to be able to convey information in a concise and understandable way. Candidates should also have an analytical mind, be organized and structured and have excellent oral and written communications skills.

72. The Consultant will coordinate data collection and analysis, and the preparation of the main report for the evaluation. S/He will ensure th1t all evaluation criteria and questions are adequately covered.

73. By undersigning the service contract with UNEP/UNON, the Consultant certifies that s/he has not been associated with the design and implementation of the project being evaluated in any way that may jeopardize their independence and impartiality towards project achievements and performance. In addition, they will not have any future interests (within six months after completion of the contract) with the project’s executing or implementing units.

6. Evaluation Deliverables and Review Procedures

74. The Consultant will, after an initial telephone briefing about the TOR with the UNEP Evaluation Office, and the UNEP Task Manager, conduct an initial desk review and submit an inception report to the UNEP Evaluation Office. This inception report shall contain a thorough review of the project context, project design quality, a draft reconstructed Theory of Change of the project, the evaluation framework and a tentative evaluation schedule (see report outline in Annex 1A of these TOR).

75. The review of design quality will cover the following aspects (see Annex 7 for the detailed project design assessment matrix):

Strategic relevance of the project;

Preparation and readiness;

Financial planning;

M&E design;

Complementarity with UNEP strategies and programmes;

Sustainability considerations and measures planned to promote replication and up-scaling.

11

http://www.unep.org/GC/GC23/documents/GC23-6-add-1.pdf 12 In the event that the Consultant is not conversant with the language(s) spoken in the project country, the services of a local interpreter may be sought during field missions.

http://www.unep.org/GC/GC23/documents/GC23-6-add-1.pdf


Page 16 of 31

76. The inception report will also present a draft, desk-based reconstructed Theory of Change of the project. It is vital to reconstruct the ToC before most of the data collection (review of reports, in-depth interviews, surveys etc.) is done, because the ToC will define which direct outcomes, drivers and assumptions of the project need to be assessed and measured – based on which indicators – to allow adequate data collection for the evaluation of project effectiveness, likelihood of impact and sustainability.

77. The evaluation framework will present in further detail the evaluation questions under each criterion with their respective indicators and data sources. The evaluation framework should summarize the information available from project documentation against each of the main evaluation parameters. Any gaps in information should be identified and methods for additional data collection, verification and analysis should be specified. The inception report will also present a tentative schedule for the overall evaluation process including, a tentative list of people/institutions to be interviewed, and a draft programme for site visits. The inception report will then be submitted for review and approval by the Evaluation Office before the implementation phase of the assignment.

78. The main evaluation report should be brief (no longer than 40 pages – excluding the executive summary and annexes), precise, and written in plain English. The report will follow the annotated Table of Contents outlined in Annex 1B. It must explain the purpose of the evaluation, exactly what was evaluated and the methods used (with their limitations). The report will present evidence-based and balanced findings, consequent conclusions, lessons and recommendations, which will be cross-referenced to each other. The report should be presented in a way that makes the information accessible and comprehensible. Any dissident views in response to evaluation findings will be appended in footnote or annex as appropriate. To avoid repetitions in the report, the author will use numbered paragraphs and make cross-references where possible.

79. Review of the draft evaluation report. The Consultant will submit what shall be referred to as a zero draft report

13 to the UNEP EO and revise this draft following the comments and suggestions made by the EO. Once a draft

of adequate quality has been accepted, the EO will share this as a first draft report with the UNEP Task Manager, who will alert the EO in case the report contains any factual errors. The Task manager will then forward the first draft report to the other project stakeholders, in particular the UNEP RSO, Regional Officers in ROLAC, representatives from the national stakeholder committees and other key project partners, for their review and comments. Stakeholders may provide feedback on any errors of fact and may highlight the significance of such errors in any conclusions. These comments would be expected within two weeks after the draft report has been shared. Any comments or responses to the draft report will be sent directly to the UNEP EO for collation. The EO will provide these review comments to the Consultant for consideration in preparing the final draft report.

80. The Consultant will submit the final draft report no later than 2 weeks after reception of stakeholder comments. The Consultant will prepare a response to comments, listing those comments not or only partially accepted that could therefore not or only partially be accommodated into the final report. The Consultant will explain why those comments have not or only partially been accepted, providing evidence/justification as required. This response to comments will be shared by the EO with the interested stakeholders to ensure full transparency.

81. Submission of the final Terminal Evaluation report. The final report shall be submitted by email to the Head of the Evaluation Office. The Evaluation Office will finalize the report and share it with the Executing Agency, and the interested Divisions and Sub-programme Coordinators in UNEP. The final evaluation report will also be published on the UNEP Evaluation Office web-site

14 and accessible by the public.

82. As per usual practice, the UNEP EO will prepare a quality assessment of the zero draft and final draft report, which is a tool for providing structured feedback to evaluation consultants. The quality of the report will be assessed and rated against the criteria specified in Annex 4.

83. The UNEP Evaluation Office will assess the ratings in the final evaluation report based on a careful review of the evidence collated by the Consultant and the internal consistency of the report. Where there are differences of opinion between the evaluator and UNEP Evaluation Office on project ratings, both viewpoints will be clearly presented in the final report. The UNEP Evaluation Office ratings will be considered the final ratings for the project.

13 The zero draft refers to the earliest report version that we receive from the consultant(s). In essence, this version should be as complete as possible and in line with the guidelines provided in the TOR and annotated table of contents presented in Annex 1. A good quality zero report will likely need minimal revision before it can be released as a First Draft report that is of an acceptable standard for circulation to stakeholders for their review and feedback. 14 www.unep.org/eou

http://www.unep.org/eou


Page 17 of 31

7. Logistical arrangements

84. This Terminal Evaluation will be undertaken by an independent evaluation consultant contracted by the UNEP Evaluation Office. The consultant will work under the overall responsibility of the UNEP Evaluation Office and will consult with the EO on any procedural and methodological matters related to the evaluation. It is, however, the consultant’s individual responsibility to arrange for their travel, visa, obtain documentary evidence, plan meetings with stakeholders, organize online surveys, and any other logistical matters related to the assignment. The Project management team will, where possible, provide logistical support (introductions, meetings, etc.) allowing the consultant to conduct the evaluation as efficiently and independently as possible.

8. Schedule of the Terminal Evaluation

85. Table 6 below presents the tentative schedule for the evaluation.

Table 6. Tentative evaluation schedule

Milestone Estimated dates

Consultants identified and contractual process initiated Early October, 2014

Consultant contracts signed Mid-late October 2014

Inception Report Early November 2014

Evaluation Mission to Ninh Hai, Viet Nam Mid-late November 2014

Zero Draft Report submitted to EO by consultant Early December 2014

First Draft Report submitted to EO by consultant Early-Mid December 2014

First Draft Report shared with UNEP Task Manager for onward circulation and review

December 2014/ January 2015

Submission of review comments to consultant January/February 2015

Final Report submitted to EO by consultant January/February 2015

End of assignment February 2015

9. Contractual arrangements

86. The Consultant will be hired under an individual Special Service Agreement (SSA). There are two options for payment: “lumpsum” or “fees only”.

87. Lumpsum: The fee is payable under the individual Special Service Agreement (SSA) of the evaluator and is inclusive of all expenses such as travel, per diem (Daily Subsistence Allowance - DSA) and incidental expenses which are estimated in advance. The Consultant will receive an initial payment covering such estimated expenses upon signature of the contract.

88. Fee only: The contract stipulates consultant fees only. The fee is payable under the individual SSA of the evaluator and is not inclusive of travel, accommodation and incidental expenses. Air tickets will be purchased by UNEP and 75% of the DSA for each authorised travel mission will be paid up front. Local in-country travel and communication costs will be reimbursed on the production of acceptable receipts. Terminal expenses and residual DSA entitlements (25%) will be paid after mission completion.

89. The payment schedule for both arrangements will be linked to the acceptance by the Evaluation Office of the key evaluation deliverables:

Final inception report: 20 percent of agreed total fee

First draft main evaluation report: 40 percent of agreed total fee

Final main evaluation report: 40 percent of agreed total fee

90. In case the Consultant is not able to provide the deliverables in accordance with these TOR, and in line with the expected quality standards by the UNEP Evaluation Office, payment may be withheld at the discretion of the Head of the Evaluation Office until the Consultant has improved the deliverables to meet UNEP’s quality standards.


Page 18 of 31

91. If the Consultant fails to submit a satisfactory final product to UNEP in a timely manner, i.e. before the end date of the contract, the Evaluation Office reserves the right to employ additional human resources to finalize the report, and to reduce the Consultant’s fees by an amount equal to the additional costs borne by the Evaluation Office to bring the report up to standard.


Page 19 of 31

Annex 1. Annotated Table of Contents of the main evaluation deliverables

A. INCEPTION REPORT

Section Notes Data Sources Max. number of pages

1. Introduction Brief introduction to the project and evaluation.

1

2. Project background Summarise the project context and rationale. How has the context of the project changed since project design?

Background information on context

3

3. Review of project design Summary of project design strengths and weaknesses. Complete the Template for assessment of the quality of project design (Annex 7 of these Terms of Reference).

Project document and revisions.

2 + completed matrix in annex of the inception report

4. Reconstructed Theory of Change

The Theory of Change should be reconstructed, based on project documentation. It should be presented with one or more diagrams and explained with a narrative.

Project document narrative, logical framework and budget tables. Other project related documents.

2 pages of narrative + diagram(s)

5. Evaluation framework The evaluation framework will contain:

Detailed evaluation questions (including new questions raised by review of project design and ToC analysis) and indicators

Data Sources It will be presented as a matrix, showing questions, indicators and data sources.

Review of all project documents.

5

6. Evaluation schedule Revised timeline for the overall evaluation (dates of travel and key evaluation milestones)

Tentative programme for the country visit

Discussion with project team on logistics.

2

7. Annexes A- Completed matrix of the overall quality of project design B- List of individuals and documents consulted for the inception report C- List of documents and individuals to be consulted during the main evaluation phase

B. MAIN REPORT

Project Identification Table An updated version of the Table 1 (page 1) of these TOR

Executive Summary Overview of the main findings, conclusions and recommendations of the evaluation. It should encapsulate the essence of the information contained in the report to facilitate dissemination and distillation of lessons. The main points for each evaluation parameter should be presented here (with a summary ratings table), as well as the most important lessons and recommendations. Maximum 4 pages.

I. Introduction A very brief introduction, mentioning the title of the evaluation and of the project, project duration, total cost.

Evaluation Objectives, approach and methodology, and limitations

II. The Project

i. Context Overview of the broader institutional and country/regional context in relation to the project’s objectives, project target areas/groups,

ii. Project objectives and components

iii. Implementation arrangements Including project partners

iv. Project financing Estimated costs and funding sources

v. Milestones in project design and implementation Including changes in design during implementation

I. Reconstructed Theory of Change of the project

III. Evaluation Findings

A. Strategic relevance The chapter on Evaluation Findings is organized according to the evaluation criteria presented in Section II.4 of the TORs and provides factual evidence relevant to the questions asked and sound analysis and interpretations of such evidence. This is the main substantive section of the report. Ratings should be provided at the end of the assessment of each evaluation criterion.

B. Achievement of outputs

C. Effectiveness: Attainment of project objectives and results

i. Direct outcomes from reconstructed TOC

ii. Likelihood of impact using RoTI and based on reconstructed TOC. Include the including the ROtI Results Scoresheet (See Table 3 in Annex 6)

iii. Achievement of project goal and planned objectives


Page 20 of 31

D. Sustainability and replication i Socio-political sustainability ii Financial sustainability iii Institutional framework iv Environmental sustainability v Catalytic role and replication

E. Efficiency

F. Factors affecting performance i Preparation and readiness ii Project implementation and management iii Stakeholder participation and public awareness iv Country ownership and driven-ness v Financial planning and management vi Supervision, guidance and technical backstopping vii Monitoring and evaluation

G. Complementarity with UNEP strategies and programmes

V. Conclusions and Recommendations

A. Conclusions This section should summarize the main conclusions of the evaluation, told in a logical sequence from cause to effect. It is suggested to start with the positive achievements and a short explanation why these could be achieved, and, then, to present the less successful aspects of the project with a short explanation why. The conclusions section should end with the overall assessment of the project. Avoid presenting an “executive summary”-style conclusions section. Conclusions should be cross-referenced to the main text of the report (using the paragraph numbering). The overall ratings table should be inserted here (see Annex 2).

B. Lessons Learned Lessons learned should be anchored in the conclusions of the evaluation. In fact, no lessons should appear which are not based upon an explicit finding of the evaluation. Lessons learned are rooted in real project experiences, i.e. based on good practices and successes which could be replicated or derived from problems encountered and mistakes made which ought to be avoided in the future. Lessons learned must have the potential for wider application and use. Lessons should briefly describe the context from which they are derived and specify the contexts in which they may be useful.

C. Recommendations All recommendations should be anchored in the conclusions of the report, with proper cross-referencing. Recommendations are actionable proposals on how to resolve concrete problems affecting the project or the sustainability of its results. They should be feasible to implement within the timeframe and resources available (including local capacities), specific in terms of who would do what and when, and set a measurable performance target. In some cases, it might be useful to propose options, and briefly analyse the pros and cons of each option. It is suggested, for each recommendation, to first briefly summarize the finding it is based upon with cross-reference to the section in the main report where the finding is elaborated in more detail. The recommendation is then stated after this summary of the finding.

Annexes These may include additional material deemed relevant by the evaluator but must include: 1. Response to stakeholder comments received but not (fully) accepted by the evaluators 2. Evaluation TORs (without annexes) 3. Evaluation Framework 4. Evaluation program, containing the names of locations visited and the names (or functions) and contacts (Email) of people met 5. Bibliography/List of documents reviwed 6. Summary co-finance information and a statement of project expenditure by activity (See Annex 3 of these TORs) 7. Matrix for the Assessment of the Quality of Project Design 8. Brief CVs of the consultants

Important note on report formatting

Reports should be submitted in Microsoft Word .doc or .docx format.

Use of Styles (Headings etc.), page numbering and numbered paragraphs is compulsory from the very first draft report submitted.

Examples of [UNEP/GEF] Terminal Evaluation Reports are available at www.unep.org/eou.

http://www.unep.org/eou


Page 21 of 31

Annex 2. Evaluation Ratings

The evaluation will provide individual ratings for the evaluation criteria described in section II.4 of these TORs. Most criteria will be rated on a six-point scale as follows: Highly Satisfactory (HS); Satisfactory (S); Moderately Satisfactory (MS); Moderately Unsatisfactory (MU); Unsatisfactory (U); Highly Unsatisfactory (HU). Sustainability is rated from Highly Likely (HL) down to Highly Unlikely (HU). In the conclusions section of the report, ratings will be presented together in a table, with a brief justification cross-referenced to the findings in the main body of the report.

Criterion Summary Assessment Rating

A. Strategic relevance HS HU

B. Achievement of outputs HS HU

C. Effectiveness: Attainment of project objectives and results HS HU

1. Achievement of direct outcomes HS HU

2. Likelihood of impact HS HU

3. Achievement of project goal and planned objectives HS HU

D. Sustainability and replication HL HU

1. Financial HL HU

2. Socio-political HL HU

3. Institutional framework HL HU

4. Environmental HL HU

5. Catalytic role and replication HS HU

E. Efficiency HS HU

F. Factors affecting project performance

1. Preparation and readiness HS HU

2. Project implementation and management HS HU

3. Stakeholders participation and public awareness HS HU

4. Country ownership and driven-ness HS HU

5. Financial planning and management HS HU

6. UNEP supervision and backstopping HS HU

7. Monitoring and evaluation HS HU

a. M&E Design HS HU

b. Budgeting and funding for M&E activities HS HU

c. M&E pPlan Implementation HS HU

Overall project rating HS HU

Rating for effectiveness: Attainment of project objectives and results. An aggregated rating will be provided for the achievement of direct outcomes as determined in the reconstructed Theory of Change of the project, the likelihood of impact and the achievement of the formal project goal and objectives. This aggregated rating is not a simple average of the separate ratings given to the evaluation sub-criteria, but an overall judgement of project effectiveness by the consultants. Ratings on sustainability. According to the GEF Office of Evaluation, all the dimensions of sustainability are deemed critical. Therefore, the overall rating for sustainability will be the lowest rating on the separate dimensions. Ratings of monitoring and evaluation. The M&E system will be rated on M&E design, M&E plan implementation, and budgeting and funding for M&E activities (the latter sub-criterion is covered in the main report under M&E design). M&E plan implementation will be considered critical for the overall assessment of the M&E system. Thus, the overall rating for M&E will not be higher than the rating on M&E plan implementation. Overall project rating. The overall project rating should consider parameters ‘A-E’ as being the most important with ‘C’ and ‘D’ in particular being very important.


Page 22 of 31

Annex 3. Project costs and co-financing tables

Project Costs

Component/sub-component Estimated cost at design Actual Cost Expenditure ratio (actual/planned)

Co-financing

Co-financing (Type/Source)

IA own Financing (mill US$)

Government

(mill US$)

Other*

(mill US$)

Total

(mill US$)

Total Disbursed (mill US$)

Planned Actual Planned Actual Planned Actual Planned Actual

Grants

Loans

Credits

Equity investments

In-kind support

Other (*) - -

Totals

* This refers to contributions mobilized for the project from other multilateral agencies, bilateral development cooperation agencies, NGOs, the private sector and beneficiaries.


Page 23 of 31

Annex 4. UNEP Evaluation Quality Assessment

Evaluation Title:

All UNEP evaluations are subject to a quality assessment by the Evaluation Office. The quality assessment is used as a tool for providing structured feedback to the evaluation consultants.

The quality of both the draft and final evaluation report is assessed and rated against the following criteria:

UNEP EO Comments Draft Report Rating

Final Report Rating

Substantive report quality criteria

A. Project context and project description: Does the report present an up-to-date description of the socio-economic, political, institutional and environmental context of the project, including the issues that the project is trying to address, their root causes and consequences on the environment and human well-being? Are any changes since the time of project design highlighted? Is all essential information about the project clearly presented in the report (objectives, target groups, institutional arrangements, budget, changes in design since approval etc.)?

Draft report:

Final report:

B. Strategic relevance: Does the report present a well-reasoned, complete and evidence-based assessment of strategic relevance of the intervention?

Draft report:

Final report:

C. Achievement of outputs: Does the report present a well-reasoned, complete and evidence-based assessment of outputs delivered by the intervention (including their quality)?

Draft report:

Final report:

D. C. Presentation of Theory of Change: Is the Theory of Change of the intervention clearly presented? Are causal pathways logical and complete (including drivers, assumptions and key actors)?

Draft report:

Final report:

E. D. Effectiveness - Attainment of project objectives and results: Does the report present a well-reasoned, complete and evidence-based assessment of the achievement of the relevant outcomes and project objectives?

Draft report:

Final report:

F. E. Sustainability and replication: Does the report present a well-reasoned and evidence-based assessment of sustainability of outcomes and replication / catalytic effects?

Draft report:

Final report:

G. F. Efficiency: Does the report present a well-reasoned, complete and evidence-based assessment of efficiency?

Draft report:

Final report:

H. G. Factors affecting project performance: Does the report present a well-reasoned, complete and evidence-based assessment of all factors affecting project performance? In particular, does the report include the actual project costs (total and per activity) and actual co-financing used; and an

Draft report:

Final report:


Page 24 of 31

assessment of the quality of the project M&E system and its use for project management?

I. H. Quality and utility of the recommendations: Are recommendations based on explicit evaluation findings? Do recommendations specify the actions necessary to correct existing conditions or improve operations (‘who?’ ‘what?’ ‘where?’ ‘when?)’. Can they be implemented?

Draft report:

Final report:

J. I. Quality and utility of the lessons: Are lessons based on explicit evaluation findings? Do they suggest prescriptive action? Do they specify in which contexts they are applicable?

Draft report:

Final report:

Report structure quality criteria

K. J. Structure and clarity of the report: Does the report structure follow EO guidelines? Are all requested Annexes included?

Draft report:

Final report:

L. K. Evaluation methods and information sources: Are evaluation methods and information sources clearly described? Are data collection methods, the triangulation / verification approach, details of stakeholder consultations provided? Are the limitations of evaluation methods and information sources described?

Draft report:

Final report:

M. L. Quality of writing: Was the report well written? (clear English language and grammar)

Draft report:

Final report:

N. M. Report formatting: Does the report follow EO guidelines using headings, numbered paragraphs etc.

Draft report:

Final report:

OVERALL REPORT QUALITY RATING


Page 25 of 31

Annex 5. Documentation list for the evaluation to be provided by the UNEP Task Manager

Project design documents including revisions

Supervision mission reports

Committee meeting documents, including agendas, meeting minutes, and any summary reports

Project progress reports, including financial reports submitted

Management memos and relevant correspondences related to project

Other documentation of supervision feedback on project outputs and processes (e.g. comments on draft progress reports, etc.).

Project revisions and extension documentation

Project Final Report

Etc.


Page 26 of 31

Annex 6. Introduction to Theory of Change / Impact pathways, the ROtI Method and the ROtI Results Score sheet

Terminal evaluations of projects are conducted at, or shortly after, project completion. At this stage it is normally possible to assess the achievement of the project’s outputs. However, the possibilities for evaluation of the project’s outcomes are often more limited and the feasibility of assessing project impacts at this time is usually severely constrained. Full impacts often accrue only after considerable time-lags, and it is common for there to be a lack of long-term baseline and monitoring information to aid their evaluation. Consequently, substantial resources are often needed to support the extensive primary field data collection required for assessing impact and there are concomitant practical difficulties because project resources are seldom available to support the assessment of such impacts when they have accrued – often several years after completion of activities and closure of the project.

Despite these difficulties, it is possible to enhance the scope and depth of information available from Terminal Evaluations on the achievement of results through rigorous review of project progress along the pathways from outcome to impact. Such reviews identify the sequence of conditions and factors deemed necessary for project outcomes to yield impact and assess the current status of and future prospects for results. In evaluation literature these relationships can be variously described as ‘Theories of Change’, Impact ‘Pathways’, ‘Results Chains’, ‘Intervention logic’, and ‘Causal Pathways’ (to name only some!).

Theory of Change (ToC) / impact pathways

Figure 1 shows a generic impact pathway which links the standard elements of project logical frameworks in a graphical representation of causal linkages. When specified with more detail, for example including the key users of outputs, the processes (the arrows) that lead to outcomes and with details of performance indicators, analysis of impact pathways can be invaluable as a tool for both project planning and evaluation.

Figure 1. A generic results chain, which can also be termed an ‘Impact Pathway’ or Theory of Change.

The pathways summarise casual relationships and help identify or clarify the assumptions in the intervention logic of the project. For example, in the Figure 2 below the eventual impact depends upon the behaviour of the farmers in using the new agricultural techniques they have learnt from the training. The project design for the intervention might be based on the upper pathway assuming that the farmers can now meet their needs from more efficient management of a given area therefore reducing the need for an expansion of cultivated area and ultimately reducing pressure on nearby forest habitat, whereas the evidence gathered in the evaluation may in some locations follow the lower of the two pathways; the improved farming methods offer the possibility for increased profits and create an incentive for farmers to cultivate more land resulting in clearance or degradation of the nearby forest habitat.

Figure 2. An impact pathway / TOC for a training intervention intended to aid forest conservation.

The GEF Evaluation Office has recently developed an approach to assess the likelihood of impact that builds on the concepts of Theory of Change / causal chains / impact pathways. The method is known as Review of Outcomes to Impacts (ROtI)15 and has three distinct stages:

a. Identifying the project’s intended impacts

b. Review of the project’s logical framework

c. Analysis and modelling of the project’s outcomes-impact pathways: reconstruction of the project’s Theory of Change

The identification of the projects intended impacts should be possible from the ‘objectives’ statements specified in the official project document. The second stage is to review the project’s logical framework to assess whether the design of the project is consistent with, and appropriate for, the delivery of the intended impact. The method requires verification of the causal logic between the different hierarchical levels of the logical

15

GEF Evaluation Office (2009). ROtI: Review of Outcomes to Impacts Practitioners Handbook.

http://www.gefweb.org/uploadedFiles/Evaluation_Office/OPS4/Roti%20Practitioners%20Handbook%2015%20June%202009.pdf

http://www.gefweb.org/uploadedFiles/Evaluation_Office/OPS4/Roti%20Practitioners%20Handbook%2015%20June%202009.pdf


Page 27 of 31

framework moving ‘backwards’ from impacts through outcomes to the outputs; the activities level is not formally considered in the ROtI method16. The aim of this stage is to develop an understanding of the causal logic of the project intervention and to identify the key ‘impact pathways’. In reality such processes are often complex: they might involve multiple actors and decision-processes and are subject to time-lags, meaning that project impact often accrues long after the completion of project activities.

The third stage involves analysis of the ‘impact pathways’ that link project outcomes to impacts. The pathways are analysed in terms of the ‘assumptions’ and ‘drivers’ that underpin the processes involved in the transformation of outputs to outcomes to impacts via intermediate states (see Figure 3). Project outcomes are the direct intended results stemming from the outputs, and they are likely to occur either towards the end of the project or in the short term following project completion. Intermediate states are the transitional conditions between the project’s direct outcomes and the intended impact. They are necessary changes expected to occur as a result of the project outcomes, that are expected, in turn, to result into impact. There may be more than one intermediate state between the immediate project outcome and the eventual impact.

Drivers are defined as the significant, external factors that if present are expected to contribute to the realization of the intended impacts and can be influenced by the project / project partners & stakeholders. Assumptions are the significant external factors that if present are expected to contribute to the realization of the intended impacts but are largely beyond the control of the project / project partners & stakeholders. The drivers and assumptions are considered when assessing the likelihood of impact, sustainability and replication potential of the project.

Since project logical frameworks do not often provide comprehensive information on the processes by which project outputs yield outcomes and eventually lead, via ‘intermediate states’ to impacts, the impact pathways need to be carefully examined and the following questions addressed:

o Are there other causal pathways that would stem from the use of project outputs by other potential user groups?

o Is (each) impact pathway complete? Are there any missing intermediate states between project outcomes and impacts?

o Have the key drivers and assumptions been identified for each ‘step’ in the impact pathway.

Figure 3. A schematic ‘impact pathway’ showing intermediate states, assumptions and impact drivers17 (adapted from GEF EO 2009)

In ideal circumstances, the Theory of Change of the project is reconstructed by means of a group exercise, involving key project stakeholders. The evaluators then facilitate a collective discussion to develop a visual model of the impact pathways using cards and arrows taped on a wall. The component elements (outputs, outcomes, intermediate states, drivers, assumptions, intended impacts etc.) of the impact pathways are written on individual cards and arranged and discussed as a group activity. Figure 4 below shows the suggested sequence of the group discussions needed to develop the ToC for the project.

Figure 4. Suggested sequencing of group discussions (from GEF EO 2009)

16

Evaluation of the efficiency and effectiveness in the use of resources to generate outputs is already a major focus within UNEP Terminal

Evaluations. 17

The GEF frequently uses the term “impact drivers” to indicate drivers needed for outcomes to lead to impact. However, in UNEP it is preferred to

use the more general term “drivers” because such external factors might also affect change processes occurring between outputs and outcomes.


Page 28 of 31

In practice, there is seldom an opportunity for the evaluator to organise such a group exercise during the inception phase of the evaluation. The reconstruction of the project’s Theory of Change can then be done in two stages. The evaluator first does a desk-based identification of the project’s impact pathways, specifying the drivers and assumptions, during the inception phase of the evaluation, and then, during the main evaluation phase, (s)he discusses this understanding of the project logic during group discussions or the individual interviews with key project stakeholders.

Once the Theory of Change for the project is reconstructed, the evaluator can assess the design of the project intervention and collate evidence that will inform judgments on the extent and effectiveness of implementation, through the evaluation process. Performance judgments are made always noting that project contexts can change and that adaptive management is required during project implementation.

The Review of Outcomes towards Impact (ROtI) method requires ratings for outcomes achieved by the project and the progress made towards the ‘intermediate states’ at the time of the evaluation. According to the GEF guidance on the method; “The rating system is intended to recognize project preparation and conceptualization that considers its own assumptions, and that seeks to remove barriers to future scaling up and out. Projects that are a part of a long-term process need not at all be “penalized” for not achieving impacts in the lifetime of the project: the system recognizes projects’ forward thinking to eventual impacts, even if those impacts are eventually achieved by other partners and stakeholders, albeit with achievements based on present day, present project building blocks.” For example, a project receiving an “AA” rating appears likely to deliver impacts, while for a project receiving a “DD” this would be very unlikely, due to low achievement in outcomes and the limited likelihood of achieving the intermediate states needed for eventual impact (see Table 1).

Table 1. Rating scale for outcomes and progress towards ‘intermediate states’

Outcome Rating Rating on progress toward Intermediate States

D: The project’s intended outcomes were not delivered D: No measures taken to move towards intermediate states.

C: The project’s intended outcomes were delivered, but were not designed to feed into a continuing process after project funding

C: The measures designed to move towards intermediate states have started, but have not produced results.

B: The project’s intended outcomes were delivered, and were designed to feed into a continuing process, but with no prior allocation of responsibilities after project funding

B: The measures designed to move towards intermediate states have started and have produced results, which give no indication that they can progress towards the intended long term impact.

A: The project’s intended outcomes were delivered, and were designed to feed into a continuing process, with specific allocation of responsibilities after project funding.

A: The measures designed to move towards intermediate states have started and have produced results, which clearly indicate that they can progress towards the intended long term impact.

Thus a project will end up with a two letter rating e.g. AB, CD, BB etc. In addition the rating is given a ‘+’ notation if there is evidence of impacts accruing within the life of the project. The possible rating permutations are then translated onto the usual six point rating scale used in all UNEP project evaluations in the following way.

Table 2. Shows how the ratings for ‘achievement of outcomes’ and ‘progress towards intermediate states translate to ratings for the ‘Overall likelihood of impact achievement’ on a six point scale.

Highly Likely

Likely Moderately Likely Moderately Unlikely Unlikely Highly Unlikely

AA AB BA CA BB+ CB+ DA+ DB+

BB CB DA DB AC+ BC+ AC BC CC+ DC+ CC DC AD+ BD+ AD BD CD+ DD+ CD DD

In addition, projects that achieve documented changes in environmental status during the project’s lifetime receive a positive impact rating, indicated by a “+”. The overall likelihood of achieving impacts is shown in Table 11 below (a + score above moves the double letter rating up one space in the 6-point scale).

The ROtI method provides a basis for comparisons across projects through application of a rating system that can indicate the expected impact. However it should be noted that whilst this will provide a relative scoring for all projects assessed, it does not imply that the results from projects can necessarily be aggregated. Nevertheless, since the approach yields greater clarity in the ‘results metrics’ for a project, opportunities where aggregation of project results might be possible can more readily be identified.

Table 3. ROtI Results Score sheet Template for presenting ‘Likelihood of Impact Achievement’

Results rating of project entitled:

Rat

ing

(D

– A

)

Rat

ing

(D –

A)

Rat

ing

(+)

Ove

rall

Outputs Outcomes Intermediate states Impact (GEBs)

1. 1. 1. 1.


Page 29 of 31

2. 2. 2. 2.

3. 3. 3. 3.

Rating justification: Rating justification: Rating justification:

Scoring Guidelines

The achievement of Outputs is largely assumed. Outputs are such concrete things as training courses held, numbers of persons trained, studies conducted, networks established, websites developed, and many others. Outputs reflect where and for what project funds were used. These were not rated: projects generally succeed in spending their funding.

Outcomes, on the other hand, are the first level of intended results stemming from the outputs. Not so much the number of persons trained; but how many persons who then demonstrated that they have gained the intended knowledge or skills. Not a study conducted; but one that could change the evolution or development of the project. Not so much a network of NGOs established; but that the network showed potential for functioning as intended. A sound outcome might be genuinely improved strategic planning in SLM stemming from workshops, training courses, and networking.

Examples

Funds were spent, outputs were produced, but nothing in terms of outcomes was achieved. People attended training courses but there is no evidence of increased capacity. A website was developed, but no one used it. (Score – D)

Outcomes achieved but are dead ends; no forward linkages to intermediate states in the future. People attended training courses, increased their capacities, but all left for other jobs shortly after; or were not given opportunities to apply their new skills. A website was developed and was used, but achieved little or nothing of what was intended because users had no resources or incentives to apply the tools and methods proposed on the website in their job. (Score – C)

Outcomes plus implicit linkages forward. Outcomes achieved and have implicit forward linkages to intermediate states and impacts. Collaboration as evidenced by meetings and decisions made among a loose network is documented that should lead to better planning. Improved capacity is in place and should lead to desired intermediate outcomes. Providing implicit linkages to intermediate states is probably the most common case when outcomes have been achieved. (Score - B)

Outcomes plus explicit linkages forward. Outcomes have definite and explicit forward linkages to intermediate states and impacts. An alternative energy project may result in solar panels installed that reduced reliance on local wood fuels, with the outcome quantified in terms of reduced C emissions. Explicit forward linkages are easy to recognize in being concrete, but are relatively uncommon. (Score A)

Intermediate states:

The intermediate states indicate achievements that lead to Global Environmental Benefits, especially if the potential for scaling up is established.

“Outcomes” scored C or D. If the outcomes above scored C or D, there is no need to continue forward to score intermediate states given that achievement of such is then not possible.

In spite of outcomes and implicit linkages, and follow-up actions, the project dead-ends. Although outcomes achieved have implicit forward linkages to intermediate states and impacts, the project dead-ends. Outcomes turn out to be insufficient to move the project towards intermediate states and to the eventual achievement of GEBs. Collaboration as evidenced by meetings and among participants in a network never progresses further. The implicit linkage based on follow-up never materializes. Although outcomes involve, for example, further participation and discussion, such actions do not take the project forward towards intended intermediate impacts. People have fun getting together and talking more, but nothing, based on the implicit forwards linkages, actually eventuates. (Score = D)

The measures designed to move towards intermediate states have started, but have not produced result, barriers and/or unmet assumptions may still exist. In spite of sound outputs and in spite of explicit forward linkages, there is limited possibility of intermediate state achievement due to barriers not removed or unmet assumptions. This may be the fate of several policy related, capacity building, and networking projects: people work together, but fail to develop a way forward towards concrete results, or fail to successfully address inherent barriers. The project may increase ground cover and or carbon stocks, may reduce grazing or GHG emissions; and may have project level recommendations regarding scaling up; but barrier removal or the addressing of fatal assumptions means that scaling up remains limited and unlikely to be achieved at larger scales. Barriers can be policy and institutional limitations; (mis-) assumptions may have to do with markets or public – private sector relationships. (Score = C)

Barriers and assumptions are successfully addressed. Intermediate state(s) planned or conceived have feasible direct and explicit forward linkages to impact achievement; barriers and assumptions are successfully addressed. The project achieves measurable intermediate impacts, and works to scale up and out, but falls well short of scaling up to global levels such that achievement of GEBs still lies in doubt. (Score = B)

Scaling up and out over time is possible. Measurable intermediate state impacts achieved, scaling up to global levels and the achievement of GEBs appears to be well in reach over time. (Score = A)

Impact: Actual changes in environmental status

“Intermediate states” scored B to A.

Measurable impacts achieved at a globally significant level within the project life-span. . (Score = ‘+’)


Page 30 of 31

Annex 7. Template for the assessment of the Quality of Project Design – UNEP Evaluation Office September 2011 Relevance Evaluation

Comments Prodoc reference

Are the intended results likely to contribute to UNEPs Expected Accomplishments and programmatic objectives?

Does the project form a coherent part of a UNEP-approved programme framework?

Is there complementarity with other UNEP projects, planned and ongoing, including those implemented under the GEF?

Are the project’s objectives and implementation strategies consistent with:

i) Sub-regional environmental issues and needs?

ii) the UNEP mandate and policies at the time of design and implementation?

iii) the relevant GEF focal areas, strategic priorities and operational programme(s)? (if appropriate)

iv) Stakeholder priorities and needs?

Overall rating for Relevance

Intended Results and Causality

Are the objectives realistic?

Are the causal pathways from project outputs [goods and services] through outcomes [changes in stakeholder behaviour] towards impacts clearly and convincingly described? Is there a clearly presented Theory of Change or intervention logic for the project?

Is the timeframe realistic? What is the likelihood that the anticipated project outcomes can be achieved within the stated duration of the project?

Are the activities designed within the project likely to produce their intended results?

Are activities appropriate to produce outputs?

Are activities appropriate to drive change along the intended causal pathway(s)?

Are impact drivers, assumptions and the roles and capacities of key actors and stakeholders clearly described for each key causal pathway?

Overall rating for Intended Results and causality

Efficiency

Are any cost- or time-saving measures proposed to bring the project to a successful conclusion within its programmed budget and timeframe?

Does the project intend to make use of / build upon pre-existing institutions, agreements and partnerships, data sources, synergies and complementarities with other initiatives, programmes and projects etc. to increase project efficiency?

Overall rating for Efficiency

Sustainability / Replication and Catalytic effects

Does the project design present a strategy / approach to sustaining outcomes / benefits?

Does the design identify the social or political factors that may influence positively or negatively the sustenance of project results and progress towards impacts? Does the design foresee sufficient activities to promote government and stakeholder awareness, interests, commitment and incentives to execute, enforce and pursue the programmes, plans, agreements, monitoring systems etc. prepared and agreed upon under the project?

If funding is required to sustain project outcomes and benefits, does the design propose adequate measures / mechanisms to secure this funding?

Are there any financial risks that may jeopardize sustenance of project results and onward progress towards impact?

Does the project design adequately describe the institutional frameworks, governance structures and processes, policies, sub-regional agreements, legal and accountability frameworks etc. required to sustain project results?

Does the project design identify environmental factors, positive or negative, that can influence the future flow of project benefits? Are there any project outputs or higher level results that are likely to affect the environment, which, in turn, might affect sustainability of project benefits?

Does the project design foresee adequate measures to catalyze behavioural changes in terms of use and application by the relevant stakeholders of (e.g.):

i) technologies and approaches show-cased by the demonstration projects;

ii) strategic programmes and plans developed

iii) assessment, monitoring and management systems established at a national and sub-regional level

Does the project design foresee adequate measures to contribute to institutional changes? [An important aspect of the catalytic role of the project is its contribution to institutional uptake or mainstreaming of project-piloted approaches in any regional or national demonstration projects]


Page 31 of 31

Does the project design foresee adequate measures to contribute to policy changes (on paper and in implementation of policy)?

Does the project design foresee adequate measures to contribute to sustain follow-on financing (catalytic financing) from Governments, the GEF or other donors?

Does the project design foresee adequate measures to create opportunities for particular individuals or institutions (“champions”) to catalyze change (without which the project would not achieve all of its results)?

Are the planned activities likely to generate the level of ownership by the main national and regional stakeholders necessary to allow for the project results to be sustained?

Overall rating for Sustainability / Replication and Catalytic effects

Risk identification and Social Safeguards

Are critical risks appropriately addressed?

Are assumptions properly specified as factors affecting achievement of project results that are beyond the control of the project?

Are potentially negative environmental, economic and social impacts of projects identified?

Overall rating for Risk identification and Social Safeguards

Governance and Supervision Arrangements

Is the project governance model comprehensive, clear and appropriate?

Are roles and responsibilities clearly defined?

Are supervision / oversight arrangements clear and appropriate?

Overall rating for Governance and Supervision Arrangements

Management, Execution and Partnership Arrangements

Have the capacities of partners been adequately assessed?

Are the execution arrangements clear?

Are the roles and responsibilities of internal and external partners properly specified?

Overall rating for Management, Execution and Partnership Arrangements

Financial Planning / budgeting

Are there any obvious deficiencies in the budgets / financial planning?

Is the resource utilization cost effective? Is the project viable in respect of resource mobilization potential?

Are the financial and administrative arrangements including flows of funds clearly described?

Overall rating for Financial Planning / budgeting

Monitoring

Does the logical framework:

capture the key elements of the Theory of Change for the project?

have ‘SMART’ indicators for outcomes and objectives?

have appropriate 'means of verification'?

identify assumptions in an adequate manner?

Are the milestones and performance indicators appropriate and sufficient to foster management towards outcomes and higher level objectives?

Is there baseline information in relation to key performance indicators?

Has the method for the baseline data collection been explained?

Has the desired level of achievement (targets) been specified for indicators of outcomes and are targets based on a reasoned estimate of baseline?

Has the time frame for monitoring activities been specified?

Are the organisational arrangements for project level progress monitoring clearly specified?

Has a budget been allocated for monitoring project progress in implementation against outputs and outcomes?

Overall, is the approach to monitoring progress and performance within the project adequate?

Overall rating for Monitoring

Evaluation

Is there an adequate plan for evaluation?

Has the time frame for evaluation activities been specified?

Is there an explicit budget provision for mid-term review and terminal evaluation?

Is the budget sufficient?

Overall rating for Evaluation

Documents

Draft terms of reference - United Nations Evaluation · PDF file2 Source: Project Document Rev. 2, and Advanced DGEF Database Information System (ADDIS) ... Thus, the area is considered