UN Development Account...United Nations Development Account DA Evaluation Framework, October 2019 3 Executive Summary Introduction 1. The United Nations (UN) Development Account (DA)

UN Development Account

Evaluation Framework

October 2019

United Nations Development Account

DA Evaluation Framework, October 2019 1

Table of Contents Abbreviations and Acronyms ........................................................................................................ 2

Executive Summary ...................................................................................................................... 3

1. Introduction ......................................................................................................................... 4

1.1 Review of the Evaluation Function of the UN Development Account .......................................... 4

2. Background of the DA and its Evaluation Function ................................................................. 5

2.1 Characteristics of the Development Account ............................................................................... 5

2.2 The DA Evaluation Function so far ................................................................................................ 6

2.3 Rationale for enhancing the DA Evaluation Function ................................................................... 6

2.4 Constraints identified in Evaluation of DA Projects ...................................................................... 7

3. Objectives of the DA Evaluation Function .............................................................................. 8

4. Components of the DA Evaluation Function ........................................................................... 9

5. Project and Programme Level Evaluations ........................................................................... 10

5.1 DA Project Level Evaluations ....................................................................................................... 10

5.2 DA Programme Level Evaluations ............................................................................................... 11

6. Roles and Responsibilities for DA Evaluation ....................................................................... 13

6.1 Overall DA Responsibilities ......................................................................................................... 13

6.2 Rolling DA Evaluation Work Plan ................................................................................................ 13

6.3 Roles and Responsibilities in DA Evaluations .............................................................................. 14

7. Generation and Use of Learning through Evaluation ............................................................ 15

7.1 Quality of Evaluations and their Results ..................................................................................... 15

7.2 Use of DA Evaluation Results ...................................................................................................... 16

7.3 Reporting to the General Assembly ............................................................................................ 17

ANNEX 1: Overview of DA Project and Programme Level Evaluations ................................................. 18

LIST OF TABLES

Table 1: DA Actors and their overall DA Responsibilities ..................................................................... 13

Table 2: DA Actors and their Responsibilities in DA Project and Programme Level Evaluations ......... 14



Abbreviations and Acronyms

CDPMO .......................... Capacity Development Programme Management Office (DESA)

DA .................................. Development Account

DESA .............................. Department of Economic and Social Affairs

ECA ................................ Economic Commission for Africa

ECE ................................ Economic Commission for Europe

ECLAC ............................ Economic Commission for Latin America and the Caribbean

ESCAP ............................ Economic and Social Commission for Asia and the Pacific

ESCWA ........................... Economic Social Commission for Western Asia

HABITAT ........................ United Nations Human Settlements Programme

IE ................................... Implementing Entity

MDG .............................. Millennium Development Goal

M&E .............................. Monitoring and Evaluation

OECD DAC ..................... Organisation for Economic Cooperation and Development, Development Assistance Committee

OIOS .............................. UN Office of Internal Oversight Services

PMT ............................... Programme Management Team

SDG ................................ Sustainable Development Goal

TOR ................................ Terms of Reference

UN ................................. United Nations

UNCTAD ........................ United Nations Conference on Trade and Development

UNEG ............................. United Nations Evaluation Group

UNEP ............................. United Nations Environment Programme

UNODC .......................... United Nations Office on Drugs and Crime

USD ................................ United States Dollar



Executive Summary

Introduction

1. The United Nations (UN) Development Account (DA) was established in 1997 by the UN General Assembly as a capacity development programme of the UN Secretariat. The DA supports the implementation of projects of five global UN Sectretariat entities and the five UN Regional Commissions, with the goal of enhancing capacities of developing countries in priority areas of the 2030 Agenda for Sustainable Development. Most of the DA projects focus on multiple countries, often across multiple regions, involving several of the ten Implementing Entities (IEs) and including partnerships with national level government, other UN agencies as well as other development partners.

The DA Evaluation Function so far

2. From the fifth tranche that started in 2006, all DA projects have been evaluated. The focus in evaluations has primarily been on accountability, with some attention to learning. A key constraint in the evaluation of projects has been the evaluation budget, which has been fairly limited and amongst other things reduced the opportunities for primary data gathering. Evaluation has depended much on surveys, with substantial non-response rates, limiting the opportunities for quantitative analysis. The quality of DA project evaluations reports has varied. Over time evaluation reports have generated similar recommendations, with lessons learned becoming repetitive. Sharing of results of evaluations across the ten IEs has been limited. There is overall the viewpoint that one can get more out of the Evaluation Function of the DA than presently realized.

3. The DA Evaluation Function should be enhanced to become more focused on learning, in particular learning about results and ways to achieve these. This is something which the present DA Evaluation Framework is meant to contribute towards.

The DA Evaluation Framework

4. The DA Evaluation Framework describes the objectives of the DA Evaluation Function as enabling organizational learning and accountability and informing the design and implementation of DA projects and tranches in support of the 2030 Agenda. Evaluations are to be conducted independently from programme/project management, making use of a results-based approach in the assessment of the achievements of the interventions. Human rights and gender need to be mainstreamed in all stages of the evaluation process, including a focus on the principle of ‘leaving no one behind’ of the 2030 Agenda. Evaluation is to be conducted in line with the norms and standards of the United Nations Evaluation Group (UNEG). The new DA Evaluation Function consists of project level evaluations, programme level evaluations, a rolling work plan and generation and use of learning through evaluation.

5. Project level evaluations remain an important component of the DA Evaluation Function. Rather than evaluating every project in a tranche, a sample of projects will be evaluated. Evaluating a more limited number of DA projects will enable a more rigorous process and a more in depth assessment and analysis of the issues concerned. This will, moreover, free up resources for more complex evaluations at the programme level.

6. Programme level evaluations consist of cluster evaluations, ex-post evaluations and meta evaluations and meta-analysis and could, moreover, include concurrent evaluations, i.e. the evaluation of a DA project as part of an evaluation of a wider programme to which its results framework contributes.

7. Selection of DA projects to be evaluated and of the programme level evaluations to be conducted will be coordinated by the DA Programme Management Team (DA PMT), through the



development of the DA evaluation work plan. Additional roles and responsibilities for project and programme level evaluations are described as part of the Framework.

8. Generation and use of learning through evaluation requires high quality evaluation reports in order to ensure confidence in use of their results. The DA Evaluation Framework provides a setup for the use of recommendations through the formulation of a management response by entities concerned and ways to follow-up on their implementation. Moreover, sharing of evaluation results and learning are important parts of the Framework, as is the contribution of the Evaluation Function to DA reporting to the General Assembly. In order to support the implementation of the DA Evaluation Framework, DA Project Evaluation Guidelines have been developed, concurrent to the present Framework, to enhance the quality of DA project level evaluations.

1. Introduction

1.1 Review of the Evaluation Function of the UN Development Account

9. The UN DA was established in 1997 by the General Assembly as a capacity development programme of the United Nations Secretariat. The DA supports the implementation of projects of five global UN Secretariat entities, i.e. the UN Conference on Trade and Development (UNCTAD), the UN Department of Economic and Social Affairs (UNDESA), the UN Environment Programme (UNEP), the UN Human Settlements Programme (UN Habitat) and the UN Office on Drugs and Crime (UNODC), and the five UN Regional Commissions, i.e. the Economic Commission for Africa (ECA), the Economic Commission for Europe (ECE), the Economic Commission for Latin America and the Caribbean (ECLAC), the Economic and Social Commission for Asia and the Pacific (ESCAP) and the Economic and Social Commission for Western Asia (ESCWA), with the goal of enhancing capacities of developing countries in priority areas of the 2030 Agenda for Sustainable Development. Evaluations have been conducted at the project level, which have been mandatory for all projects since the fifth tranche (2006-2009), with two percent of project budgets earmarked for project level evaluation, conducted towards the end of project implementation. Over 200 DA projects have been evaluated to date. Evaluations have so far been used primarily as a tool for accountability, with some attention to learning.

10. The present DA Evaluation Framework is aimed at enhancing the Evaluation Function of the DA, including its ability to contribute to learning. Enhancing the learning aspects of the Evaluation Function of the DA is an important means to inform the design of new projects with learnings from past experiences and to enhance the results that can be achieved by projects concerned. This will enable the ten IEs to enhance their support to Member States in implementing the 2030 Agenda for Sustainable Development. The importance of the Evaluation Function of the DA has been stressed by the Advisory Committee on Administrative and Budgetary Questions and the Fifth Committee of the General Assembly.

11. The DA Evaluation Framework addresses the limitations and constraints of the present DA evaluation system. It is aimed at enabling and facilitating learning through the conduct of evaluations and to enhance generation and use of knowledge on the ways in which to achieve high quality results in DA projects. Such learnings will be used in reporting on results of the DA to the General Assembly, and in the design of projects of future DA tranches. Enhanced learning and reporting on results is, moreover, intended to address the needs of recipient countries, DA implementing entities, and the DA PMT.

12. The present DA Evaluation Framework is focused on the DA Evaluation Function. This function does not exist in a vacuum, but is situated within the context of the Evaluation Functions of the DA IEs. The Framework complements the existing Evaluation Functions in the IEs and creates linkages between the evaluation operations of the DA and those in each of the ten IEs. The DA Evaluation Framework is meant to support an improvement of the DA Evaluation Function over time.



13. The Framework was prepared by a senior consultant. It was informed by a review commissioned by the DA Programme Manager and managed by the DA PMT. The review included a desk review, semi-structured interviews with a variety of DA stakeholders and an assessment of a selection of DA evaluation reports, across the ten IEs of the DA in the 8th and 9th tranches. The development of the Framework, moreover, was informed by the review undertaken by the DA Task Team in the period 2012-2013 as well as earlier reviews.

14. The DA Evaluation Framework was presented to and discussed with the DA Network, including DA Focal Points, DA Network Members and Evaluation Specialists of the IEs. The Framework is complemented by the DA Project Evaluation Guidelines, which detail how the Framework will be operationalized at the level of DA projects. Both the Framework and the Guidelines have, in principle, been endorsed by the DA Steering Committee. Information on the new DA evaluation system has also been shared with the General Assembly through the DA Progress Report.

15. While the Framework is expected to provide the overall direction over the longer term, the guidelines focus on how to put the Framework into practice at the project level, taking into consideration the specific characteristics of DA projects. The Framework is expected to remain valid for the medium term, while the DA Project Evaluation Guidelines are expected to be adapted more regularly. Details of the DA Evaluation Framework are presented below, focusing on background to the DA, objectives of the DA Evaluation Function and detailing the components of this function, i.e. project level evaluations, programme level evaluations, a rolling evaluation workplan and generation and use of learning through evaluation. An overview of project and programme level DA Evaluations is provided in Annex 1.

2. Background of the DA and its Evaluation Function

2.1 Characteristics of the Development Account

16. The DA provides the ten IE, which are mostly non-resident in beneficiary countries, with the ability to operationalize their vast knowledge and know-how and to deliver capacity development support on the ground to selected stakeholders. Much of the work includes policy level engagement, at times supplemented with the development of tools and guidelines in order to facilitate policy implementation. Through DA funded projects, the IEs are able to follow-up on normative and analytical work and inter-governmental processes through concrete projects at multi-country, sub-regional, regional and global levels. The Account aims to support opportunities for innovation, which after a successful project are meant to be picked up by the entities themselves, with funding from outside of the Account. All projects implemented through the DA are formulated based on mandates and needs assessments of IEs and requests from participating countries.

17. New DA tranches were until the 11th tranche initiated every two years. From 2020 onwards starting with the 12th tranche, new DA tranches are launched on an annual basis, with half the number of projects of the previous biennial tranches. Project implementation periods, however, remain at a four year period.

18. For the DA implementing entities, the Account is an important funding mechanism that enables them to conduct operational capacity development activities, as most of the entities’ own resources are limited in this respect. The financial aspect has become more important over time, with mobilization of donor funds increasingly challenging. Over the years, the DA has proved to be a funding stream that entities can rely upon. As such the Account is important in the delivery of the mandate of all ten entities.

19. Most of the DA projects focus on multiple countries, often across multiple regions, involving several of the IEs and including partnerships with national level government partners, other UN



agencies as well as other development partners. The DA has provided a means for the various IEs to engage with and partner with other DA entities, as well as UN agencies at the country level. Cooperation with other development partners has often resulted in leveraging of additional resources for the projects concerned. The DA supports this engagement with multiple partners to implement initiatives, and assists in the identification of partners in the stage of development of concept notes and in the process of project design. This cooperation across UN agencies is important as it is a requirement for the achievement of the Sustainable Development Goals (SDGs), which are interrelated, with each of the goals requiring the support of multiple agencies. Enhanced cooperation is, moreover, one of the components of the on-going UN organizational reform led by the Secretary-General.

2.2 The DA Evaluation Function so far

20. The DA requires all projects to be evaluated. Implementation of evaluations of DA projects varies across the DA entities and their evaluation sections. Apart from the requirement for a DA project evaluation to be conducted at the end of a project’s implementation, IEs have used their own guidelines and systems to implement DA project evaluations. Even so a template for an evaluation TOR and report were included in the guidelines developed by the DA Task Team in 2013, these weren’t officially published. In practice each of the IEs have made use of their own systems, formats and templates. This has resulted in differences across the evaluations of DA projects in terms of their management and outputs.

21. The evaluation sections of IEs have usually been the ones managing and supporting the DA project evaluations. This has meant that the evaluation sections have developed the TOR and provided quality assurance of the inception report and draft and final evaluation reports. In a few entities, DA evaluations have, for various reasons, been managed by the substantive section implementing the project. Nevertheless, in all cases, DA project evaluations have been conducted by independent external consultants.

22. Some of the IEs have had their own quality assurance mechanisms in place, through which they assessed the evaluation deliverables, making use of United Nations Evaluation Group (UNEG) guidelines and entity specific assessment tools.

2.3 Rationale for enhancing the DA Evaluation Function

23. A focus on learning is an important aspect for DA evaluations. This as IEs need to be able to show evidence in terms of results achieved, lessons learned and good practices identified. Also, the ambitious 2030 Agenda requires DA project evaluations to contribute to the generation of knowledge on ways to achieve sustainable development in an integrated way across the 17 SDGs. A focus on learning is important as it is of essence not only to show results, but also to be able to generate learning through DA support provided to inform future DA development initiatives.

24. Project level learning has been used by the IEs in terms of follow-up to the thematic areas of the projects concerned, while some lessons learned and good practices have had wider applications. However, the extent to which learnings have been identified in evaluation reports has varied. Identification of lessons learned and documentation of good practices, have often concerned experiences within project contexts, rather than learnings and practices that can be used beyond the context in which they were developed.

25. There is a view that more could be gained from the Evaluation Function of the DA than is presently realized. Focusing only at the project level might be useful for accountability purposes, but is not necessarily ,the best and single option when prioritizing learning through evaluation.

26. After conducting multiple rounds of project level evaluations in several DA tranches, the results are regarded to provide similar recommendations, with lessons learned becoming repetitive.



Though evaluation reports have the opportunity to enhance reporting on results achieved, this has often not been sufficiently realized in practice.

27. Sharing of learning and development of knowledge across agencies has so far been limited. Though most of the entities make their DA project evaluation reports available online, there is no system in place to share results of evaluations across the entities.

28. Follow-up to evaluation results has been varied, although the IEs have systems in place to develop management responses and follow up on recommendations aimed at their own organization.

29. For many of the IEs, the DA projects are part of larger programme level initiatives. Nevertheless the DA projects are evaluated separately, as stand-alone projects, in line with the requirements of the DA. Evaluation of DA projects as part of the larger programme of which they are part has occurred sporadically.

30. Limitations in terms of learning through project level evaluations have been recognized, with the understanding that there is a need to enhance learning through evaluation and to focus project level evaluations more on understanding of results achieved and the way in which these results were achieved. In addition, there is the realization that for the balance of the focus of evaluation to shift from primarily accountability towards enhanced inclusion of learning, the approaches used for evaluation need to be expanded. This in particular in terms of moving beyond the use of project level evaluations, to include programme level types of evaluation, through which learning can be generated. This includes grouping of selected projects from one tranche and conducting a cluster evaluation. Moreover, there is the option of ex-post evaluation, making an assessment sometime after project activities have been terminated and the opportunity to conduct meta-evaluation on quality aspects of evaluation reports as well as meta analysis on selected topics across a range of project evaluation reports. These approaches to evaluation would support the identification and documentation of good practices and lessons learned and synthesize results of evaluations, the learning which, in turn, could guide the conceptualization and design of new DA initiatives. These more complex forms of evaluation would contribute to shifting the balance between accountability and learning aspects of DA evaluation towards learning.

2.4 Constraints identified in Evaluation of DA Projects

31. A key constraint in the evaluation of DA projects concerns the project evaluation budget. Presently, the budget available for DA project evaluations is limited, with the DA project document guidelines indicating that at least 2% of the project budget should be allocated to the conduct of an external evaluation. For the average project, with a budget of 590,000 USD, this would mean an evaluation budget of 11,800 USD. The limitations concerned are clear in comparison with the minimum budget for project evaluation in some of the IEs, like UNEP, where project evaluation guidelines suggest a budget of at least 25,000 USD.

32. Budget limitations limit the opportunities for primary data gathering, as well as the possibility to hire an evaluation team of two, rather than a single consultant. Due to limitations in the evaluation budget, project evaluations depend much on desk reviews and online surveys, which usually have a low response rate, making them less useful for quantitative data analysis. Travel to countries is normally limited to participation of the evaluator in the last workshop of the project, during which s/he can interview part of the participants as well as participate in this project activity. When primary data gathering at country level is included in the evaluation, country visits are usually conducted to one selected country out of the number of countries participating in the project.

33. A system of annual project progress reporting is in place. Each DA project further has a results framework included in the design phase, however, with monitoring primarily activity oriented and with usually few data gathered against indicators of the project results framework. This leaves



evaluation at the end of the project period with limited monitoring data to review and use in order to inform the assessment of development results achieved.

34. The quality of past DA evaluation reports has varied. Often the different sections of the evaluation reports have varied in quality as well. Much of the quality of evaluation reports has been dependent on the quality of the evaluator, as well as if the evaluator has been a topic specialist or an evaluation specialist. Moreover, the capacities of the evaluation manager are important, in terms of providing the necessary support to the entire evaluation process.

35. Project evaluations needed to be conducted towards the end of the project cycle, with the evaluation report produced within three months after project closure. This has meant that usually primary data has needed to be gathered towards the end of the project, when not all activities have yet been implemented and in particular with higher level results unlikely to have been realized yet. As many of the projects have focused on workshops and training, a substantial period of time after the workshop or training activity is normally required, for the expected policy or policy implementation related changes to come to fruition. This has limited the changes that project evaluations, finalized three months after the official closure of the project, have in practice been able to assess.

36. IEs have taken the opportunity to learn from some of their DA project evaluations conducted, but have not necessarily been able to take note of the learnings from other IEs, with no system in place to share learnings from DA projects and their evaluations across the entities, DA Focal Points and DA Project Managers concerned. Sharing of learning would be of particular importance in terms of informing the design of new DA projects, which would benefit from the ability to refer to a database of lessons learned, good practices and recommendations made in previous project evaluations, easily accessible and searchable. There is the impression amongst many of the entities that evaluation results at present do not sufficiently inform the design of new DA projects.

3. Objectives of the DA Evaluation Function 37. In line with the UNEG Norms and Standard for Evaluation, the following definition of Evaluation is used:

An assessment, conducted as systematically and impartially as possible, of an activity, project, programme, strategy, policy, topic, theme, sector, operational area or institutional performance. It analyses the level of achievement of both expected and unexpected results by examining the results chain, processes, contextual factors and causality using appropriate criteria such as relevance, effectiveness, efficiency, impact and sustainability. An evaluation should provide credible, useful evidence-based information that enables the timely incorporation of its findings, recommendations and lessons into the decision-making processes of organizations and stakeholders.

(United Nations Evaluation Group, Norms and Standards for Evaluation, New York, 2016).

38. The objective of the DA Evaluation Function is:

To enable organizational learning and accountability and to inform the design and implementation of new DA projects and tranches in support of the 2030 Agenda.

39. This will be achieved through:

• Ensuring the timely conduct of DA evaluations at project and programme levels that are relevant and of use to the DA and its implementing entities and partners

• Enhancing the learning aspects of evaluations, in addition to aspects of accountability towards the General Assembly and stakeholders



• Identifying lessons learned and good practices through a rigorous approach, ensuring their applicability in contexts beyond the one in which they were identified

• Enriching reporting on results and ways in which results have been achieved, partially achieved, or not achieved, in order to support learning from successful as well as less successful projects

• Supporting the quality of evaluations and evaluation reports, in order to maximize the confidence with which the results of evaluations can be used

• Promoting and facilitating the use of evaluation results and learnings through the formulation of management responses and the implementation of these responses

• Enhancing cooperation and synergies across the ten DA entities, with other UN agencies and with development partners, through learning from evaluation

40. The Evaluation Function of the DA combines learning and accountability objectives and aims to instil evaluative thinking throughout the DA programme cycle, through the use of a variety of evaluation approaches, in correspondence with the special characteristics of the DA. Evaluations are implemented independently from project management and make use of a results-based approach, with the ability to demonstrate achievement of results. Human rights and gender are mainstreamed in all stages of the evaluation process, including a focus on the principle of ‘leaving no one behind’, of the 2030 Agenda. Evaluation is conducted in line with the norms and standards of the UNEG. This includes (i) the utility of evaluations, serving the needs of users; (ii) their credibility, making use of transparent processes, applying rigorous methods and involving relevant stakeholders; (iii) their independence, in terms of the evaluators as well as the organizational independence of the Evaluation Function; (iv) their impartiality and use of the highest ethical standards; and (v) adding to UN committed goals and targets and contributing in particular to the 2030 Agenda for Sustainable Development.1

41. The DA evaluation system complements the evaluation systems in place in the ten DA IEs by providing DA specific evaluation policies and guidelines.

4. Components of the DA Evaluation Function 42. In order to perform its roles, the Evaluation Function will include two types of evaluations: project and programme level evaluations. Project level evaluations will continue under the adjusted Evaluation Function, though will no longer be conducted for all DA projects in a tranche. Programme level evaluations will include more complex forms of evaluations, beyond the level and timeframe of DA projects. Moreover, the Evaluation Function includes sharing of evaluation results and related learnings across the DA entities and beyond, the development and implementation of management responses to recommendations concerned and the reporting on evaluations and their results to the General Assembly.

43. In order to enable the expansion of the DA Evaluation Function beyond the project level and to enable sufficient resources for evaluations at both project and programme levels and to have resources available to support learning based on evaluation results, four percent of the total DA budget in a tranche, up from two percent previously, will be earmarked for the Evaluation Function in line with UN and other donor standards.

44. Details on the Evaluation Function will be included in the rolling DA Evaluation Work Plan, prepared by the DA PMT and endorsed by the DA Steering Committee. This will include the selection of project and programme level evaluations as well as financial aspects of the DA Evaluation Function. The budget for project level evaluations will continue to be included in the budgets of the projects

1 United Nations Evaluation Group, Norms and Standards for Evaluation, New York, 2016/2017.



selected for evaluation and be used by the IE units that manage the evaluation. Programme level evaluations will be managed by the DA PMT in consultation with the IEs as relevant. In some instances, it may also be possible for an IE to take the lead in the management of a programme level evaluation.

5. Project and Programme Level Evaluations

5.1 DA Project Level Evaluations

45. Evaluation of projects will remain an important part of the DA Evaluation Function, with the dual objectives of accountability and learning. Focus of project evaluation includes the evaluation criteria of relevance, effectiveness, efficiency and sustainability. Moreover, the project’s contribution to the SDGs, partnerships, the mainstreaming of human rights and gender equality, as well as the use of innovation will also be addressed as more DA focused criteria. Rather than evaluating every project in a tranche, a sample of projects of each tranche will be evaluated. Evaluating half of DA projects per tranche will provide substantial coverage of DA projects implemented. At the same time, it will enable a more rigorous process and more in depth assessment and analysis of the issues concerned made possible through increased resources per project evaluation. It will, moreover, free up resources for more complex evaluations at the programme level.

46. Selection of projects for evaluation within a tranche will be based on a purposive sample, with random selection of a number of projects of each of the implementing agencies. Rather than taking a sample from all DA projects, a sample is taken from the projects of each IE in a tranche. The size of the sample for each of the IEs is in line with the relative number of their projects in the tranche. For those entities with a limited number of projects per tranche, there will be a minimum sampling of one project to be evaluated over two consecutive tranches. Projects with a budget of USD 1 million and more will by default be included among the projects selected for a project evaluation. The selection will be carried out in a transparent manner by the DA Steering Committee.

47. Project evaluations will be managed by the evaluation sections of the IE, independently from the DA project manager.

48. Project evaluations will be conducted by an independent evaluator or evaluation team, ideally gender balanced and if feasible, consisting of an evaluation as well as a substance matter specialist. Project evaluations should ideally include primary data gathering through a possible visit to one or more selected countries and face-to-face meetings and interviews with stakeholders concerned as well as possible participation of the evaluator in the last workshop of the project, in addition to desk review and additional phone or internet based interviews.

49. The timing of the project evaluation needs to take into consideration the start and end dates of the project and its implementation, avoiding to be timed at a too early stage, at which it would be difficult to assess the realization of the results of the project. Thus, a project evaluation should not be carried out before the project has finalized the implementation of all its activities. On the other hand, there is a need for the evaluation to be finalized within a time frame of three months after project closure. This time frame could be extended to up to six months depending on workload and complexity of the project and its evaluation. The timeframe needs to be agreed with the DA PMT and specified in the TOR of the evaluation.

50. The focus of project evaluations includes deliverables and output level changes achieved as a result of these, and outcome level changes to which the project contributed.



51. For all DA projects, project managers prepare a final project report. The DA guidelines for the final project report have been updated to reflect the present DA Evaluation Framework2. This report provides an overview of the project and its achievements, primarily from the perspective of the project manager and forms an important input to the project evaluation. A draft of the report needs to be available for the evaluator at the start of or during the evalution process.

52. For projects from the 12th tranche onwards that have not been randomly selected and thus not subject to evaluation, a review process for the final report drafted by the project manager, will be developed, in order to ensure that relevant details have been addressed to inform DA accountability and learning processes.

5.2 DA Programme Level Evaluations3

53. Programme level evaluations will be selected strategically, informed by the learning needs of the DA and its IEs. The decision on the types of programme level evaluations to undertake as part of the DA evaluation workplan, will vary over time and will be informed by the types of programme level evaluations performed in previous years, enabling an incremental learning process. The selection of cluster, ex-post, concurrent and meta-evaluations and analysis depends to a certain extent on the composition of projects in a tranche. It will be useful to include a short evaluability assessment for programme level evaluations, in order to ensure that conditions are in place to conduct the evaluation successfully. The selection of programme level evaluations will be included in the rolling DA Evaluation Work Plan that will be prepared by the DA PMT in consultation with IEs (see also section 6 for further information). Specifics on the types of programme level evaluations are provided below.

Types of programme level evaluation as part of the DA Evaluation Function

• Cluster Evaluation

A cluster evaluation consists of an evaluation of a cluster of DA projects, which are grouped based on specific criteria. Through clustering, the evaluation moves beyond a single project as the unit of analysis, enabling to address issues at the level of the cluster, including selected themes as well as management and strategic issues across a selection of DA projects.

Ways to cluster projects for evaluation include:

o Based on a specific topic, e.g. illicit financial flows, statistical capacity development, economic modelling, urban agenda

o Based on characteristics of projects, i.e. global projects, (sub-) regional projects, partnerships for project implementation

o DA projects focused on one or a selection of SDGs o Based on cross-cutting issues, such as gender equality and human rights o All or a selection of projects of one IE in a tranche or across a number of tranches

• Ex-post Evaluation

This type of evaluation is suited in particular for those DA projects that promise to generate results at the country level, but for which substantial additional time beyond the project period is required for their fruition. Ex-post evaluation would in particular focus on the evaluation criteria of effectiveness and sustainability of results, including those development changes to which the expected accomplishments of the project contributed. They need to be conducted at least one year after the closure of the project. Government and other partners need to be aware of the DA intent to conduct an ex-post evaluation, in order to be able to support such an initiative.

2 See the following link for information on the preparation of the final report: https://www.un.org/development/desa/da/static-guidance-public/ 3 Evaluations that go beyond the scope and timeframe of the DA project level evaluations.



• Concurrent Evaluation

A concurrent evaluation concerns the evaluation of a DA project at the time of the evaluation of the larger (sub-) programme of which it is a part. With many of the DA projects being part of larger programmes, this type of evaluation provides the opportunity to assess the contribution of the project to the wider programme and its results framework. There have been some examples of such concurrent DA evaluations in the past, although differences in the end dates of project and programme can prove challenging in terms of implementation.

• Meta-Evaluation and Meta-Analysis4

Meta-evaluation focuses on the quality aspects of a selection of project evaluations. It identifies and analyses these issues across a selection of DA evaluation reports in order to assess the quality of the evaluations concerned and to provide recommendations on ways to improve the quality of evaluation reports. Based on the selection of a wider set of DA evaluation reports, moreover, recommendations could include suggestions for improvement of the DA Evaluation Function. Meta-evaluation can be performed on all project evaluations conducted in a tranche or multiple tranches or on a sub-set of these. When meta-evaluation would be conducted at annual or biennial intervals, trends could be analysed.

Some of the entities have been conducting meta-evaluation on (parts of) their DA project evaluation reports and for some of the entities meta-evaluation on individual evaluation reports has been conducted by the UN Office of Internal Oversight Services (OIOS). DA meta-evaluation will build on possible existing meta-evaluations that have been conducted by OIOS or other entities .

Meta-analysis can be conducted on a set of DA evaluation reports based on a specific topic or in order to extract recommendations, good practices and lessons across a selection of DA project evaluation reports. Meta-analysis has been conducted by some of the IEs on selected topics with the inclusion of DA project evaluation reports. In terms of meta-analysis, the DA programme level evaluation will be complementary and avoid duplication of similar initiatives of IEs.

54. Cluster and meta-evaluations may concern joint evaluations5, i.e. an evaluation in which multiple IEs participate. This can, moreover, be the case in ex-post and concurrent evaluations of projects in which more than one IE participate. The joint character of these evaluations allows for mutual learning and capacity development and can enhance credibility of evaluation results. Experience from joint evaluations shows that similar levels of commitment of the various partners to the evaluation process, sufficient time spent in the design phase of the joint evaluation and clear management arrangements for its implementation are important factors for their success.6

4 The term meta-evaluation is used for the assessment of quality aspects of the conduct of evaluations and their outputs,

often through the assessment of a set of evaluation reports. The term meta-analysis is used for the assessment of the results of a set of evaluations, which can be organized along substantive thematic areas or can concern analysis of recommendations, lessons learned or good practices across a set of evaluation reports. 5 The Development Assistance Committee of the Organisation for Economic Cooperation and Development, (OECD DAC)

defines Joint Evaluation as an evaluation to which different donor agencies and/or partners participate. It is noted that there are various degrees of “jointness”, depending on the extent to which individual partners cooperate in the evaluation process, merge their evaluation resources and combine their evaluation reporting. OECD DAC, Glossary of Key Terms in Evaluation and Results Based Management, Paris, 2002. 6 OECD DAC, Joint Evaluations: Recent Experiences, Lessons Learned and Options for the Future, 2005. United Nations

Evaluation Group, Resource Pack on Joint Evaluations, December 2013.



6. Roles and Responsibilities for DA Evaluation

6.1 Overall DA Responsibilities

55. A variety of stakeholders are involved in the DA and its Evaluation Function. An overview of stakeholders and their overall responsibilities in the DA is provided in table 1 below.

Table 1: DA Actors and their overall DA Responsibilities

Actor Overall DA Responsibility

Steering Committee

Assists and advises the Programme Manager on strategic and policy related aspects of the DA; endorses the theme for each tranche; endorses the shortlist for projects to be funded in each tranche; chaired by the Assistant Secretary-General of UN DESA;

DA Programme Management Team

(DA PMT)

Supports DA management, including programming, monitoring, evaluation and reporting; liaises with DA focal points on all aspects of projects; provides the secretariat of DA Steering Committee, the DA Quality Assurance Group and the DA Network

DA Quality Assurance Group Provides quality assurance of concept notes and project documents, in line with relevant UN standards and DA guidelines; advises DA PMT on good practices in design of concept notes and project documents

Executive Heads of IEs Submit project proposals In line with IE programme level priorities; responsible for delivery of project results in compliance with DA requirements

DA Focal Points Coordinate all activities of the IE related to the DA; liaises with the Capacity Development Programme Management Office (CDPMO) of UN DESA; ensure internal quality assurance of aspects of DA projects

Evaluation sections of IEs Manage or support management of the evaluation of DA projects

DA Project Managers Manage all aspects of the project and its budget; liaise with the IE DA Focal Point; coordinate with other DA projects as well as other IEs to reach synergy and enhance results of the project

DA Network Members Sharing of operational knowledge, experiences and good practices of the IEs on all aspects of DA projects in order to enhance efficiency and effectiveness of the DA

6.2 Rolling DA Evaluation Work Plan

56. Project and programme level evaluations will be specified in the rolling DA evaluation work plan. The evaluation work plan consists of two parts. The first part is focused on project evaluations and the selection of projects to be evaluated in a tranche at the start of that tranche. The timing of the identification of project evaluations at the start of a tranche is meant to enhance their evaluability. This can enable ascertaining that the objective and outcomes are clearly defined and verifiable. Moreover, early identification of projects concerned can support the functionality of the project monitoring system and the ability of monitoring data to inform the evaluation process at the end of the project period. The second part of the evaluation plan concerns the programme level evaluations to be conducted. Decisions on the programme level evaluations and the projects that will be included in these evaluations will be taken on an annual basis in line with section 5.2.



6.3 Roles and Responsibilities in DA Evaluations

57. The various stakeholders to the DA play different roles in the DA Evaluation Function, including project and programme level evaluations (see table 2 below).

Table 2: DA Actors and their Responsibilities in DA Project and Programme Level Evaluations

Actor Role in Project Evaluations Role in Programme Level Evaluations

Steering

Committee

Endorses the DA Evaluation System; endorses the rolling DA Evaluation Work Plan; randomly selects half the projects to be evaluated in each tranche through purposive sampling; considers the conclusions, recommendations and lessons learned of programme level evaluation reports

DA

Programme

Management

Team

(DA PMT)

Prepares the rolling DA evaluation work plan, including a financial plan, comprising a list of projects to be evaluated, selected through purposive sampling in the Steering Committee and identification of the programme level evaluation(s) in consultation with the IEs as relevant

Issues the DA Project Evaluation Guidelines, makes results of the evaluations, including lessons learned and good practices, available to all IEs; supports the use of project level evaluation results in the design of new DA projects

Manages programme level evaluations in consultation with the IEs as relevant; manages the budget for programme level evaluations; provides oversight of the evaluation process, in terms of preparation of the TOR and the hiring of consultants to conduct the programme level evaluation, setting up of the programme level Evaluation Reference Group, review of the inception report and quality assurance on the draft evaluation report; reviews recommendations aimed at the DA overall and develops a management response to those agreed with; makes results of the evaluations, including lessons learned and good practices, available for internal use and summaries on the DA website for external use; supports the use of programme level evaluation results in the design of new DA projects

DA Focal

Points

Provide support to the DA project evaluations to be conducted by the IE in a tranche

Provide support to the DA programme level evaluation to be conducted, in case the IE is a party to the projects selected for the programme level evaluation concerned, participate in the Evaluation Reference Group of the programme level evaluation when the IE is a party to the projects selected for the evaluation concerned; review the recommendations focused on the IE and develops the management response

Evaluation

sections of

IEs

Manage the DA project evaluations to be conducted by the IE in a tranche and the related budget (which is part of the project budget), including development of TOR, hiring of independent consultant(s), management of the inception phase and review of the inception report; manage the data gathering phase in close coordination with the DA Project Manager; review draft and final evaluation reports; manage the quality assurance process; responsible for the overall coordination of the management response process; monitor the management response by the substantive section concerned

Provide support to the DA programme level evaluation when the IE is a party to the projects selected for the evaluation concerned; may, in some instances, take the lead in the management of the programme level evaluation; participates in the Evaluation Reference Group of the programme level evaluation as relevant and provide inputs to the quality assurance process of the deliverables of the evaluation; monitor the management response by the substantive section concerned

DA Project

Managers

Support the DA project evaluations to be conducted; provide secondary resources including monitoring and reporting details and the final project report; arrange stakeholder interviews and facilitate meetings with stakeholders as part

Provide support to the evaluation process as relevant; provide relevant documentation; facilitate access to project stakeholders if needed; support the implementation of the management



Actor Role in Project Evaluations Role in Programme Level Evaluations

of fieldwork, if included in the evaluation design; review the recommendations focused on the IE and develops the management response

response; support the use of programme level evaluation results in the design of new DA projects

DA Network

Members

Take note of the results of the evaluations that relate to their practice, including the knowledge, lessons and good practices provided and make use of these in their regular DA and other work

Independent

Evaluators

Conduct the project level evaluation in an independent manner, in accordance with the professional standards of evaluators and with the use of the UNEG standards and norms for evaluation; gather and analyse primary and secondary data and prepare the draft and final project inception and evaluation reports, with the latter including evidence-based findings, conclusions, lessons learned/good practices and actionable as well as prioritized recommendations, with clear responsibilities identified;

Evaluators need experience in evaluation of international development programming, making use of the UNEG evaluation criteria, and be familiar with one or more of the DA IEs

Need to be competent in project evaluation in the programmatic area concerned with extensive work experience in this regard

Need to be competent in complex evaluation with extensive work experience in this regard

59. The programme level evaluations are usually managed by the DA PMT in consultation with the DA Focal Point in the IEs, but the evaluation sections of the IEs can also take the lead in relevant programme evaluations. Based on the rolling DA evaluation work plan, the DA PMT or the evaluation sections of the IEs hire the independent consultants who will conduct the programme level evaluations. The DA PMT/the evaluation sections of the IEs manage the entire evaluation process, including quality assurance of inception and draft and final evaluation reports. The evaluation process of programme level evaluation is supported by an Evaluation Reference Group, consisting of relevant representatives of the IEs participating in the evaluation and a representative of the DA PMT. The entire evaluation process is, moreover, supported by the project managers of the projects included in the programme level evaluation, as well as by the DA Focal Points of the IEs of which projects are part of the evaluation.

7. Generation and Use of Learning through Evaluation

7.1 Quality of Evaluations and their Results

60. In order for evaluation results to be used for learning, evaluations and the reports that they generate need to be of high quality. Only in this way can the Evaluation Function present reliable results, which can be used to distil learnings and good practices, which in turn can feed into the design of new DA projects, enhancing the quality of the design, and be used in reporting on DA achievements. Evaluation quality, in both project and programme level evaluations, will be ensured through the alignment of all stages of the evaluation process with the UNEG Evaluation Norms and Standards.7 This includes the utility, credibility, independence, impartiality, ethics and transparency standards of evaluations and processes concerned. Due attention is to be paid to human rights and gender equality, with commitment to the Agenda 2030 principle of ‘leaving no-one behind’. In terms of data gathering, there should be an appropriate balance between primary and secondary data gathering. In the hiring of independent consultants, the DA Evaluation Function will make use of national as well as

7 United Nations Evaluation Group, Norms and Standards for Evaluation, New York, 2016/7.



international evaluation capacities. The DA will support the conduct of high quality and rigorous evaluations, and promote and facilitate the use of evaluation results and learning.

7.2 Use of DA Evaluation Results

61. Evaluation reports are only as relevant as the use that they are put to, as without use of the evaluation results and learnings, there is little reason for getting the quality of those results as rigorous as possible. Use of evaluation results needs to be included from the very start of the evaluation process. This includes identification of users of the evaluation and an understanding of their specific needs. The need for evaluation and details on the requirements concerned should be made explicit at the design phase of DA projects, making use of a results-based and learning focused perspective.

Responding to Recommendations

62. Recommendations of the DA evaluations are aimed at a variety of DA stakeholders and need to be formulated bearing in mind the specific characteristics of the DA, in particular the innovative intention of DA projects and the usual lack of a second project phase, with follow-up mostly organized outside of the DA.

63. For the project evaluations, the DA IEs and their evaluation sections are first in line in terms of responsibility for the evaluation recommendations. This includes owning those recommendations aimed at the IE concerned and to develop a management response. IEs do not necessarily need to agree with each and every recommendation, but need to provide a rationale or indicate otherwise why a response would not be required. The manager of the evaluation further needs to ensure that other recommendations are conveyed to the relevant stakeholders, who will need to own those recommendations and develop their management response, in order to address the issues concerned. Follow-up to recommendations on project evaluations will be done in line with the regular systems that each of the IEs has in place for evaluation of non-DA projects and programmes. Recommendations at the level of the DA Management will need to be handled by the DA PMT.

64. In terms of programme level evaluations, the DA PMT will play the role of ensuring that recommendations are conveyed to the relevant stakeholders, including IEs and other relevant parties. Recommendations aimed at the DA overall will be considered by the DA Steering Committee. For those recommendations aimed at IEs, each of the IEs concerned will develop a management response and follow-up on the issues concerned, making use of their own management systems.

Sharing of Evaluation Results

65. An important aspect of evaluation use concerns the sharing of evaluation results. Many of the entities do publish their DA project evaluation reports on their websites. The entities disclose or publish evaluations as per their disclosure policies. A more pro-active internal sharing of project evaluation results will enhance possible use of the results and the learnings from these evaluations in new DA tranches. The evaluation reports of programme level evaluations will be shared with the Steering Committee and all IEs. Summaries of results will be posted on the DA website.

66. Another important enabler for learning concerns the development of an accessible and easily searchable database of evaluation reports for use of the IEs and the DA PMT. Provided additiional available resources, this can be an important means for sharing and use of learning from the results of past DA project and programme level evaluations. In addition to evaluation reports, the database could include lessons learned, good practices and recommendations. The use of the results and learnings of DA evaluations is of particular importance in the development of concept notes and project designs for new DA projects in new tranches.

Support to Learning through Evaluation

67. In addition to a repository and information sharing, there can be more active ways of supporting learning, sharing of experiences, lessons and good practices across the ten IEs and with



member countries. The DA already has in place the DA Network, consisting of the IE and DA PMT staff engaged in DA project design, implementation and evaluation. Learning functions of the network could be enhanced in a variety of ways:

• Development of a platform to share evaluation results across network members

• Organize a regular exchange to share evaluation results with implementing entities

• Creating a community of DA evaluation practitioners, providing opportunities to share learning and development of capacities on DA related aspects of evaluation

68. In order to enhance and guide learning, it will be useful to address learning at the start of a DA project, in project design. Inclusion of a focus on learning will help the specification of the evaluation criteria and evaluation questions. Some of the learning will be substantive, related to the thematic areas that the projects address. On the other hand, learning can also focus on cross-cutting issues like gender and human rights or process issues such as like partnerships and engagement with country level stakeholders. The DA project evaluation guidelines provide support to enhance learning through project level evaluation.

7.3 Reporting to the General Assembly

69. DA reporting to the General Assembly includes an account of DA projects implemented and results achieved as well as an overview of evaluations conducted at project and programme levels. The DA Evaluation Function provides an important opportunity to inform DA reporting to the UN General Assembly through the results of project and programme level evaluations. Both types of evaluation will provide important details for reporting on results and how these have been achieved. Moreover, reporting will include a focus on lessons learned, good practices identified and knowledge built, in particular in support of reaching the SDGs. Given the innovative and synergetic approach of DA support, in addition to detailing results achieved, a focus on learning obtained and knowledge developed will be important components of DA reporting.



ANNEX 1: Overview of DA Project and Programme Level Evaluations

PROJECT EVALUATIONS

Category Requirement for Evaluation Fund Allocation Evaluation Management Timing of the Evaluation

Projects with a budget of USD 1 million or more

Default selected for evaluation Project budget of IE IE evaluation section Within a time frame of three months after project termination, which could be extended up to six months depending on workload and complexity of the project and its evaluation, to be agreed with the DA PMT and specified in the TOR of the evaluation

Projects with a budget under USD 1 million and randomly selected for evaluation

Up to 50% of projects selected through purposive sampling (selected by the Steering Committee at beginning of a tranche)

Project budget of IE IE evaluation section Within a time frame of three months after project termination, which could be extended to up to six months depending on workload and complexity of the project and its evaluation, to be agreed with the DA PMT and specified in the TOR of the evaluation

PROGRAMME LEVEL EVALUATIONS

Category Requirement for Evaluation Fund Allocation Evaluation Management Timing of the Evaluation

Programme level evaluation(s) identified

Selection proposed by DA PMT, in consultation with IEs, and approval by Steering Committee, of either:

• Cluster evaluation

• Ex-post evaluation

• Concurrent evaluation

• Meta-evaluation/Meta-analysis

DA PMT DA PMT in consultation with the IE as relevant

6-12 months period

SUPPORT TO EVALUATION INFORMED LEARNING

Category Requirement for Evaluation Fund Allocation Management of the support Timing of the Evaluation

Support to sharing and use of evaluation reports and their results

DA PMT DA PMT On-going

Documents

UN Development Account...United Nations Development Account DA Evaluation Framework, October 2019 3 Executive Summary Introduction 1. The United Nations (UN) Development Account (DA)