UNODC Evaluation Policy Independent Evaluation Unit · evaluation standards, as well as other evaluation related matters. 20. As per respective resolutions, “The Executive Director

UNITED NATIONS OFFICE ON DRUGS AND CRIME

Vienna

UNODC Evaluation Policy

Independent Evaluation Unit

UNITED NATIONS New York, 2015

UNODC Evaluation Policy 2

Contents I. Introduction..................................................................................................................... 03

II. Definition of the Institutional Framework, roles and responsibilities ............................ 05

III. Conceptual framework for evaluation at UNODC ......................................................... 08

IV. Evaluation principles ...................................................................................................... 10

V. Planning, managing and budgeting evaluations ............................................................. 12

VI. Follow-up mechanisms to evaluations ............................................................................ 14

VII. Disclosure and dissemination procedures ....................................................................... 14

Annexes

A. Resolutions of the Commission on Narcotic Drugs (CND 52/14) and the Commission on

Crime Prevention and Criminal Justice (CCPCJ 18/6) ........................................................... 16


I. Introduction 1. This document presents the United Nations Office on Drugs and Crime (UNODC) evaluation policy, as the set of principles and rules that guide the Organisation’s decisions and actions when planning, conducting, disseminating and using evaluations. The policy has been consulted with the Executive Director, Senior Management of UNODC and Member States and fully replaces the previous evaluation policy (2004). 2. The Unit responsible for carrying out the independent evaluation function of UNODC is the Independent Evaluation Unit (IEU). This Unit was originally established in 2003 under the Division for Policy Analysis and Public Affairs (DPA). During 2009, the Unit ceased to exist and was re-established as a stand-alone, independent unit on 27 January 2010, as per resolutions of the Commission on Narcotic Drugs(CND) 52/14 (2 December 2009) and Commission on Crime Prevention and Criminal Justice (CCPCJ) 18/6 (3 December 2009)1. 3. This document provides staff, as well as Member States and external evaluators with information on international principles for evaluation, the role and application of evaluation in UNODC, as well as related mandates and resolutions. It serves as a frame, which is complemented by the Evaluation Handbook2. 4. This policy also situates independent evaluations at UNODC into a larger evaluation context at the United Nations. 3 For example, UNODC is an active member of the United Nations Evaluation Group (UNEG), a network which, i.a. develops evaluation Norms and Standards (www.uneval.org). 5. This evaluation policy responds to formal requests from the Office of Internal Oversight Services (OIOS), the General Assembly and the Secretary General, summarized in the regulations “Rules Governing Programme Planning, the Programme Aspects of the Budget, the Monitoring of Implementation and the Methods of Evaluation”

1 “16. Decides that the proposed consolidated budget for the biennium 2010-2011 for the United Nations Office on Drugs and Crime should contain adequate provisions for the establishment of a sustainable, effective and operationally independent evaluation unit.”1

2 The Evaluation Handbook is available online and is updated annually in light of evolving changes, norms and standards in the field of evaluation: http://www.unodc.org/unodc/en/evaluation/index.html 3 In particular, building national evaluation capacity and utilising evaluation practices to foster transparent and evidence-based decision making within Member States’ institutions has been one of several important topics at UNEG. Another salient topic continues being the application of Human Rights and Gender to the services and products an evaluation function like IEU offers. Furthermore, self-reflection and critical thinking is fostered through UNEG by carrying out peer-reviews of evaluations functions across the UN system. UNODC, through IEU, actively participates in these initiatives and shares respective experiences and knowledge with internal and external stakeholders.

http://www.uneval.org/


(ST/SGB/20008). This rule “’mandates the conduct of evaluation in the Secretariat decision making cycle”. 6. Furthermore, the above stated mandate is complemented by UNEG Norms and Standards for evaluation in the UN system4; in particular UNEG Norm 3, which states that “each organization should develop an explicit policy statement on evaluation”, UNEG Standard 1.2: “UN organizations should develop an evaluation policy and regularly update it, taking into account the Norms and Standards for Evaluation in the UN system ”; and UNEG Standard 1.3 “The evaluation policy should be approved by the Governing Bodies of the organization and/or the Head of the organization, and should be in line with the applicable UNEG Norms for Evaluation and with organizational corporate goals and strategies.” 7. UNODC governing bodies resolutions from CND5 and CCPCJ6, contain the request to establish a sustainable, effective and operationally independent evaluation function and invite Member States and the Organisation to contribute to implementing an evaluation culture. In resolution 20/1, the CCPCJ “requests the Secretariat to promote a culture of evaluation throughout the United Nations Office on Drugs and Crime […]”78 . 8. This policy is informed by existing evaluation policies within the UN while meeting the specific needs of UNODC. It also intends to reflect the experience accumulated by UNODC’s evaluation function during the last eight years and adapting the function to new emerging organisational, accountability and learning needs. 9. This policy is divided into 7 sections and one annex with the key CND and CCPCJ resolution.

4 United Nations Evaluation Group (UNEG), Norms for Evaluation in the UN System (2005) and Standards for Evaluation in the UN System (2005). 5 CND 52/14 2 December 2009 6 CCPCJ 18/6 3 December 2009 7 CCPCJ 20/1, para 7 8 Furthermore, CND (56/11, L.11/Rev. 1.1; para 5) and CCPCJ (22/2, L.3, para 5) requested the Independent Evaluation Unit in relation to the standing open-ended intergovernmental working group on improving the governance and financial situation of UNODC, to: “(a) Providing the working group with evaluation findings of the thematic and regional programmes of the United Nations Office on Drugs and Crime; (b) Consulting with the working group on a road map of current and future activities and results; (c) Promoting a culture of evaluation throughout the Office at all stages of programme planning, development and implementation; (d) Working with the Office in monitoring the implementation of recommendations made by relevant oversight bodies.”


II. Definition of the Institutional Framework, roles and responsibilities

A. Independent Evaluation Unit (IEU) 10. The unit responsible for carrying out the evaluation function of UNODC is IEU. To comply with the independence and impartiality inherent to the evaluation function in the United Nations, IEU has to operate on an independent and predictable budget and is to be adequately staffed by evaluation experts as defined by UNEG standards. 11. The Head of IEU is responsible for the independent evaluation function of UNODC9 and reports directly and simultaneously to the Executive Director and to Member States, thus enabling it to conduct its work with impartiality and independence.

12. The Head of IEU is ultimately responsible for ensuring that the Unit meets high professional standards in the implementation of its work plan and the corporate evaluation plan. S/he is also responsible for managing the budget of the Unit and for the appointment of its staff. The Head of IEU is supported in the implementation of the Office work plan by a Deputy, Evaluation Officers, and Associates in addition to an adequate number of general service staff and consultants/analysts who report directly to her or him. 13. IEU’s work is based on three pillars10:

1) Evaluation Reports; 2) Normative Work; 3) Evaluation Culture.

14. The biennial work plan is adopted by IEU following full consultation with Member States and UNODC management based on several criteria including, i.a., relevance, budget, accountability, risk priorities, innovation, replicability potential and evaluation history.11 The plan outlines the type, timing and budget of respective evaluations. It provides an overall framework and allows scheduling and prioritisation of evaluations, while allowing flexibility and responsiveness to evolving needs with provision for changes when required.

9 Following CND Res. 52/14 and CCPCJ Res. 18/6: “The unit should circulate its reports simultaneously to the Executive Director and to Member States, with management responses being provided subsequently, and the reports should be submitted to the Commission for its consideration.” 10 Please find more information in the Evaluation Handbook. Detailed information on the criteria is available in the Evaluation Handbook.


15. A biennial Evaluation Report on the status of the evaluation function in UNODC will be produced by IEU and presented to the Executive Director and Member States. The report will be presented to the Open-ended Working Group on Governance and Finance (FinGov) and, if possible, to CND and CCPCJ. 16. IEU represents UNODC in the United Nations Evaluation Group and other evaluation networks and events as appropriate. 17. IEU is the focal point in UNODC for OIOS and Joint Inspection Unit (JIU) evaluation exercises and studies. 18. IEU coordinates and complements the efforts of Oversight Functions (e.g. OIOS, JIU) both at headquarters (HQ) and the field.

B. Executive Director 19. The Executive Director is formally consulted on the biannual work plan of IEU and receives the final evaluation reports for information. The Head of IEU briefs the Executive Director on a regular basis on evaluation results, progress made on building an evaluation culture, recommendations implementation rate, compliance with evaluation standards, as well as other evaluation related matters. 20. As per respective resolutions, “The Executive Director should ensure independent, timely and effective evaluations, in line with the standards and norms of the United Nations Evaluation Group and following the recommendations of the Office of Internal Oversight Services”12 contributing to establishing and keeping an enabling environment where an independent evaluation culture can take root.

C. Commission on Narcotic Drugs (CND) and the Commission on Crime Prevention and Criminal Justice (CCPCJ) 21. Governing bodies of UNODC (CCPCJ and CND), are key stakeholders in accountability matters for the organisation. The resolutions from these governing bodies establish the legal and operational conditions for the independence of the evaluation function. IEU, to the extent possible, informs the Governing bodies about the biennial work plan and budget and they may also comment on and acknowledge the biennial Evaluation Meta-Analysis that may be used for strategic decision making.

12 (CCPCJ, Res. 18/6, para 16.b, 2009)


D. Intergovernmental Working Group on Governance and Finance (FinGov) 22. As the major permanent representation of the governing bodies of UNODC (CCPCJ and CDN), FinGov may monitor that operational conditions for the independence of IEU are in place, contributing thereby to creating an enabling environment for a culture of evaluation for UNODC. 23. Member States at FinGov are briefed on the implementation of the IEU work plan and in-depth evaluation results and provide feedback on evaluation-related matters. Members States in this group are consulted on evaluation priorities and the biennial work plan of IEU.

E. Executive Committee 24. The Executive Committee is consulted regarding IEU´s proposed Biennial Work Plan, which details the evaluation priorities for the two upcoming years. IEU briefs the Executive Committee on evaluation results and progress made on evaluation culture and engages with this Committee as requested.

F. Programme Review Committee (PRC) 25. IEU is a member of PRC and ensures that provision for evaluations are included both at substantive and financial level for all UNODC interventions. Therefore, “all country/regional and thematic programmes must be cleared by IEU […]. In particular IEU will be asked to propose a concrete evaluation methodology for all programmes at the planning stage itself.”13 Furthermore, consultations with IEU are mandatory for projects to be cleared by PRC, as well as projects approved using direct approval methodology.14 These provisions support to a certain extent the conditions for evaluability of projects and programmes15.

G. UNODC Senior Management 26. Evaluation is part of the programme and project cycle and is therefore an institutional responsibility to be taken up by all managers in UNODC. Senior Management ensures that adequate resources for evaluation are reserved under budget line 5700 and draws on evaluation findings to guide strategic decision-making on future programming. International standards followed in the United Nations

13 Interoffice Memorandum 24 May 2011, “Evaluation in the Planning process” 14 Ibid. 15 Evaluability furthermore depends on UNODC´s ability to institute sound monitoring and results-based management.


recommend allocating a range of 2% to 3% of the overall budget of a project or programme for evaluation purposes. IEU adheres to and applies these standards to ensure the validity, relevance and usefulness of the evaluation deliverables. 27. Project and Programmes selected for In-Depth evaluations will open a specific budget segment prior to start the evaluation where they will allocate the evaluation funds to be managed by IEU during the evaluation process. This constitutes a shift, which will be implemented through the launch of this evaluation policy. The implementation of this shift will be closely monitored and results will be shared within one year of implementation.

H. Member States 28. IEU presents evaluation findings, the recommendations’ implementation rate, the compliance with evaluation standards in UNODC, and other evaluation-related topic to Member States on a regular basis. This reporting is done through FinGov as well as bi- and multilateral meetings with Member States. 29. Member States are invited to actively participate in evaluations, e.g. as Core Learning Partners, in order to increase the ownership and use of evaluation results.

III. Conceptual framework for evaluation at UNODC

A. Definition of evaluation 30. UNODC subscribes to UNEG’s definition 16 of evaluation: “An assessment, as systematic and impartial as possible, of an activity, project, programme, strategy, policy, topic, theme, sector, operational area, institutional performance, etc. It focuses on expected and achieved accomplishments, examining the results chain, processes, contextual factors and causality, in order to understand achievements or the lack thereof”. 31. Evaluation at UNODC aims at determining the relevance, effectiveness, efficiency, impact and sustainability of the interventions and contributions of the organization’s institutional performance to fulfil its specific mandates. Furthermore, additional criteria related to measuring the dimensions of partnerships, gender and human rights are mandatorily added to the above criteria, reflecting the need to understand how UNODC has managed to mainstream gender and human rights into its investments and how partnerships with different stakeholder groups have been identified and managed.

16 Norms for evaluation in the UN System UNEG/FN/Norms(2005)


B. Purpose of evaluation 32. At UNODC IEU provides for evidence-based information that is credible, reliable and useful to serve three main purposes: accountability, organizational learning and knowledge generation. In addition, the evaluation function contributes to developing UNODC’s capacities in terms of innovation and organizational change, providing building blocks to a learning Organisation. 1. Accountability 33. The evaluation function plays a critical role in supporting accountability by independently conducting evaluations which assess compliance with the established conventions, treaties, norms, policies and plans, and report fairly and accurately on results to the Executive Director, UNODC at large and Member States. 2. Organizational Learning 34. Evaluation includes measuring the extent to which intended and unintended results are or are not achieved and their impact on stakeholders. It also deals with answering difficult questions, such as whether the Organisation is doing the right things and whether it is doing things right. In this sense, it becomes an important source of evidence about institutional performance and a key contributor to organizational learning. Through the timely incorporation of recommendations and lessons learned into the decision-making processes of the Organisation, evaluation aims at being of use throughout the project or programme cycle as well as at the level of policy formulation. 3. Knowledge Generation 35. Undertaking evaluations produces substantive knowledge on the specific topics under UNODC’s mandates. This knowledge is delivered in form of evaluation reports which contain recommendations and lessons learned. IEU seeks to compile, synthesize and disseminate this knowledge to the benefit of UNODC’s stakeholders as well as the United Nations Organisations at large and its Member States.

C. Types of evaluation in UNODC 36. As evaluation is part of the project and programme management, all projects and programmes in UNODC are to be evaluated according to UNODC evaluation policy, handbook and guidelines at least every 4 years or 6 months before the project or programme finalizes. In addition, policies or instruments may also be evaluated, as per the biannual evaluation plan.


37. The evaluation function in UNODC produces two different types of evaluations:

1. In-Depth Evaluations: Strategic evaluations of relevance to the organisation, such as country, regional, thematic, global programmes, cross-cutting issues, for example gender, human rights, etc., or corporate policies. A cluster evaluation is an evaluation of a set of related projects which aims at identifying commonalities and synergies across projects as well as determining the progress made towards a wider programming objective. A set of related projects identified by management and IEU as of key relevance for the Organisation may qualify as an In-Depth Evaluation. A joint evaluation, in which UNODC jointly carries out an evaluation with another implementing partner, also qualifies as an In-Depth evaluation. 2. Independent Project Evaluations: the unit of analysis is an individual project designed to achieve specific objectives within specified resources, in an adopted time span and following an established plan of action.

38. Participatory Self-Evaluation, included in the previous Evaluation Policy for small projects with a budget below USD 1,000,000, will not be continued in UNODC. After a thorough review of the methodology and feedback received from internal and external stakeholders, Participatory Self-Evaluations were discontinued as of 31 December 2014. All such UNODC projects or programme will need to plan and budget for independent project evaluations.

IV. Evaluation principles 39. UNODC is a member of UNEG and has adopted its Norms and Standards for Evaluation17. These norms reflect the singularity of the United Nations system, which is characterized by its focus on people and respect for their rights, the importance of international values and principles, universality and neutrality, its multiple stakeholders, its needs for global governance, its multi-disciplinarily and complex accountability system.

A. Evaluation is independent and impartial 40. The evaluation unit at UNODC ”should be functionally and operationally independent and should be part of the Office of the Executive Director”18 Following UNEG Norms for Evaluation in the UN System: “The evaluation function has to be

17 Norms for evaluation in the UN System UNEG/FN/Norms(2005) Standards for Evaluation in the UN System UNEG/FN/Standards(2005) 18 CND 52/14 2 December 2009; CCPCJ 18/6 3 December 2009


located independently from the other management functions so that it is free from undue influence and that unbiased and transparent reporting is ensured.” 19 The objective is to have full authority to plan, conduct and present evaluation reports to appropriate decision-making levels directly. Management must not impose any type of restrictions and specifically on the scope, content and recommendations of evaluation reports.

B. Evaluation is transparent and participatory 41. The evaluation process at UNODC is transparent and involves relevant stakeholders at key stages of the UNODC Evaluation process, starting with the drafting of the Terms of Reference. Information on the evaluation approach, design and methodology must be shared throughout the process. This is essential for the credibility and utility of the evaluation and facilitates consensus building and ownership of the findings and recommendations. The transparency in the evaluation process improves credibility and the quality of the evaluations.

C. Evaluation is utilization focused 42. Evaluation products must be timely and tailored to meet the needs of its intended users. The analysis of findings by evaluators has to consider the realities of the programme or country context, and recommendations have to be practical and realistic to be implemented. 43. Evaluations require a commitment to respond and act upon by the respective authorities and managers, addressing the recommendations derived from evaluation. A systematic follow-up on the implementation of the recommendations is facilitated through respective electronic tools provided by IEU. Responsibility to implement recommendations rests with management, as delineated in the “Follow-up Action Plan” for each evaluation. 44. IEU monitors the implementation rate of recommendations and requests responsible managers to update the status of the recommendations. IEU reports the recommendation implementation rate to the Executive Director and Member States on an annual basis.

D. Evaluation conforms to internationally accepted standards 45. The standards on evaluation of UNEG and Development Assistance Committee (OECD/DAC) guide the activity of the evaluation function in UNODC. Each evaluation

19 N6 - Norms for evaluation in the UN System UNEG/FN/Norms(2005)


should employ design, planning and implementation processes that are inherently quality oriented, covering appropriate methodologies for data-collection and analysis.

E. Human Rights and Gender 46. UNODC incorporates specific principles and safeguards to ensure that all evaluations undertaken or commissioned by UNODC include a focus on protection of human rights and gender issues following UNEG guidance UNEG/G(2011)2 Integrating Human Rights and Gender Equality in Evaluation 20

F. Ethics 47. Evaluators should have personal and professional integrity and abide by the UNEG ethical guidelines for evaluation and the UNEG code of conduct for evaluation in the United Nations system, to ensure that the rights of individuals involved in an evaluation are respected. Evaluators must act with cultural sensitivity and pay particular attention to protocols, codes and recommendations that may be relevant to their interactions with women, minority groups, etc.

V. Planning, managing and budgeting evaluations

A. Planning for evaluations 48. As evaluation is part of the project and programme cycle and all projects and programmes in UNODC are to be evaluated, the planning for evaluation in line with UNODC evaluation policy and handbook is the responsibility of Project and Programme Managers. 49. The Project or Programme Manager is responsible for reserving sufficient funding for the mandatory evaluations under budget line 5700 and planning for the evaluation exercise on time. In accordance with international standards, the budget reserved for any type of evaluation should be 2-3% of the overall budget of the project or programme.

B. Budget and Management Arrangements 50. All UNODC projects and programmes are evaluated and therefore the subsequent managerial and budgeting arrangements need to be included at the design stage of the

20

Integrating Human Rights and Gender Equality in Evaluation Towards UNEG Guidance UNEG/G(2011)2


intervention before a final approval can be granted. Therefore, IEU must be consulted on a mandatory basis prior to any project or programme approval or revision.21 51. In-depth evaluations are financed by the programme and supported by IEU through staff time and expertise. IEU commissions, manages and implements this type of evaluation in consultation with respective managers, ensuring the overall quality, validity, relevance and usefulness of the evaluation deliverables. IEU adheres to and applies the international standards to arrange for an allocation of 2-3% of the overall budget of the project or programme for evaluation purposes. Project or Programmes selected for In-depth evaluations will need to reserve these funds under budget line 5700 and open a specific budget segment at the onset of the evaluation, allowing for these funds to be managed by IEU during the evaluation process. Only after such budget segment is set up, IEU starts the planning phase of the evaluation by initiating the drafting of the Terms of Reference (TOR). 52. For all Independent Project Evaluations, managers must include an evaluation plan in their project document and arrange for an allocation of 2-3% of the overall budget of a project to cover all related evaluation costs under budget line 570022. Following detailed guidelines in the Evaluation Handbook, managers and independent evaluators will work according to clearly assigned roles in the evaluation process. 53. Member States representatives – including donors – may observe an evaluation process, following an agreement as per the Evaluation Handbook and IEU. 54. The IEU website23 includes a step by step evaluation platform to manage evaluation processes with guidelines, templates and evaluation reports, financed and maintained by IEU to provide services to UNODC staff who is engaged in the evaluation process both at HQ and the field. 55. External quality rating of in-depth evaluations may be carried out through an independent, external service provider, following best practices within the UN system, thereby disclosing to the public the overall quality of the evaluation report following a distinct set of criteria. 56. In order to promote transparency and ensure the maximum objectivity of evaluations, UNODC relies on external, independent evaluators selected on the basis of

21 Special Memo 11/2011 Chair of PRC: “submitting offices must undertake consultations with IEU for all projects and programmes (and revision to the same) to be approved […] consultations with IEU should be undertaken well before the project document/project revision is submitted in ProFi. […] IEU will provide advice and assistance regarding details of the planned evaluation to be included in the document, e.g. type of evaluation, timing, scope, purpose, etc” 22 UNODC Evaluation Handbook 23 http://www.unodc.org/unodc/en/evaluation/index.html

http://www.unodc.org/unodc/en/evaluation/index.html


their competence, independence and integrity. In the case of Independent Project Evaluations, evaluators are selected by means of a transparent process by the Project Manager, following Human Resources Management Service rules as well as IEU norms and standards24. In the case of In-depth Evaluations, evaluators are identified and selected by IEU in consultation with managers. 25 57. Independent evaluators must have an understanding and experience of evaluation concepts, techniques and ethics, and be able to work as part of an international team. Whenever possible, local evaluators should also be included since they are familiar with the local context and speak the local language. The involvement of local evaluators also contributes towards building local evaluation capacity. Furthermore, the composition of the evaluation team should be gender balanced. 26

VI. Follow-up mechanisms to evaluations 58. IEU facilitates the systematic follow-up of management on the status of implementation of evaluation recommendations, in which management details, responsibilities and timelines to implement the respective recommendations are included. In-Depth Evaluations are accompanied by a Management Response, which delineates management’s reaction to the findings and recommendations contained in the report. The Management Response, as well as the Follow-up Action Plan, are attached to and published with the final evaluation report. 59. IEU is responsible for establishing dissemination and feedback mechanisms for communicating evaluation results within UNODC, as well as to Member States. Making information easily accessible to UNODC´s main stakeholders allows evaluation recommendations, best practices and lessons learned to inform the planning of future interventions and policies.

VII. Disclosure and dissemination procedures 60. IEU disseminates all Independent Project and In-Depth Evaluations through its website27. 61. The further dissemination of evaluation reports rests with the respective managers or stakeholders.

24 Further information on the selection of evaluators is provided in the Evaluation Handbook. 25 More details regarding the qualification of independent evaluators can be found in the Evaluation Handbook: http://www.unodc.org/unodc/en/evaluation/index.html 26 More details regarding the qualification of independent evaluators can be found in the Evaluation Handbook: http://www.unodc.org/unodc/en/evaluation/index.html 27 http://www.unodc.org/unodc/en/evaluation/publications.html



http://www.unodc.org/unodc/en/evaluation/publications.html


62. Both, IEU and management are responsible for disseminating independent evaluation findings and encouraging the development of a learning community. Country and Regional Offices are encouraged to translate the executive summary into local languages and use other means in order to inform stakeholders of findings and to enhance learning.


VIII. Annexes

A. Resolutions of the Commission on Narcotic Drugs (CND 52/14) and the Commission on Crime Prevention and Criminal Justice (CCPCJ 18/6)




Documents

UNODC Evaluation Policy Independent Evaluation Unit · evaluation standards, as well as other evaluation related matters. 20. As per respective resolutions, “The Executive Director