Regional Evaluation Strategy 2018-2021 UNICEF East Asia ... › eap › sites › unicef.org.eap...UNEDAP United Nations Evaluation Development for Asia and the Pacific UNEG United

Evaluation section

UNICEF EAPROJune 2017

Regional Evaluation Strategy 2018-2021UNICEF East Asia and the Pacific

Copyright: United Nations Children’s Fund (UNICEF) East Asia and the Pacific Regional Office Evaluation SectionDate of Final Version: June 2017Cover photo: A young girl with a cooking pot over her head at the local market close to the Sin Tet Maw camp for internally displaced persons (IDPs) in Arakan State, Rakhine State, Myanmar, Saturday 8 April 2017. © UNICEF/UN061856/Brown

Evaluation section

UNICEF EAPROJune 2017

Regional Evaluation Strategy 2018-2021UNICEF East Asia and the Pacific

AcknowledgementsThis regional evaluation strategy and action plan is the result of hard work of the EAPRO Evaluation section, the COs in the East Asia and the Pacific, the Evaluation Office in New York as well as colleagues from the Regional Office for South Asia and the meaningful contribution from Michael Quinn Patton.

ii

ForewordDear colleagues,

In his opening statement at the June 2017 Executive Board meeting, Antony Lake, UNICEF’s Executive Director, indicated that “our evaluation function is helping design, target and deliver interventions that will make the biggest difference in children’s lives. Evaluations demonstrate what works and what does not, and help us build a strong evidence base to constantly improve our programmes”. By supporting organizational learning and accountability, evaluations help UNICEF to continually improve its performance and results. Good evaluations serve UNICEF’s mission and promote its mandate to protect and promote children’s rights.

Our Evaluation Policy defines an evaluation “as a shared function within UNICEF” and calls for regional offices, to develop regional strategies that move the role of evaluations beyond project accountability and contribute towards better programme results, organizational performance and institutional advocacy. Thus, I am pleased to share the East Asia and the Pacific “Regional Evaluation Strategy and action plan: 2018–2021” approved during our Regional Management Team (RMT) meeting in April 2017.

This Regional Evaluation Strategy has been designed to help UNICEF senior managers strengthen the evaluation function in the East Asia and Pacific region so that the organization generates good-quality evidence that informs policy, programming and advocacy and ultimately contribute towards better results for children.

Kind regards,

Karin Hulshof Regional DirectorEast Asia and the Pacific Regional Office

iii

Executive summary By supporting organizational learning and accountability, evaluations help UNICEF to continually improve its performance and results. Good evaluations serve UNICEF’s mission and promote its mandate to protect and promote children’s rights.

Our 2013 revised Evaluation Policy reflects UNICEF’s commitment to demonstrate results and improve performance, learning and accountability. The evaluation function is carried out at all levels of the organization and in all contexts, from humanitarian crisis to transition situations to more steady development environments.

The Evaluation Policy defines an evaluation “as a shared function within UNICEF” and calls for regional offices, under the leadership of the Regional Directors, to develop regional strategies that move the role of evaluations beyond project accountability and contribute towards better programme results, organizational performance and institutional advocacy.

In East Asia and the Pacific, the UNICEF Evaluation Office in New York, the Regional Office (EAPRO) and its country offices are to work together to strengthen the evaluation function. EAPRO, however, retains an oversight, guidance, technical assistance and quality assurance role so that evaluations managed or commissioned by UNICEF (regional office and country offices) uphold high-quality standards.

Purpose of the strategy

As noted in the Global Meta-Evaluation Report 2014, “Because UNICEF is decentralized in nature, its evaluations are generally commissioned and managed at the country office level. On one hand, such an arrangement helps ensure that report analyses remain highly focused on the national context, but on the other, this decentralized system makes it difficult to maintain uniform quality, high credibility and utility of the evaluations produced organization-wide.”

This Regional Evaluation Strategy was designed to help senior managers corporately prioritize the evaluation function so that the organization generates good-quality evidence that informs policy, programming and advocacy and ultimately contribute towards better results for children. It intends to contribute to improve country office evaluation planning, budgeting, implementation, dissemination and use of findings.

In April 2017, the Regional Management Team approved the Strategy and action plan, thus endorsing five priorities: (i) prioritize evaluations and embed the process into the results-based management cycle; (ii) introduce or strengthen quality assurance systems; (iii) reinforce UNICEF staff capacity development; (iv) support national evaluation capacity development; and (v) maintain independence and credibility of evaluation findings. This will trigger transformational learning and adaptive management within UNICEF.

iv

To achieve the strategic priorities, the UNICEF Regional Director and Representatives in the East Asia and Pacific region have agreed to:

· Allocate dedicated and qualified human and financial resources and set up effective management and governance structures that preserve the independence and impartiality of the evaluation function. EAPRO and country offices will allocate, on average, 1 per cent of their budgets to cover the evaluation function.

· Carry out a minimal number of evaluations per management plan cycle. EAPRO will conduct at least two evaluations during its new regional office management plan cycle (2018–2021). Larger country offices in the East Asia and Pacific region have agreed to conduct at least five evaluations per country programme cycle, while medium-sized and smaller country offices will carry out at least three evaluations.

· Systematically use evaluation findings for strategic decision-making, such as reorienting the country programme or adjusting the programmatic area objectives. When commissioning and conducting evaluations, EAPRO and country offices need to have a clear intention to use the resulting analysis, conclusions and recommendations to inform decisions and actions.

· Prioritize national evaluation capacity development initiatives that engage government and development partners. Within the 2030 Agenda for Sustainable Development, evaluations have been given elevated significance because of their utility in helping countries measure their progress towards achieving the targets of the Sustainable Development Goals.

Intended audience

The primary audience of this document is senior management in the East Asia and Pacific Regional Office and country offices. In addition to the regional director, the deputy regional director and the section chiefs, the country office representatives, deputy representatives and planning, management and evaluation staff as well as programme staff will find the Regional Evaluation Strategy of importance to their work. The Evaluation Office and Field Results Group, the Office of Research, the Office of Emergency Programmes and the programme division at headquarters comprise the secondary audience.

v

ContentsAcknowledgements ii

Foreword iii

Executive summary iv

Abbreviations vii

Context and the need for an improved evaluation culture 1

i. The changing developmental paradigm gives a central role to evaluations 1

ii. Overview of the UNICEF evaluation function in the East Asia and Pacific region 2

iii. What do country offices request in terms of regional office support and technical assistance to improve the evaluation function? 6

Regional Evaluation Strategy 9

iv. What does the region need to prioritize? 9

v. How? The way forward. 11

Action plan 16

Process 16

Impact statement 16

Outcome statement 16

Intermediary outcomes 16

Specific outputs 16

Annexes 26

Annex 1. UNICEF accountabilities to evaluate at the regional and country levels 27

Annex 2. Comments on UNICEF country offices progress and challenges in the East Asia and Pacific region, 2015–2016 29

Annex 3. GEROS-reviewed completed evaluations 32

Annex 4. UNICEF [country office]: Standard operating procedures for better evaluations (Draft – 19 June 2015) 34

Annex 5. Analytics of the requests received in 2016 42

List of figures

Figure 1: Theory of Change on how to strengthen the UNICEF evaluation function in the East Asia and the Pacific region 17

vi

AbbreviationsAPEA Asia Pacific Evaluation Association

CEP costed evaluation plan

CO country office

CP country programme

CPD Country Programme Document

DREAM Data Research Evaluation and Monitoring Annual Meeting

DROPS deputy representatives and operations

EAPRO East Asia and the Pacific Regional Office

EMOPS Office of Emergency Programmes

EO Evaluation Office

GEROS Global Evaluation Reports Oversight System

IMEP Integrated Monitoring and Evaluation Plan

JPO Junior Professional Officer

M&E monitoring and evaluation

MR management response

NECD National Evaluation Capacity Development

OECD Organisation for Economic Co-operation and Development

PME planning, monitoring and evaluation

PRIME Integrated Monitoring Evaluation and Research Planning

QA quality assurance

RBM results-based management

RD regional director

RMT Regional Management Team Meeting

RO regional office

ROMP Regional Office Management Plan

ROSA Regional Office for South Asia

SOP standard operating procedures

UNDAF United Nations Partnership Development Framework

UNDP United Nations Development Programme

UNEDAP United Nations Evaluation Development for Asia and the Pacific

UNEG United Nations Evaluation Group

UNICEF United Nations Children’s Fund

UNITAR United Nations Institute for Training and Research

WASH water, sanitation and hygiene

vii

8-day-old son (no name yet) at Marara Clinic in Honiara. Sabina came for general check ups of her baby and holds him while she waits, Solomon Islands/2017© UNICEF/UN062221/Sokhin

viii

UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–2021 1

Context and the need for an improved evaluation culture

i. The changing developmental paradigm gives a central role to evaluations

1. Despite the various breakthroughs that the Millennium Development Goals achieved, it became evident late in that experience that the shortfalls were partly due to the absence of appropriate monitoring and evaluation systems. The next iteration of development targets would not be remiss. During the 2015 United Nations Evaluation Group’s (UNEG) Norms and Standards for Evaluation High-Level Group Event, the former United Nations Secretary-General recognized that “evaluation is everywhere and, at every level, will play a key role in implementing the new development agenda”. Thus, as the 17 goals came together within the 2030 Agenda for Sustainable Development, the evaluation function became an imperative for performance measurement, learning and general accountability of the development paradigm.1

2. United Nations Member States also recognize that evaluations are a core function in their development processes because they help strengthen and support development results.2 And development partners accept that they need to generate and use evidence to demonstrate that they are achieving results.

3. This shift towards greater learning and accountability represents opportunity for UNICEF to advocate for independent, credible, good-quality and useful evaluations for evidence-based policy-making at the global, regional, national and local levels. Evaluation findings should inform the implementation, follow up and review of progress towards the SDGs at the global and national levels. National development policies need to be informed by credible and independent evidence. To do so properly, adequate national government, bilateral and multilateral donors’ resources need to be invested.

1 According to the General Assembly draft outcome document on the post-2015 development agenda.2 United Nations General Assembly resolution A/RES/69/237 on capacity building for the evaluation of development activities at the country level.

UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–20212

ii. Overview of the UNICEF evaluation function in the East Asia and Pacific region

The 2013 UNICEF revised Evaluation Policy governs the organization’s evaluation function and provides a comprehensive framework for all evaluation activities we undertake. The policy states that evaluations “unequivocally serve the organization’s mission and supports UNICEF in fulfilling its mandate”. By supporting organizational learning and accountability, evaluations help UNICEF continually improve its performance and results. As the policy notes, evaluations in UNICEF serve “to support planning and decision-making and to provide a basis for informed advocacy—aimed at promoting the well-being of all children, everywhere.” In focusing on the substantive rationale, value and performance of interventions and institutional functions, evaluations improve results and stakeholder satisfaction. This function is carried out at all levels of the organization and in all contexts, from humanitarian crisis to transition situations to more steady development environments.

The policy also acknowledges that evaluations at the regional and country levels are especially important because they provide reliable evidence to inform decision-making within UNICEF and among its partners and stakeholders and for well-founded advocacy and advice. The Evaluation Policy calls for regional offices, under the leadership of the respective regional directors, to develop regional strategies and engage senior management attention in the Regional Management Team (RMT) and elsewhere. The policy regards the evaluation practice “as a shared function within UNICEF”.3 Roles are distributed across senior leaders and oversight bodies, heads of offices, technical evaluation staff and sector-based programme staff. Accountabilities are distributed at (i) the headquarter level, (ii) regionally and (iii) the country level.4

34

4. The UNICEF Evaluation Office in New York, its East Asia and Pacific Regional Office (EAPRO) and its country offices generally collaborate to strengthen the organization’s evaluation function. The regional office has an oversight, guidance, technical assistance and quality assurance role, aiming to ensure that the evaluations managed or commissioned by UNICEF (regional office and country offices) uphold the high-quality standards set for them. The regional office and country office evaluation activities also include developing nationally and regionally specific evaluation strategies, engaging in partnerships for evaluation and supporting national evaluation capacity development.

5. Because it is an institutional priority, the evaluation function has been established over time in all country offices. With EAPRO 2014–2017 priorities aimed at strengthening the use of the evaluation function “to support evidence-based and critical decision-making at the programmatic and policy level”, the quality of evaluations being conducted (Annexes 2 and 3) and the use of findings has been steadily improving.5

3 UNICEF (2011) defines an evaluation as a “judgement [on] the relevance, appropriateness, effectiveness, efficiency, impact and sustainability of development efforts, based on agreed criteria and benchmarks among key partners and stakeholders. It involves a rigorous, systematic and objective process in the design, analysis and interpretation of information to answer specific questions. It provides assessments of what works and why, highlights intended and unintended results, and provides strategic lessons to guide decision-makers and inform stakeholders.”

4 For details, see the revised Evaluation Policy of E/ICEF/2013/14, pp. 7–10.5 For example, the 2015 Malaysia equity evaluation, the Timor-Leste water, sanitation and hygiene evaluation and the Viet Nam mother tongue

evaluation.


6. But the improvements are uneven across the region, with the foundations of the evaluation function and the quality, credibility and use of evaluations findings still weak in several country offices. The following discusses the continuing challenges to a strong evaluation function in UNICEF’s work as well as opportunities to reach the level of quality required.

Challenges

7. There is a need for a plan to strengthen the evaluation function generally.6 There is a proliferation of strategies across UNICEF,7 and the level of effort needed to roll them out within the organization is challenging because they all demand dedicated resources, proper systems and processes.

8. With few exceptions, evaluations tend not be used as the basis for strategic decision-making (such as reorienting the position of the country office or the country programme). Several country offices still plan their evaluations on an annual basis, drafting their Integrated Monitoring and Evaluation Plan essentially as a wish list. Project-level evaluations prevail, generally driven by bilateral donors’ demands for upwards accountability. This often triggers “evaluation fatigue”. To overcome this, better planning and prioritization and better use of evaluation findings are critical.8

9. Despite country office efforts, dedicated and qualified professional human resources for planning and managing evaluations and overseeing the quality and use of deliverables are limited. Country office planning, management and evaluation (PME) staff9 and monitoring and evaluation staff continue to dedicate most of their time to planning and monitoring and are left without proper time and resources to plan and manage evaluations or to properly promote use of the findings. This, coupled with the downsizing of many country offices, is affecting evaluation capacity, with monitoring and evaluation posts being cut or downgraded. Country offices tend to overcome this human resource deficit by engaging sector programme staff in the management of evaluations. But these individuals tend to be unfamiliar with the UNEG-defined norms and standards for evaluations, which can jeopardize the evaluation function’s credibility. This is also affecting the independence and impartiality of the evaluation standards set in the Evaluation Policy, with programme managers evaluating their own programmes. A recent self-assessment found that only 22 per cent of country offices globally have an environment in which PME or monitoring and evaluation staff report to the country representative. Many staff report to the planning and monitoring staff in charge or the deputy country representative, and 23 per cent report to a section chief, with roles and responsibilities interpreted differently across country offices, despite the guidance provided by the Evaluation Office.

6 Global Evaluation Committee, June 2015.7 As noted during the September 2014 Global Evaluation Committee meeting.8 According to the Evaluation Policy, a country office needs to ensure an evaluation is undertaken: (a) before a programme replication or scaling up

(pilot initiatives); (b) when responding to major humanitarian emergencies; (c) following long periods of unevaluated programme implementation, especially when the programme has been implemented for at least five years without any evaluation activity; (d) when expenditure for each outcome has reached US$10 million; and (e) when the average annual expenditure for each outcome exceeds US$1 million.

9 According to a 2011 global survey, PME staff only dedicate 14 per cent of their time to evaluations. This limited time for evaluations was noted during the June 2015 deputy representatives and operations meeting.


10. An alternative approach recently tested in three country offices (Cambodia, Malaysia and Myanmar) is a specialist evaluation staff position reporting directly to the representative to ensure independence from the programmes and making technical reports to the regional evaluation adviser.10 As pointed out in a recent exploratory study on the decentralized evaluation functions across UNEG agencies, this approach can boost evaluation capacity at the country level and promote efficiencies. The multi-country approach allows the sharing of costs between country offices. Although more coordination is required, the approach allows staff to positively influence the evaluation system and culture of country offices. It also allows greater consistency, access to resources and the sharing of monitoring and evaluation tools. And it facilitates replication of good practices. By technically reporting to the regional evaluation adviser, the specialist is in a better position to implement the regional strategy at the country level. Before engaging further in shared posts, however, the human resources section is evaluating whether this option could be more systematically applied in our region and in others.11

11. Evaluation teams are often led by consultants with sound technical sector expertise but with limited evaluation experience. Teams that are not familiar with good evaluation methods and UNEG’s quality standards can produce poor-quality reports, especially when evidence is not sufficiently triangulated. Several evaluation reports submitted to the regional office for quality assurance, for example, read more like progress reports than a proper independent and evidence-based evaluation. This improper format inhibits adequate learning and accountability at both the regional and national levels.

12. There is still need for quality assurance and effective use of the evaluation findings. Often the purpose and objectives of the evaluation are not always shared at the country office level (as reflected in the terms of reference); stakeholders are not involved throughout the evaluation process, thus limiting the level of ownership and active engagement. As noted in a recent meta-evaluation, “Because UNICEF is decentralized in nature, its evaluations are generally commissioned and managed at the country office level. On one hand, such an arrangement helps ensure that report analyses remain highly focused on the national context, but on the other, this decentralized system makes it difficult to maintain uniform quality, high credibility and utility of the evaluations produced organization-wide.”12 Emerging good practices in UNICEF’s work especially need to be more robustly documented through evaluations.

13. There are no indicators to determine the use of evaluation findings for advocacy purposes or as inputs for programming and other decision-making, even though the evaluation management response submission rate has reached 100 per cent, and the completion rate of actions required has steadily increased.

10 This approach allows country offices to have evaluation specialists report to the representative while programme managers report to the deputy representative. This appears to be a successful option when roles and responsibilities of the shared evaluation post are articulated by each country office in relation to other PME or M&E staff. Other country offices, such as Lao PDR, Thailand and Viet Nam, initially considered establishing a similar shared post but soon realized that they did not have sufficient resources to fund the position for at least two years.

11 Other options that could be considered would be that the Social Policy section takes the lead on the PME function, supplemented by a national officer, technical assistance and ad hoc consultancies for managing and providing quality assurance of evaluations.

12 GEROS: Global Meta-Evaluation Report 2014, Universalia (2015), p. 2.


Opportunities

14. Coverage and quality of evaluations is progressively improving in the region. In the past three years, 13 of the 14 country offices completed at least one evaluation.13 According to the 2017 evaluation office report to the UNICEF Executive Board, the quality of country office evaluations in the East Asia and Pacific region have progressively improved.

15. The average budget use for evaluation in the region has skyrocketed, going from 0.2 per cent in 2014 to 1.8 per cent in 2016.14 Over this period, East Asia and the Pacific progressed from the second-lowest ranking region in terms of budget use for evaluations to the highest rank. The number of country offices spending more than 1 per cent of their programme expenditure quintupled between 2014 and 2016. Despite that staggering progress, unevenness prevails in the region; some country offices spend 3 per cent of their budget for evaluations, while the regional office only dedicates 0.1 per cent.

16. Since 2014, a costed evaluation plan accompanies every Country Programme Document (CPD),15 thus anchoring the evaluation function in UNICEF’s results-based management cycle. In 2017, a total of 11 country office CPDs will have a costed evaluation plan (such as Cambodia, Indonesia, Thailand and the Philippines). This should allow the country offices to take a more strategic medium-term approach for ensuring programmatic coverage and progressively engage UNICEF to support country-led evaluations.

17. See Annex 2 for more detailed comments on UNICEF country offices progress and challenges in the East Asia and Pacific region, 2015-2016.

13 EAPRO has not completed a regional evaluation since 2013, although it did co-manage and quality monitor two bi-regional evaluations with ROSA in 2016.

14 Only three other regions spend more than 1 per cent: Latin America and Caribbean Regional Office, at 1.4 per cent; Eastern and Southern Africa Office, at 1.3 per cent; and the Regional Office for South Asia, at 1.1 per cent.

15 Costed evaluation plans will be developed for every new UNDAF, which is an important development for those countries in the region that have a common programme of cooperation with their host government.


iii. What do country offices request in terms of regional office support and technical assistance to improve the evaluation function?

18. Most country offices’ requests seek guidance on planned and ongoing evaluations and for quality assurance of evaluation deliverables. In 2016, the Evaluation section provided support, quality assurance and comments to more than 94 evaluation deliverables (see Annex 5), including terms of references and inception, draft and final evaluation reports from country offices in the East Asia and Pacific region, bi-regional and global evaluations. An assessment of those items indicate that quality assurance mechanisms are not in place at the country office level. With few exceptions, country offices have neither established a peer review group nor a management group to provide proper quality assurance.16 To address this systemic issue, the Evaluation section provided guidance for the development of the UNICEF Cambodia Standard Operating Procedures for Better Evaluation (see Annex 4). After being piloted in the Cambodia Country Office, those standard operating procedures (SOPs) were used and adapted by other country offices in the region (such as the Democratic People’s Republic of Korea, Malaysia, Mongolia and Myanmar). These SOPs can now be adapted to help country office management and staff ensure that evaluations are well planned and managed on time and on budget and that they produce credible, relevant and useful reports.

19. Country offices have often asked for help in professionalizing UNICEF and other UN staff through capacity development. Because staff competencies tend to vary and staff turnover is high,17 developing and facilitating specific training for UNICEF staff and other UN staff on the evaluation function’s core components has been the second-most frequent request. In response, capacity development sessions have been organized to develop UNICEF staff and partner staff capacities in the Democratic People’s Republic of Korea, the Lao People’s Democratic Republic, Mongolia, Myanmar, Papua New Guinee, Philippines and Viet Nam. Additionally, sessions on the new UNEG norms and standards, evaluability and evaluation management were facilitated at the joint UN Evaluation Development Group for Asia and the Pacific (UNEDAP) in the 2015 and 2016 training on “Evaluation in the UN Context”. Together with the Regional Office for South Asia and the Evaluation Office in New York, EAPRO organized joint evaluation network meetings in Kathmandu and Bangkok. These events contributed towards increasing staff capacity to manage and use evaluations as well as to ensure coherence with the evaluation function at the global level and with other UN agencies.18 In the future, country office PME staff and dedicated evaluation staff could support each other through peer reviews and training that would further contribute towards developing professional competencies.

16 Quality assurance for two final evaluation reports (on the Thailand Country Office’s National Child and Youth Development Plan and the Lao PDR WASH country programme) was requested three times for each. This shows that, even when a review team was set up, standardized procedures, quality assurance processes and mechanisms were not effectively working at the country office level. The regional evaluation adviser recommended these two country offices look to what extent the consultants’ team had addressed comments previously shared before sending the deliverables to the regional office. In the case of the Thailand Country Office evaluation, the regional evaluation adviser met the team leader and participated in the debriefing to provide direct advice.

17 Ian C. Davies and Julia Brummer: Final Report to the UNEG Working Group on Professionalization of Evaluation, Geneva, 2015; and UNEG: Evaluation Competency Framework, Geneva, 2016.

18 Beyond the previously noted trainings, in-house capacity development on evaluations is de-prioritized, with no country office learning plan prioritizing this critical function. This is partly because the previous regional evaluation adviser thoroughly supported country office capacity development needs.


20. Technical assistance represents the third most frequent request from country offices. As opposed to other country office requests, this is the most diverse in nature. Support has ranged from hands-on guidance on an after-action review of UNICEF’s emergency response to cyclone Pam to guidance on a country programme evaluation (Indonesia and Philippines) as well as on United Nations Partnership Development Framework (UNDAF) evaluations for Cambodia Fiji, Papua New Guinea and Viet Nam.

21. Technical assistance has been given on how to develop and prioritize evaluations in the costed evaluation plans in the Democratic People’s Republic of Korea, the Lao People’s Democratic Republic, Mongolia, Thailand, Viet Nam and Pacific island country offices. The Cambodia, Indonesia, Malaysia, Mongolia, Philippines and Viet Nam country offices have requested regional office support for their ‘evaluability’ assessments,19 although guidance is still under development. EAPRO has assisted in a review of the adequacy of the Indonesia country programme design and the availability of data and systems to carry out an evaluation as well as to understand whether stakeholders are on board to do an evaluation and whether they have sufficient resources available to do an evaluation. The regional office evaluability assessment support may trigger country office senior management buy-in for conducting more strategic evaluations at the outcomes level.

22. National evaluation and partner capacity development represents the fourth-most frequent request. Despite the strong emphasis that UNICEF places on developing national evaluation capacity, which includes not only strengthening the evaluation systems of national governments but also those of civil society partners, this demand is nascent. To accommodate the growing requests, partnerships with other UN agencies have been critical (mainly the United Nations Development Programme (UNDP) and UN Women) and development partners (Asian Development Bank, the World Bank and the Asia Pacific Evaluation Association) because it’s an area that is broader than UNICEF’s core mandate and priorities.

19 The OECD’s Development Assistance Committee defines an ‘evaluability’ assessment as “the extent to which an activity or a project can be evaluated in a credible fashion. Based on country office demand, the REA supports evaluability studies. These may enable UNICEF to save resources and correct the design flaws and to understand whether data and the environment is conducive before launching an evaluation.


Boys play on a frozen body of water, in the ‘soum’ (district) of Ulaan-Uul in the northern Khövsgöl ‘Aimag’ (province), Mongolia/2012© UNICEF/UNI134453/Sokol


Regional Evaluation Strategy23. The Regional Evaluation Strategy focuses on what the regional office and country offices can

do (their respective roles) to reinforce the evaluation function, especially the use of evaluation findings. The strategy provides guidance for conducting high-quality evaluations that inform senior management decision-making and respond to country office and regional office learning and accountability needs. The strategy aims to foster the credibility, use and quality of evaluations in a highly decentralized organization.

24. The strategy’s 2021 goal is to have an evaluation function that generates useful evidence that strategically informs policy, programming and advocacy and thus contributes towards better results for children. Of critical importance to the strategy is the involvement of children and young people throughout the evaluation activities across the region. Ultimately, the strategy envisions that evaluations will trigger transformational learning and adaptive management within the organization and among its partners.

iv. What does the region need to prioritize?20

25. To improve the evaluation function across the region, the strategy targets five strategic priorities: (a) prioritizing evaluation and embedding the function in the results-based management cycle;21 (b) strengthening the quality assurance system; (c) reinforcing the regional office and country office internal evaluation capacities; (d) supporting national evaluation capacity development; and (e) maintaining independence and fostering credibility and use of findings. Following through on these five priorities will nurture a stronger evaluation culture throughout UNICEF and among its core partners.

26. Prioritizing evaluation and embedding it in the results-based management cycle: When evaluations are better understood as a core component of results-based management, they will be better planned and of better quality and utility to UNICEF. When commissioning evaluations and conducting an evaluation, the regional office and country offices should have “a clear intention to use the resulting analysis, conclusions and recommendations to inform decisions and actions”.22

20 Regional offices, under the leadership of the regional director, provide regional leadership in (a) governance and accountability (especially in developing regional strategies and engaging senior management), (b) guidance and quality assurance, (c) conducting evaluations, (d) partnerships for evaluation, (e) development and professionalization of the UNICEF evaluation function and (f) national evaluation capacity development. For more details, see the UNICEF Evaluation Policy, p. 9.

21 By linking it more strongly to strategic positioning and planning.22 See norm 2 in UNEG: Norms and Standards for Evaluation, Geneva, 2016, .

http://www.unevaluation.org/document/detail/1914


27. The regional office and country offices must take a strategic approach to evaluations to ensure adequate coverage and a medium-term perspective in their respective costed evaluation plan. The Evaluation section can provide guidance and support towards improving country office evaluation planning, budgeting, implementation, dissemination and use of findings. It can also review the planned evaluations and evaluation priorities with representatives, deputy representatives and PME sections. It can help country offices articulate their evaluation scope and purpose (organizational learning and improvement, accountability, transparency and increased use for evidence-based advocacy and decision-making).

28. Strengthening quality assurance system: The EAPRO Evaluation section can assist country offices in designing, managing and monitoring the quality of evaluations against the UNEG norms and standards. Systems, such as SOPs (Annex 4), can be adapted by country offices and applied throughout all phases of their evaluations.

29. When needed, the EAPRO Evaluation section can also help clarify roles and responsibilities of the country offices, the regional office and headquarters: who is accountable for the evaluation function and who manages them. Country offices need to identify adequate financial and human resources and procedures to ensure that evaluation quality and use of findings, conclusions and recommendations meet the minimum standards.

30. Reinforcing UNICEF staff capacity: When the capacity of PME and programme staff to manage and quality assure evaluations is weak, the EAPRO Evaluation section can support the recruitment of evaluation specialists or managers and support the regional office and country office capacity development initiatives. Training and coaching of staff are provided as per country office requests.

31. Internally: Qualified national and international resources are to be recruited to dedicate appropriate time to implement the evaluation function. When budget constraints are present or the volume of planned individual country evaluations is likely to increase, a shared evaluation specialist post could be an option that neighbouring countries consider. The regional Evaluation section can support country offices’ (i) recruitment processes by participating in interview panels, (ii) coaching staff and extending other capacity-building activities. Country offices can use the Human Resource Development Plan, including capacity building for national staff.

32. Externally: To carry out evaluations, the regional office and country offices contract qualified independent evaluators, supported by sector experts when needed. The Evaluation section can provide an up-to-date quality-controlled roster of external evaluators and firms that carry out high-quality evaluations. The market for the international development evaluation suppliers in the region is recognized as underdeveloped. When engaging local or regional suppliers, UNICEF country office staff should make sure they are aware of the UNEG evaluation standards and expectations on all evaluation products.


33. Supporting national evaluation capacity development: In the post-2015 development priorities, UNICEF will further contribute toward improving national capacity to conduct country-led evaluations. Country offices, with the support of the regional office, can identify supply and demand as well as partners and priority actions for national evaluation capacity development. The strengthening of national capacities should involve working with other UN agencies, bilateral donors, government ministries (such as planning and finance) and universities. Country office road maps will need to be established.

34. Maintaining independence credibility and use of evaluations: Independence of evaluation activities is necessary for their credibility, which in turn underpins the use of evaluation findings, conclusions and recommendations. It allows evaluators to be impartial and free from undue pressure. As outlined in norm 4 of the UNEG, “The independence of the evaluation function comprises two key aspects—behavioural independence and organizational independence.” Considering the highly decentralized nature of the evaluation function, it is critical to preserve this degree of independence by separating the roles and responsibilities for the evaluation function within the country office. Those responsible for the evaluation function should report directly to the country representative. This arrangement mirrors the regional office set-up, with the evaluation advisor reporting to the regional director.

35. Considering that conducting evaluations represents a growing investment in the region, intentionality and use of findings, conclusions and recommendations are critical to consider throughout the evaluation cycle. Active involvement of stakeholders helps to boost their ownership and trigger learning with the organization and among external stakeholders, including children, youth, civil society, government and donors.

v. How? The way forward.

36. To strengthen the evaluation function at the regional office and in country offices, political leadership and adequate funding are needed, together with clear norms, mechanisms and expectations.

37. All stakeholders in the regional and country offices need to corporately prioritize evaluations. Country representatives and deputy representatives have relevant evaluation targets in their own plans and performance reviews. Large country offices should conduct five evaluations per country programme cycle, while medium-sized and smaller country offices should carry out at least three evaluations over the same period.23 The regional office has committed to at least two evaluations in the new Regional Office Management Plan.

23 In the East Asia and Pacific region, large country offices have more than $12 million in operational resources per year, while medium-sized and small country offices have less than 12 million OR. Country offices with more than $20 million in operational resources should allocate 3 per cent of their budget to the evaluation function.


38. The Evaluation section can help EAPRO and country offices to plan and budget their evaluations when the CPD and costed evaluation plan are being developed. The region’s combined costed evaluation plans should allocate an average of 1 per cent of programme expenditure to the evaluation process. By allocating adequate human and financial resources and setting up effective management and governance structures, the independence and the impartiality of the evaluation function can be preserved. Each evaluation report will be supplemented with a management response that will be implemented.

39. Evidence from recent evaluations should be systematically incorporated into the new CPD. Knowledge management initiatives, such as the Strategic Moments of Reflection, the Annual Synthesis, the Evaluate newsletter, the UN Evaluation day and joint network meetings, are to be taken forward to support the dissemination and adoption of evaluation lessons. Joint regional network meetings are to be arranged every 18 months.

40. The regional and country offices need to develop a wider learning agenda and establish peer learning groups. To help fulfil existing knowledge gaps on emerging evaluative practices (evaluability assessments, developmental evaluations and national evaluation capacity development), good practices on evaluations that make a difference for children in the region must be regularly documented.

Integrating evaluation with results-based management

41. Evaluation findings, conclusions and recommendations are to be used for organizational learning, informed decision-making and accountability. A mechanism should be established to ensure that strategic decisions (such as reorienting the position of the country office or the country programme) at the regional and country office levels require evidence from evaluations of past interventions

42. Evaluation needs are to be more explicitly embedded in results-based management, with emphasis on CPD evaluability assessments, through theories of change, well-defined results, SMART indicators and the consistent establishment of baselines and monitoring systems. Evaluability assessments could improve UNICEF’s understanding of the adequacy of the programme design perspective, the availability of data and information to carry out an evaluation and guidance on possible approaches to evaluations. Once headquarters finalize the guidance, the regional office will share it and a checklist on evaluability with country offices.

43. Country offices should allocate adequate time and resources to planning and managing evaluations. Staff with relevant skill sets must manage and provide adequate guidance to consultants. Considering the representatives’ accountability for the evaluation function at the country level, they should allocate commensurate resources that are in line with the host government’s evaluation capacity and the size of the country programme. Country offices should include the evaluation function in the job description of their PME staff to report to the country representative.


44. All EAPRO and country office evaluations are to be adequately managed and quality checked against the UNEG norms and standards. The quality assurance mechanism is to be strengthened to ensure that there is improved planning, implementation, use, dissemination and monitoring of the uptake of evaluation results, both at the regional office and country office level. Country offices should adapt and incorporate the SOPs for better evaluations (Annex 4) that were trialled in South-East Asia. Considering the number of evaluations, the EAPRO Evaluation section will prioritize the most strategic evaluations, based on relevance and budget and ask country offices to start progressively setting aside a proportion of their funding to use existing CEECIS, MENA and ROSA long-term agreement24.

45. Additionally, indicators to determine the use of evaluation findings for advocacy purposes are to be defined (explicit inputs for into programming and decision-making) and captured. Country offices and national partners should follow ethics review standards and procedures when conducting research, studies and evaluations. Once the research strategy is completed, an ethical board should be established to review evaluations and research results.

46. Within the East Asia and Pacific region, UNICEF will give attention to its internal capacity development as well as the capacity development needs of UN agencies and other partners. UNICEF capacity development is based on country office demands and met through available online training resources as well as specific in country activities. UN agencies capacity development will continue through joint UN training in Asia and the Pacific with additional focus on UNDAF evaluations. The government capacity development is described further on. UN system-wide support is required for national evaluation capacity development.

47. UNICEF will support rigorous and evidence-based country-led evaluations by helping to strengthen national data systems and national evaluation capacity. UNICEF (together with other UN agencies), in accordance with General Assembly resolution A/RES/69/237 on capacity building for the evaluation of development activities at the country level, will support “upon request efforts to further strengthen the capacity of Member States for evaluation, in accordance with their national policies and priorities”. UN agencies should work towards a common national evaluation capacity objective and should apply a systemic and synergistic approach to assisting countries. In each country, UNICEF, together with the UN system, should identify each agency’s comparative advantage; we can then leverage that advantage to maximize results. By partnering with other UN agencies and development actors, national evaluation capacities will be strengthened through the mapping of existing development partners’ supply and national government demand to reinforce their evaluation function.

24 Other regional offices outsource quality assurance to private companies and universities through global and regional long-term agreements. In EAPRO, financial resources are not currently available for this function; rather, it is being implemented by country offices and the regional evaluation adviser. Indicatively, country offices should set aside 1–5 per cent of budget resources for evaluations and knowledge generation, including quality assurance.


48. Country offices should participate in diagnostic studies and stakeholder mapping to identify actors and entry points. EAPRO, the country offices and UNICEF headquarters can support member States and partners to mainstream evaluation through:

· awareness raising and advocacy;

· knowledge sharing of existing good practices and policies (in Malaysia, the Philippines and Thailand, where national evaluation policies have been developed with UNICEF support);

· capacity development; and

· evaluation action plan development.

49. The 2030 Agenda for Sustainable Development and aid effectiveness reform promote national ownership, alignment as well as evidence-based decision-making. National evaluation strategies could be developed as per country office demands. To keep track, UNICEF as well as others should report on its implementation.

50. During the first quarter of 2017, UNICEF EAPRO and UNDP Asia-Pacific regional Office decided to identify emerging national evaluation capacity development practices in the region by jointly launching a series of country case studies. The initial phase will include five country case studies—Indonesia, Malaysia, Thailand, Sri Lanka and either Nepal or Philippines—with further country case studies to be initiated over the course of 2018. A regional synthesis report on emerging good practices will be developed, based on the country case studies. This will serve to showcase existing national evaluation champions and emerging country practices in the region, distil key success factors, trends and lessons learned. Participating in this study will help UNICEF country offices understand what national evaluation capacity there is in terms of infrastructure and with which strategic partners UNICEF could further work. This may also foster South–South cooperation.

51. Monitoring and review. Progress on the Regional Evaluation Strategy and action plan will be reviewed every two years by the RMT. The action plan will be monitored on an annual basis by the regional evaluation section, and progress reports will be provided to the Regional Director and the RMT.


A girl washes her hands at a UNICEF-provided water point at a new elementary school, built with UNICEF assistance, in the village of Neusok Teubaluy in the district of Aceh Besar, Indonesia/2007© UNICEF/UNI48741/Estey


Action plan

Process:The implementation plan was drafted in late 2016 and incorporated two rounds of country office comments. The plan was then presented and validated at the Joint EAPRO ROSA Evaluation Network meeting in March 2017 and then endorsed at the RMT in April 2017. The Theory of Change below was subsequently developed.

Impact statementEvaluations make a difference in children’s lives

Outcome statement The UNICEF East Asia and Pacific Regional Office and country office evaluation function is corporately prioritized and strengthened.

Intermediary outcomesThe evaluation function contributes to UNICEF’s organizational learning, informed decision-making and accountability for results.

Quality, credibility and utility of the evaluations are improved through better planning, implementation, quality assurance, dissemination and use of evaluations as well as to staff and partners’ capacity development.

To achieve these outcomes, a series of outputs and a set of actions that, respectively, the regional office and country offices should prioritize. Because these are process components, some specific indicators are proposed. Yet, overall progress and performance will be measured against global key performance indicators, as reported in the global dashboard.

Specific outputs These are directly linked with the strategy priorities.

· Evaluation function is systematically embedded in UNICEF’s results-based management. · Evaluations are planned with an annual and multiple-year horizons. · The evaluation function at the regional office and the country offices is adequately resourced.· All EAPRO and country office evaluations are adequately managed and quality assured

against the UNEG Norms and Standards.· EAPRO and country offices actively foster evaluation use.· EAPRO and country offices prioritize national evaluation capacity development.· EAPRO and country offices strategically position UNICEF with regional United Nations

interagency evaluations.· EAPRO regularly interacts with the Evaluation Office.


Fig

ure

1: T

heo

ry o

f C

han

ge

on

ho

w t

o s

tren

gth

en t

he

UN

ICE

F ev

alu

atio

n f

un

ctio

n in

th

e E

ast

Asi

a an

d t

he

Pac

ific

reg

ion

Sp

ecifi

c o

utp

uts

Inte

rmed

iary

o

utc

om

es

Ou

tco

me

stat

emen

t

Imp

act

stat

emen

t

Sys

tem

atic

ally

em

bed

ded

in

UN

ICE

F’s

resu

lts-

bas

ed

man

agem

ent

Ad

equ

atel

y re

sou

rced

An

nu

al a

nd

m

ult

iple

-yea

r h

ori

zon

s

Ch

ildre

n, d

uty

b

eare

rs &

ri

gh

ts h

old

ers

invo

lved

Act

ivel

y fo

ster

use

of

eval

uat

ion

fi

nd

ing

s

Pri

ori

tize

n

atio

nal

ev

alu

atio

n

cap

acit

y d

evel

op

men

t

Reg

ula

rly

inte

ract

wit

h

the

Eva

luat

ion

O

ffice

UN

DA

F ev

alu

atio

n

qu

alit

y as

sura

nce

Ad

equ

atel

y m

anag

ed

& q

ual

ity

assu

red

Eva

luat

ion

s m

ake

a d

iffe

ren

ce in

ch

idre

n’s

live

s

By

2021

, UN

ICE

F ev

alu

atio

n f

un

ctio

n is

co

rpo

rate

ly p

rio

riti

zed

and

str

eng

then

ed a

t th

e E

AP

RO

an

d c

ou

ntr

y o

ffice

leve

ls

Co

ntr

ibu

tes

to o

rgan

izat

ion

al le

arn

ing

, in

form

ed d

ecis

ion

-mak

ing

an

d

acco

un

tab

ility

fo

r re

sult

s

Qu

alit

y, c

red

ibili

ty a

nd

uti

lity

of

the

eval

uat

ion

s ar

e im

pro

ved

Sup

port

cou

ntry

-led

eval

uatio

n an

d

stra

tegi

cally

pos

ition

UN

ICEF

with

re

gion

al U

nite

d N

atio

ns in

tera

genc

y ev

alua

tions


Co

un

try

offi

ce le

vel

Reg

ion

al o

ffice

leve

lIn

dic

ato

rs (

incl

. fr

equ

ency

of

rep

ort

ing

Mea

ns

of

veri

fica

tio

n

Bas

elin

e (r

ef. y

ear

and

so

urc

e)Ta

rget

(re

f. y

ear

and

so

urc

e)

1.

Eva

luat

ion

fu

nct

ion

is s

yste

mat

ical

ly e

mb

edd

ed in

UN

ICE

F’s

resu

lts-

bas

ed m

anag

emen

tC

Os

new

CP

Ds

and

P

SN

s sy

stem

atic

ally

use

ev

iden

ce f

rom

eva

luat

ion

s

RO

sha

res

a co

mpi

latio

n of

less

ons

lear

ned

and

reco

mm

enda

tions

(201

4-20

16)

and

revi

ews

thei

r in

tegr

atio

n in

th

e C

PD

s an

d P

SN

s, S

itAns

Trac

king

of

eval

uativ

e ev

iden

ce u

se in

the

C

PD

s an

d P

SN

s de

velo

pmen

t pr

oces

s (t

hrou

gh a

tem

plat

e to

be

deve

lope

d by

th

e R

O).

CO

CP

Ds,

PS

Ns

and

coun

try

prog

ram

me

resu

lts

fram

ewor

ks a

nd t

rack

ing

repo

rt

Them

atic

eva

luat

ion

reco

mm

enda

tions

inco

rpor

ated

in

the

new

glo

bal s

trat

egy

(e.g

. H

IV) a

nd r

egio

nal s

trat

egie

s (e

.g.

regi

onal

nut

ritio

n st

rate

gy o

r C

4D

stra

tegy

) and

in

4 C

Os

(Cam

bodi

a, C

hina

, In

done

sia

and

Mal

aysi

a) in

201

6 (C

PD

s an

d P

SN

s)

Glo

bal t

hem

atic

ev

alua

tions

fee

d in

to n

ew

RO

MP

Less

ons

lear

ned

and

reco

mm

enda

tions

fr

om e

valu

atio

ns a

re

inco

rpor

ated

in a

ll C

Os

new

CP

Ds,

PS

Ns

and

new

RO

MP

by

2021

CP

D o

utc

om

es a

re

eval

uat

ion

rea

dy

(eva

luab

ility

)

RO

MP

pro

gram

me

outc

omes

ar

e ev

alua

tion

read

y

RO

val

idat

es e

valu

abili

ty o

f C

PD

s

CP

Ds

and

RO

MP

are

be

tter

des

igne

d an

d ev

alua

tion

read

y

All

CO

s an

d th

e R

O

cond

uct

an e

valu

abili

ty

asse

ssm

ent

of C

PD

/R

OM

P t

o be

com

e ev

alua

tion

read

y by

202

1

Whe

n re

ady,

RO

sha

res

HQ

gui

delin

es a

nd p

rovi

des

com

men

ts a

nd t

echn

ical

as

sist

ance

on

eval

uabi

lity

asse

ssm

ent

of C

P a

nd p

ilot

initi

ativ

es

# of

CP

D e

valu

abili

ty

asse

ssm

ents

# of

and

use

of:

-in

depe

nden

t ev

alua

bilit

y as

sess

men

t re

port

-R

O e

valu

abili

ty

asse

ssm

ent-

supp

ort

mis

sion

s

2/14

CO

s : I

ndon

esia

in 2

015

(RO

ev

alua

bilit

y as

sess

men

t-su

ppor

t m

issi

on),

Mal

aysi

a in

201

6 (C

PD

ev

alua

bilit

y as

sess

men

t re

port

)

(eva

luab

ility

ass

essm

ents

pl

anne

d in

201

7) 4

CO

s (M

ongo

lia, T

haila

nd, D

emoc

ratic

P

eopl

e’s

Rep

ublic

of

Kor

ea a

nd

Phi

lippi

nes)

, in

2017

(upc

omin

g ev

alua

bilit

y re

port

)

N/A

to

curr

ent

RO

MP

201

4-20

17

1 R

OM

P e

valu

abili

ty

asse

ssm

ent

by 2

018

https://www.unicef.org/about/execboard/files/2015-PL21-Cambodia_CPD-ODS-EN.pdfhttps://www.unicef.org/about/execboard/files/2015-PL22-China_CPD-ODS-EN.pdfhttps://www.unicef.org/about/execboard/files/2015-PL23-Indonesia_CPD-ODS-EN.pdfhttps://www.unicef.org/about/execboard/files/2015-PL24-Malaysia_CPD-ODS-EN.pdfhttps://icon.unicef.org/apps02/cop/edb/Lists/Evaluation Reports/Attachments/7674/EA Malaysia Final Report (31 August 2016).pdfhttps://icon.unicef.org/apps02/cop/edb/Lists/Evaluation Reports/Attachments/7674/EA Malaysia Final Report (31 August 2016).pdf


Co

un

try

offi

ce le

vel

Reg

ion

al o

ffice

leve

lIn

dic

ato

rs (

incl

. fr

equ

ency

of

rep

ort

ing

Mea

ns

of

veri

fica

tio

n

Bas

elin

e (r

ef. y

ear

and

so

urc

e)Ta

rget

(re

f. y

ear

and

so

urc

e)

Co

nd

uct

per

iod

ic

trai

nin

g t

o s

tren

gth

en

CO

pro

gra

mm

e st

aff

mem

ber

s’ u

nd

erst

and

ing

o

f th

e ro

le o

f ev

alu

atio

n

and

its

pla

ce in

res

ult

s-b

ased

man

agem

ent

(RB

M)

(ori

enta

tio

n o

f ev

alu

atio

n a

nd

ro

le o

f ev

alu

atio

n in

pla

nn

ing

an

d

imp

lem

enta

tio

n)

Con

duct

per

iodi

c tr

aini

ng t

o st

reng

then

RO

pro

gram

me

staf

f m

embe

rs’ u

nder

stan

ding

of

the

role

of

eval

uatio

n an

d its

pla

ce in

R

BM

(orie

ntat

ion

of e

valu

atio

n an

d ro

le o

f ev

alua

tion

in p

lann

ing

and

impl

emen

tatio

n)

# of

eva

luab

ility

, ev

alua

tion

and

RB

M

trai

ning

s fa

cilit

ated

by

RO

and

CO

s

Cos

and

RO

tra

inin

g re

port

s on

eva

luab

ility

ev

alua

tion

and

RB

M

6 ev

alua

bilit

y as

sess

men

t tr

aini

ngs

(3 C

Os:

Indo

nesi

a,

Phi

lippi

nes,

Vie

t N

am, 1

RO

and

1

trai

ning

in r

egio

nal n

etw

ork

mee

ting

in 2

016)

, 7 e

valu

atio

n ca

paci

ty d

evel

opm

ent

trai

ning

(2

015:

3 C

Os,

1 jo

int

UN

; 201

6: 2

C

Os,

1 R

O, a

nd 1

join

t U

N 2

017:

1

CO

and

1 jo

int

UN

) and

4 R

BM

tr

aini

ngs

(201

6: 1

CO

s, 3

RO

, tr

aini

ngs)

To b

e de

fined

tog

ethe

r w

ith t

he R

O p

lann

ing

unit

2.

Eva

luat

ion

s ar

e p

lan

ned

wit

h a

n a

nn

ual

an

d m

ult

i-an

nu

al h

ori

zon

:A

ll C

Os

eval

uat

ion

s ar

e p

lan

ned

, des

ign

ed a

nd

b

ud

get

ed in

th

e co

un

try

pro

gra

mm

e (C

P)

in t

he

cost

ed e

valu

atio

n p

lan

(C

EP

) an

d IM

EP

or

PR

IME

RO

rev

iew

s C

O c

oste

d ev

alua

tion

plan

s an

d C

PD

and

R

RF

budg

ets

# an

d %

of

CO

s w

ith a

for

mal

cos

ted

eval

uatio

n pl

an a

nd

IME

P o

r P

RIM

E

CE

Ps

and

IME

Ps

or P

RIM

E

Indi

catio

n of

pla

nned

ev

alua

tion

in t

he n

ew

RO

MP

11 C

Os

(Cam

bodi

a, C

hina

, DP

R

Kor

ea, I

ndon

esia

, Lao

PD

R,

Mal

aysi

a, M

ongo

lia, V

iet

Nam

, Th

aila

nd, M

yanm

ar a

nd t

he

Phi

lippi

nes)

hav

e co

mpl

eted

a

cost

ed e

valu

atio

n pl

an a

s pa

rt o

f th

eir

CP

D in

201

7

All

CO

CP

hav

e a

cost

ed e

valu

atio

n pl

an

by 2

019

and

upda

ted

PR

IME

or

IME

Ps

draw

n on

Eva

luat

ion

Pol

icy

pred

efine

d pr

iorit

izat

ion

crite

ria25

and

eng

agem

ent

with

CO

sen

ior

lead

ersh

ip

The

RO

MP

incl

udes

a r

egio

nal

cost

ed e

valu

atio

n pl

an,

spec

ifyin

g m

ulti-

coun

try

and

them

atic

eva

luat

ion

to b

e co

nduc

ted

Exi

sten

ce o

f a

form

al

eval

uatio

n pl

an a

s pa

rt o

f th

e ne

w

RO

MP

Rep

orts

of

RO

sup

port

to

CO

req

uest

sC

oste

d ev

alua

tion

plan

: N/A

to

curr

ent

RO

MP

201

4-20

17A

ll 14

CO

s an

d th

e R

O

have

a c

oste

d ev

alua

tion

plan

by

2018

RO

tim

ely

and

qual

itativ

e ad

vice

on

CO

eva

luat

ion

plan

ning

and

bu

dget

ing

Per

cept

ion

surv

ey

25

Acco

rdin

g to

the

Polic

y, th

e CO

nee

ds to

ens

ure

eval

uatio

n is

und

erta

ken:

bef

ore

prog

ram

me

repl

icat

ion

or s

calin

g up

(e.g

., pi

lot i

nitia

tives

); w

hen

resp

ondi

ng to

maj

or e

mer

genc

ies;

follo

win

g lo

ng p

erio

d of

une

valu

ated

pro

gram

me

impl

emen

tatio

n, e

spec

ially

whe

re th

e pr

ogra

mm

e ha

s be

en im

plem

ente

d fo

r at l

east

five

yea

rs w

ithou

t any

eva

luat

ion

activ

ity; f

or e

ach

outc

ome,

whe

n ex

pend

iture

has

reac

hed

US$1

0 m

illio

n; fo

r eac

h ou

tcom

e, w

here

the

aver

age

annu

al

expe

nditu

re e

xcee

ds U

S$1

mill

ion


Co

un

try

offi

ce le

vel

Reg

ion

al o

ffice

leve

lIn

dic

ato

rs (

incl

. fr

equ

ency

of

rep

ort

ing

Mea

ns

of

veri

fica

tio

n

Bas

elin

e (r

ef. y

ear

and

so

urc

e)Ta

rget

(re

f. y

ear

and

so

urc

e)

CM

Ts o

r P

MT

ap

pro

ve

eval

uat

ion

pla

ns

bas

ed o

n

stra

teg

ic a

lign

men

t an

d

reso

urc

es, r

egu

larl

y re

view

th

eir

imp

lem

enta

tio

n

PM

T an

d R

MT

appr

ove

and

regu

larly

rev

iew

reg

iona

l ev

alua

tion

plan

s.

RO

sha

res

less

ons

lear

ned

and

good

pra

ctic

es in

CE

P

deve

lopm

ent

with

CO

s

# of

CO

str

ateg

ic

eval

uatio

ns

# of

mul

ti-co

untr

y st

rate

gic

eval

uatio

ns

in t

he r

egio

n

Min

utes

of

the

CM

Ts,

PM

Ts a

nd R

MTs

CP

DS

, CE

Ps

and

IME

Ps

or

PR

IME

Eva

luat

ion

is a

n ag

enda

item

in:

-4/1

4 C

Os

CM

Ts (m

inut

es)

-PM

T ite

m o

nce

a ye

ar (m

inut

es)

-RM

T ite

m (i

n 20

15 a

nd 2

017,

m

inut

es)

Eva

luat

ion

is a

CM

T or

P

MT

agen

da it

em in

all

CO

s an

d in

one

PM

T an

d R

MT

mee

ting

per

year

A

ll C

O a

nd R

Os

plan

ned

eval

uatio

ns a

re

stra

tegi

cally

cho

sen

by

2019

RO

rev

iew

of

CP

and

C

EP

eff

ectiv

enes

s on

pro

gram

me

impl

emen

tatio

n

RO

MP

and

reg

iona

l ev

alua

tion

repo

rts

repo

rts

of R

O s

uppo

rt t

o C

Os

CM

T e

nsu

res

that

all

CP,

al

l pilo

t p

roje

cts

and

all

pro

gra

mm

e co

mp

on

ents

th

at f

ulfi

l eva

luat

ion

p

rio

riti

zati

on

cri

teri

a se

t in

th

e E

valu

atio

n P

olic

y ar

e ei

ther

rev

iew

ed o

r ev

alu

ated

PM

T en

sure

s th

at e

nsur

es t

hat

RO

MP

and

sec

tor

prog

ram

mat

ic

area

s th

at f

ulfil

eva

luat

ion

prio

ritiz

atio

n cr

iteria

set

in t

he

Eva

luat

ion

Pol

icy

are

eval

uate

d

# of

pro

gram

me

com

pone

nts

eval

uate

d in

the

•

CP

cyc

le p

er C

O•

RO

MP

cycl

e pe

r RO

GE

RO

S a

nd P

RIM

E o

r IM

EP

sA

vera

ge 2

pro

gram

me

com

pone

nts

eval

uate

d pe

r cy

cle

at t

he e

nd o

f 20

16 (G

ER

OS

)

N/A

for

RO

in R

OM

P c

ycle

20

14–2

017

Eas

t A

sia

and

Pac

ific

regi

on d

oes

at le

ast

8 ev

alua

tions

per

yea

r

In la

rge

CO

s, a

t le

ast

five

prog

ram

me

com

pone

nts

are

eval

uate

d in

the

co

urse

of

the

prog

ram

me

cycl

e

In m

ediu

m a

nd s

mal

l CO

s,

at le

ast

thre

e co

mpo

nent

s ar

e ev

alua

ted

At

leas

t tw

o co

re

com

pone

nts

of t

he R

OM

P

are

eval

uate

d by

202

1

3.

Th

e ev

alu

atio

n f

un

ctio

n a

t co

un

try

offi

ce a

nd

reg

ion

al o

ffice

leve

l is

adeq

uat

ely

reso

urc

ed26

Rep

s en

sure

th

at d

edic

ated

h

um

an r

eso

urc

es f

or

eval

uat

ion

are

in p

lace

to

su

pp

ort

th

e ev

alu

atio

n

fun

ctio

n (

CO

)

RD

and

DR

D e

nsur

e th

at

RO

eva

luat

ion

sect

ion

is

adeq

uate

ly r

esou

rced

to

supp

ort

the

eval

uatio

n fu

nctio

n an

d im

plem

ent

regi

onal

str

ateg

y an

d su

ppor

t th

e E

ast

Asi

a an

d P

acifi

c C

Os

# of

CO

s w

ith

adeq

uate

Hum

an

reso

urce

s to

sup

port

ev

alua

tion

func

tions

co

nfirm

ing

requ

isite

st

anda

rds

(CO

and

R

O)(a

nnua

lly)2

7

CO

and

RO

org

anog

ram

s an

d JD

s

EA

PR

O M

&E

foc

al

poin

t lis

t th

at is

upd

ated

qu

arte

rly

3 C

Os

(Cam

bodi

a, M

yanm

ar,

Mal

aysi

a) h

ad o

ne e

valu

atio

n sp

ecia

list

shar

ed p

ost,

9 C

Os

with

PM

E o

r M

&E

spe

cial

ists

, 2

CO

s w

ith M

&E

offi

cers

and

in 1

C

O p

ost

rela

ted

to e

valu

atio

n is

un

der

recr

uitm

ent

at t

he e

nd o

f M

arch

201

7

All

CO

s ha

ve a

dequ

ate

hum

an r

esou

rces

to

supp

ort

eval

uatio

n fu

nctio

ns c

onfir

min

g re

quis

ite s

tand

ards

(CO

an

d R

O)2

8

RO

reg

iona

l eva

luat

ion

advi

ser

P5

and

eval

uatio

n of

ficer

JP

O o

r N

OB

26

The

tota

l cos

t of t

he e

valu

atio

n de

pend

s on

the

com

plex

ity o

f the

pro

gram

me,

the

eval

uatio

n ty

pe, d

esig

n an

d ex

tern

al e

xper

tise

requ

ired.

The

Pol

icy

allo

cate

s a

min

imum

of 1

to 3

per

cen

t of o

vera

ll pr

ogra

mm

e ex

pend

iture

to e

valu

atio

n. 2

7 M

ay in

itial

ly v

ary

acco

rdin

g to

mul

tiple

fact

ors

incl

udin

g si

ze o

f the

pro

gram

me,

ava

ilabi

lity

of R

R an

d OR

28

Hum

an re

sour

ce re

view

s al

tern

ativ

es, i

nclu

ding

sha

red

eval

uatio

n sp

ecia

lists

pos

t opt

ion

acro

ss th

e re

gion

bef

ore

scal

ing

up, t

o be

dis

cuss

ed a

t the

RM

T.


Co

un

try

offi

ce le

vel

Reg

ion

al o

ffice

leve

lIn

dic

ato

rs (

incl

. fr

equ

ency

of

rep

ort

ing

Mea

ns

of

veri

fica

tio

n

Bas

elin

e (r

ef. y

ear

and

so

urc

e)Ta

rget

(re

f. y

ear

and

so

urc

e)

Rep

s en

sure

th

at r

equ

isit

e

fin

anci

al r

eso

urc

es a

re

ded

icat

ed f

or

eval

uat

ion

to

su

pp

ort

th

e ev

alu

atio

n

fun

ctio

n

RD

and

DR

D e

nsu

re t

hat

re

qu

isit

e fi

nan

cial

res

ou

rces

ar

e d

edic

ated

fo

r ev

alu

atio

n

to s

up

po

rt t

he

eval

uat

ion

fu

nct

ion

an

d im

plem

ent

regi

onal

str

ateg

y an

d su

ppor

t th

e E

ast

Asi

a an

d P

acifi

c C

Os

# of

CO

s us

ing

1%

of t

heir

budg

et f

or

eval

uatio

n an

nual

ly

CO

ann

ual r

epor

t an

d R

O

annu

al r

epor

t

Vis

ion

and

glob

al

dash

boar

d

6 C

Os

used

mor

e th

an 1

% o

f th

eir

budg

et f

or e

valu

atio

n w

hile

R

O u

sed

0.2%

in 2

016

(glo

bal d

ashb

oard

)

As

an a

ggre

gate

, the

Eas

t A

sia

and

Pac

ific

regi

on r

each

ed 0

.7%

in

201

5 an

d 1.

7% o

f its

tot

al b

udge

t ex

pend

iture

for

eva

luat

ion

func

tion

at t

he e

nd o

f 20

16 (S

AP

)

The

Eas

t A

sia

and

Pac

ific

regi

on, i

nclu

ding

CO

and

R

O a

s an

agg

rega

te, u

se

at le

ast

1% o

f an

nual

bu

dget

for

eva

luat

ion

func

tion

on a

n an

nual

ba

sis

All

CO

s an

d R

O u

se a

t le

ast

1% o

f ea

ch a

nnua

l bu

dget

for

eva

luat

ion

by

2021

RO

s us

ed 1

% o

f th

e C

O b

udge

t fo

r ev

alua

tion

annu

ally

29

All

Eas

t A

sia

and

Pac

ific

CO

sen

ior

man

agem

ent

and

PM

E s

taff

or

M&

E s

taff

jo

b d

escr

ipti

on

s in

clu

de

the

eval

uat

ion

fu

nct

ion

.

Whe

n ne

eded

the

RO

can

su

ppor

t C

Os

to r

evie

w jo

b de

scrip

tions

% o

f P

ME

and

M&

E

staf

f jo

b de

scrip

tions

in

clud

e ke

y ta

sks

and

resp

onsi

bilit

ies

to s

uppo

rt t

he C

O

eval

uatio

n fu

nctio

n.

Rev

ised

JD

sN

/AA

ll JD

s of

Sen

ior

man

agem

ent,

PM

E s

taff

or

M&

E s

taff

incl

ude

the

eval

uatio

n fu

nctio

n(20

19)

4.

All

Eas

t A

sia

and

Pac

ific

cou

ntr

y o

ffice

an

d r

egio

nal

offi

ce e

valu

atio

ns

are

adeq

uat

ely

man

aged

an

d q

ual

ity

assu

red

ag

ain

st t

he

UN

EG

No

rms

and

Sta

nd

ard

s.C

Os

dev

elo

p (

An

nex

4)

or

revi

ew t

hei

r ev

alu

atio

n

SO

P

Rol

l out

sta

ndar

d op

erat

iona

l gu

idel

ines

and

pro

cedu

res

for

bett

er e

valu

atio

n pl

anni

ng,

impl

emen

tatio

n, u

se a

nd

diss

emin

atio

n (a

dapt

atio

n of

S

OP

s de

velo

ped

in C

ambo

dia

in

2015

see

Ann

ex 4

)

# of

CO

dev

elop

ing

or a

dapt

ing

eval

uatio

n S

OP

s

% o

f C

O S

OP

s re

view

ed b

y R

O

Act

ual S

OP

sha

red

with

R

O

Documents

Regional Evaluation Strategy 2018-2021 UNICEF East Asia ... › eap › sites › unicef.org.eap...UNEDAP United Nations Evaluation Development for Asia and the Pacific UNEG United