UNICEF Child Protection Meta-Evaluation › french › videoaudio › PDFs › Final_CP... · 2019-11-12 · UNICEF Child Protection Meta-Evaluation Final 15 May 2008 Commissioned

UNICEF Child Protection Meta-Evaluation

Final 15 May 2008

Commissioned by Child Protection Section, Programme Division

United Nations Children’s Fund, New York USA 3 UN Plaza, NY New York 10017

Focal Points

Ms. Shirin Nayernouri Child Protection Specialist, Child Protection Section

Telephone: 212-326-7680 Facsimile: 212-735-4413

E-mail: [email protected]

Mr. Sam Bickel Senior Advisor, Evaluation Office

Telephone: 212-326-7504 Facsimile: 212-824-6492

E-mail: [email protected]

Evaluator Anne Sheeran, Ph.D., M.R.P.

Sheeran Consulting, Inc 4203 South Findlay Street

Seattle, WA 98118 USA E-mail: [email protected]

Tel/Fax (206) 322 0817 PST

Sheeran Consulting, Inc.

mailto:[email protected]

Sheeran Consulting, Inc. 2008 UNICEF Child Protection Meta-Evaluation Final Report, 15 May 2008 ii

Executive Summary Background Child Protection (CP) is one of five core UNICEF programming focus areas. Its mandate, to protect children from violence, exploitation and abuse, directs CP to address issues including violence against children, child labor, juvenile justice reform, trafficking, discrimination, support to children deprived of parental care, and child protection in emergencies. 1. The long range aim of the UNICEF CP strategy is to facilitate systemic changes in national

and societal support for protective laws, polices and social conventions. The blueprint for this is the “Protective Environment Framework” (PEF), a web of eight interconnected programming areas that have steered CP investments since 2002.

Purpose/Objectives of the Meta-evaluation 2. The purpose of the meta-evaluation is to take the measure of how the PEF is understood

and addressed in the field by UNICEF and others. The primary objective is to provide UNICEF CP with an authoritative evidence base for global lessons sharing with partners and for global, regional, and national programming and advocacy efforts. An important subsidiary objective is to explore knowledge management software options to help index evidence based good practices.

Research Methods and Constraints 3. The meta-evaluation analyzes 59 CP evaluation reports that were produced between 2002

and 2007, or 40% of all CP evaluations in that period. Selection was based upon availability and language (English). Scales were established to assign values to characteristics of interest addressing programming quality, evidence quality, and PEF outputs and outcomes.

4. Each report’s catalogue of PEF investments was coded for delivery into the askSam database, which allows text searching of materials from a variety of document types.

5. Limitations: in most cases, the depth of the database for identifying best practices is constrained by (a) insufficient analysis of institutional contexts (e.g. donor requirements, host government sensitivities), (b) scarce time series information (e.g. 64% of the reports have no longitudinal data), and (c) an alarming over-reliance on Key Informant Interviews with an under-emphasis on sampling and scale (e.g. 80% lack a sampling plan).

Findings – Protective Environment Framework and Cross-cutting Issues 6. Regarding the eight interconnected elements of the PEF:

a. UNICEF’s best work is at the national level, Strengthening Government Commitments through national plans/policies and building a Legislative Framework.

b. Conversely, Monitoring and Oversight ranks as the most deficient area of the entire PEF. Investments that align to centralized political power –e.g. relating to enforcement of a national plan--are among the least likely to be monitored for quality assurance.

c. The quality of programming addressing Access to Necessary and Child Friendly Services is second only to “Strengthening Government Commitments.” In five cases, investments in child friendly services have arguably laid the groundwork for broader, systemic changes. Nevertheless, evaluators do not assess systemization with any depth or consistency: only 15 reports (one-quarter of the meta-evaluation

Sheeran Consulting, Inc. 2008 UNICEF Child Protection Meta-Evaluation Final Report, 15 May 2008 iii

data universe) had relevant information on this topic and of these only 10 had a reliable evidence base.

d. Building Capacities of Those Closest to the Child is the third most highly rated area of the PEF for programming design and execution quality. UNICEF appears to be skilled at dealing with professionals: capacity building programmes for professionals were significantly better in quality over capacity building programs for families and communities. And, strengthening of capacities of professionals is linked to systematization and/or integration of CP programming. Further,

i. The meta-evaluation tracks 31% more activity in the “response” category than in the “recognition” category

ii. Community involvement in program planning, design and implementation appears to be a facilitating factor for recognition of and response to protection needs by families and communities.

iii. Yet six studies suggest that over-reliance on volunteer workers impedes CP recognition and response. And, 27% of the reports say that a lack of basic needs (e.g. water, food) makes families, communities and governments less willing and able to engage in CP issues.

iv. 20% of the reports do not touch upon systemization of capacity building; 29% do so but without a credible evidence base.

e. Neither Life Skills nor Child Participation are robustly represented in the reports that were available for the meta-evaluation, however:

i. Outcomes are good for CP in at least four instances where service providers had steady contacts with young people’s networks, which allowed for a sort of rolling needs assessment.

ii. Life Skills investments were described in terms of activities or outputs in 17 reports; of these, 16 could not deliver reliable information.

f. Open Discussion/Civic Engagement is meagerly represented. Neither programmes nor evaluations distinguish among knowledge, awareness, and advocacy.

g. Changing Attitudes, Customs, and Practices, another meagerly represented category of investments, shows a validated, replicable, methodology that packages peer-to-peer counseling, participatory monitoring, and training in rights awareness, with community education.

7. Regarding the Cross-Cutting Issues and Emergencies: a. Gender should have been a cross-cutting issue; seniority levels of CP partners are

un/under-reported; cost-analysis is virtually non-existent but a consistent theme is “do-more-for-less-itus.”

b. CP in Emergencies shows good overall coherence though situation analyses are not always asking the right questions.

8. A theme and gap analysis showed: a. Substantial missed opportunities for strategic and programme communications in

support of CP investments; only six reports discussed media-related investments and mostly at a descriptive level (e.g. activities or outputs).

b. A persistent concern in 71% of the reports that girls’ and boys’ different opportunities, roles, and needs, are only superficially accounted for in the programmes and in some cases missing altogether.

Sheeran Consulting, Inc. 2008 UNICEF Child Protection Meta-Evaluation Final Report, 15 May 2008 iv

Conclusions 9. Leadership role: Technical assistance supports to government and civil society partners are

possibly the premier deliverable of UNICEF to CP globally. This follows from findings showing that closest to central political power, UNICEF is making regular good use of its status as a convening agency with outputs including situation analyses and coalition development.

10. Context analysis: Without information about upstream and institutional factors, such as UNICEF global managerial priorities and donor commitments/expectations, the frame of reference for interpreting programming decision making (e.g. regarding monitoring or baselines) is narrow and un-illuminating.

11. Enforcement gap: While UNICEF finds and uses its “clout” most effectively with national level government partners, the meta-evaluation finds consistent gaps in enforcement and monitoring. These may reflect an undercurrent of instability in government commitments to the CP agenda, but the meta-evaluation is not conclusive on this point. It raises the question of whether enforcement weaknesses are basically the price of doing business on CP, or whether UNICEF could better leverage international agreements on aid effectiveness (e.g. Paris Declaration) in forging these agendas with host governments.

12. Internalizing the PEF: The central emphasis of the PEF, systematization of national and societal protections, is not clearly articulated in the evaluation reports; three findings indicate that programming is not systematic about systemization either: no evidence of contingency planning to address the difficulties of obtaining baseline data, insufficient analysis of pervasive inequalities such as gender, and ad hoc planning addressing linkages between poverty alleviation and community level buy in.

13. Strategic communications: The absence of investments on this topic could reflect sampling biases, or, alternatively, a weak emphasis on strategic communications within CP with a corollary need for technical support in design, monitoring, and implementation.

14. Trainings and volunteerism: Findings concerning community engagement and capacities development/training suggest that when these work well, they are powerful tools. A consistent standard about how best to utilize volunteers, however, is missing, and it is unclear why trainings –when they can be so powerful and additive—might ever be under resourced or poorly orchestrated.

15. Knowledge Management: The askSam database has been indispensible; used synergistically with Excel, askSam allowed for statistical and textual evidence to be contrasted side by side.

Sheeran Consulting, Inc. 2008 UNICEF Child Protection Meta-Evaluation Final Report, 15 May 2008 v

Acknowledgements The author expresses appreciation to the meta-evaluation managers, Shirin Nayernouri (Child Protection Section) and Sam Bickel (Evaluation Office) for technical vision and collegiality on this project. Howard Dale (Knowledge Management) consulted with us on the incorporation of the askSam database into the meta-evaluation. The author also wishes to thank Dnyanada Sharma (Child Protection) for administrative support. Participants in the one-day workshop on the meta-evaluation at UNICEF Global Headquarters (April 2008) provided depth and interpretation to the conclusions and generated an initial set of recommendations. Tyler Blake Davis, Ph.C ., oversaw the statistical components of the meta-evaluation and the askSam database development. Claire Jones, Ph.D., acted as a second reader and contributed particularly to the capacities development contents of the report. Ratna Menon, M.A., assisted in coding data for the askSam database and on the annotations of the Protective Environment Framework Classification System. Lisa Maynard proofread the final document. dzMedia supervised document layout and graphics. The author expresses great appreciation for all of these contributions. This study was commissioned by UNICEF Child Protection, New York, USA and remains the property of UNICEF. The views expressed in this report are those of the author and do not necessarily reflect those of UNICEF.

Sheeran Consulting, Inc. 2008 UNICEF Child Protection Meta-Evaluation Final Report, 15 May 2008 vi

Table of Contents

Executive Summary ................................................................................................................ ii

Acknowledgements................................................................................................................ v

List of Figures ........................................................................................................................ ix

Findings Guide ...................................................................................................................... xi

Chapter 1: Introduction ....................................................................................................1 Overview ...................................................................................................................................... 1 Tools and Methods ...................................................................................................................... 3 Limitations and Biases ................................................................................................................. 7 Profile of the Meta-evaluation Reports ....................................................................................... 9

Chapter 2: Government Commitment to Fulfilling Protection Rights of Children ............... 18 Introduction and Overview ........................................................................................................ 18 National Plans and Policies ........................................................................................................ 20 Implementation ......................................................................................................................... 25

Chapter 3: Legislation and Enforcement .......................................................................... 28 Introduction and Overview ........................................................................................................ 28 Legislation .................................................................................................................................. 29 Enforcement .............................................................................................................................. 32

Chapter 4: Attitudes, Traditions, Customs, Behavior, and Practices .................................. 36 Introduction and Overview ........................................................................................................ 36

Chapter 5: Open Discussion and Engagement .................................................................. 41 Introduction and Overview ........................................................................................................ 41 Increase in Public Knowledge About the Issue .......................................................................... 42 Open Discussion on Sensitive Issues .......................................................................................... 45 Use of Media for Advocacy Work .............................................................................................. 46 Formal Linkages to CRC Are Visible, Part of the Program ......................................................... 48 Children Feel Able to Speak and/or Act More Freely About Protection Concerns ................... 50

Chapter 6: Children’s Life Skills, Knowledge, and Participation ......................................... 51 Introduction and Overview ........................................................................................................ 51 Insights into What Children and Young People Know and Need to Know ................................ 52 Relevant Information and Services Are Provided ...................................................................... 54 Children and Young People Get Involved .................................................................................. 58

Chapter 7: Building Capacities of Those Closest to the Child ............................................. 62 Introduction and Overview ........................................................................................................ 62 Major Actors Are Facilitated in Recognizing Child Protection Needs ........................................ 64 Strengthened Capacities for Responding to Child Protection Needs ........................................ 68 Capacity Development Is Becoming Systematized and Integrated, not Ad Hoc ....................... 77

Sheeran Consulting, Inc. 2008 UNICEF Child Protection Meta-Evaluation Final Report, 15 May 2008 vii

Chapter 8: Services Delivery ............................................................................................ 86 Introduction and Overview ........................................................................................................ 86 Provision of Necessary and Child-Friendly Services .................................................................. 88 Strengthened Social Welfare Systems ....................................................................................... 96

Chapter 9: Monitoring and Oversight ............................................................................ 102 Introduction and Overview ...................................................................................................... 102 Monitoring Systems, CP Data Collection ................................................................................. 104 Participatory and Locally Based Mechanisms .......................................................................... 109 Reliable Reporting Systems ..................................................................................................... 110 Community Surveillance Systems ............................................................................................ 111

Chapter 10: Child Protection in Emergencies ................................................................... 113 Introduction and Overview ...................................................................................................... 113 Ability to Execute Rapid Assessments ..................................................................................... 114 Overall Management to Deliver a Holistic Response .............................................................. 116 Protection and Care of Separated/Unaccompanied Children and Children Associated with Armed Groups .......................................................................................................................... 120 Psychosocial Support Interventions Are Targeted and Quickly Established ........................... 121 Prevention/Response Capacities for Gender-Based Violence Are Quickly in Place ................ 124 Gender-Differentiated Needs Are Addressed in Emergencies ................................................ 125

Chapter 11: Cross-Cutting Issues ..................................................................................... 128 Introduction and Overview ...................................................................................................... 128 Costs ......................................................................................................................................... 129 Partnership Arrangements ...................................................................................................... 131 Child Protection Themes, Sectoral Good Practices ................................................................. 133 Scale ......................................................................................................................................... 136 Prevention or Response ........................................................................................................... 138

Chapter 12: Conclusions ................................................................................................. 140 Overview .................................................................................................................................. 140 Conclusions .............................................................................................................................. 142

Appendix One: Terms of Reference .................................................................................... 150

Appendix Two: Protective Environment Classification System ............................................. 153 Introduction ............................................................................................................................. 153

Appendix Three: Evidence Quality Reporting Framework .................................................... 159 Introduction ............................................................................................................................. 159 Evidence Quality Findings ........................................................................................................ 161 Limitations and Biases ............................................................................................................. 162

Appendix Four: Programming Practices Reporting Framework ............................................ 164 Introduction ............................................................................................................................. 164 Programming Practice Findings ............................................................................................... 165 Limitations and Biases ............................................................................................................. 166

Appendix Five: Meta-Eval Reports/Evidence Ratings ........................................................... 167

Appendix Six: Frequency of Evidence, Protective Environment Elements ............................. 179

Appendix Seven: Thematic Overview, Meta-Evaluation Reports .......................................... 181

Sheeran Consulting, Inc. 2008 UNICEF Child Protection Meta-Evaluation Final Report, 15 May 2008 viii

Appendix Eight: How Large Are the Programs and the Evaluations? .................................... 185 Introduction ............................................................................................................................. 185 Target Population and Sampled Population ............................................................................ 185

Appendix Nine: Stakeholders’ Initial Recommendations ..................................................... 188 Introduction ............................................................................................................................. 188 Recommendations ................................................................................................................... 188

Sheeran Consulting, Inc. 2008 UNICEF Child Protection Meta-Evaluation Final Report, 15 May 2008 ix

List of Figures Figure 1: Geographic Distribution of Meta-evaluation Reports .............................................. 10

Figure 2: Average Quality, Protective Environment Elements (Aggregates) ............................ 12

Figure 3: Programme Practices: Leaders and Laggards ........................................................... 13

Figure 4: Average Quality, Protective Environment Elements (Disaggregated) ..................... 157

Figure 5: Evidence Quality Reporting Matrix ....................................................................... 160

Figure 6: Evidence Quality, Average Per Variable ................................................................ 161

Figure 7: Programme Practices Reporting Framework ......................................................... 164

Figure 9: Frequency of Evidence, Protective Environment elements .................................... 180

Figure 10: Scale of Evaluations versus Scale of Programmes/Projects .................................. 187

Sheeran Consulting, Inc. 2008 UNICEF Child Protection Meta-Evaluation Final Report, 15 May 2008 x

Sheeran Consulting, Inc. 2008 UNICEF Child Protection Meta-Evaluation Final Report, 15 May 2008 xi

Findings Guide Because the meta-evaluation tracked at least 64 distinct characteristics over a relatively small number of reports (59), “findings” for any one of these characteristics can consist of a single item in a single report. Critical mass, in other words, was not a determinant in applying the label “findings.” As such, this guide could be equally appropriately titled “Guide to investment strategies.” Further:

For the most part, findings that form the basis of the Conclusions chapter are those which have the largest degree of recurrence, and

The applicability of findings to Child Protection thematic areas, for instance, juvenile justice or trafficking, must be determined. The meta-evaluation focused on the Protective Environment Framework in practice, not on thematic sub-sections per se.

Executive Summary ................................................................................................................ ii

List of Figures ........................................................................................................................ ix

Findings Guide ...................................................................................................................... xi

Chapter 1: Introduction ....................................................................................................1 1.1 Most/least frequent Protective Environment investments ............................................. 11 1.1.2 Average Quality, Protective Environment investments ................................................ 12

1.2 Programming Practices: Leaders and Laggards ................................................................... 12 1.2.1 Strategic planning ......................................................................................................... 13 1.2.2 Context-appropriateness .............................................................................................. 14 1.2.3 Community ownership .................................................................................................. 14 1.2.4 Facilitative role (UNICEF’s practical and moral persuasion) ......................................... 14

1.3 Theme and Gap Analysis ...................................................................................................... 15 1.3.1 de Facto gender bias ..................................................................................................... 15 1.3.2 Limited Communication for Development (C4D) investments ..................................... 16 1.3.3. Missing quality assurance accountabilities .................................................................. 16 1.3.4 Non-systematic sampling plans and small scale evaluations ........................................ 16 1.3.5 Broad donor-political contexts are under-analyzed ..................................................... 17

Chapter 2: Government Commitment to Fulfilling Protection Rights of Children ............... 18 2.1 Investment characteristics that strengthen government commitments ............................ 20

2.1.1 Technical assistance, coalition building ........................................................................ 20 2.1.2 Technical assistance, policy development .................................................................... 22 2.1.3 Technical assistance, pilot projects ............................................................................... 23

2.2 Constraints to strengthening government commitments ................................................... 25 2.2.1 Limited follow through mechanisms and accountabilities ........................................... 25 2.2.2 Lessons lost due to a weak evidence base .................................................................... 27

Chapter 3: Legislation and Enforcement .......................................................................... 28 3.1 Investment characteristics that support legal reform ......................................................... 30

3.1.1 Technical supports that grasp the big picture .............................................................. 30 3.2 Constraints to legal reform .................................................................................................. 30

Sheeran Consulting, Inc. 2008 UNICEF Child Protection Meta-Evaluation Final Report, 15 May 2008 xii

3.2.1 Legislative efforts are not addressing accountabilities for enforcement ..................... 30 3.3 Investment characteristics that support enforcement ........................................................ 33

3.3.1 Community stakeholders build on the threat of enforcement ..................................... 33 3.4 Constraints to enforcement ................................................................................................. 34

3.4.1 Enforcement is largely at the level of moral persuasion .............................................. 34

Chapter 4: Attitudes, Traditions, Customs, Behavior, and Practices .................................. 36 4.1 Investment characteristics that facilitate changed attitudes, practices .............................. 37

4.1.1 Intergenerational alliances, with peer counseling and health education ..................... 37 4.2 Constraints to changing attitudes, practices ....................................................................... 38

4.2.1 Lessons lost due to a weak evidence base .................................................................... 38

Chapter 5: Open Discussion and Engagement .................................................................. 41 5.1 Supports to increased public knowledge ............................................................................. 43

5.1.1 Leveraging already mature networks/administrative mechanisms ............................. 43 5.2 Constraints to expanding public knowledge ........................................................................ 44

5.2.1 Investments in awareness-raising without service or policy follow up ........................ 44 5.3 Supports to facilitating open discussion .............................................................................. 45

5.3.1 Intergenerational alliances, with peer counseling and health education ..................... 45 5.4 Constraints to facilitating open discussion .......................................................................... 45

5.4.1 Lessons lost due to a weak evidence base .................................................................... 45 5.5 Constraints to media/advocacy ........................................................................................... 46

5.5.1 Lessons lost due to a weak evidence base .................................................................... 46 5.6 Supports to facilitating rights awareness ............................................................................ 48

5.6.1 Using the CRC to a programming lingua franca ............................................................ 48 5.7 Constraints to facilitating rights awareness ........................................................................ 49

5.7.1 Investments are framed without reference to the CRC ................................................ 49 5.7.2 Lessons lost due to a weak evidence base .................................................................... 50

Chapter 6: Children’s Life Skills, Knowledge, and Participation ......................................... 51 6.1 Finding out what young people know and need to know ................................................... 52

6.1.1 Regular contact acts as a rolling needs assessment ..................................................... 53 6.2 Constraints to finding out what young people know and need to know ............................ 53

6.2.1 Top-down programming missed vital details ................................................................ 53 6.2.2 Lessons lost due to a weak evidence base .................................................................... 54

6.3 Supports to delivering relevant life skills ............................................................................. 55 6.3.1 Regular contact acts as a rolling needs assessment ..................................................... 55

6.4 Constraints to delivering relevant life skills ......................................................................... 55 6.4.1 Life skills component is missing .................................................................................... 56 6.4.2 Training deficiencies constrain delivery ........................................................................ 56 6.4.3 No impact monitoring ................................................................................................... 56 6.4.4 Not taken to scale ......................................................................................................... 57 6.4.5 Lessons lost due to a weak evidence base .................................................................... 57

6.5 Investment characteristics that support child participation ............................................... 58 6.5.1 Close proximity of service provides and young people ................................................ 58

6.6 Constraints to child participation ........................................................................................ 59 6.6.2 Investments are not framed with respect to participation .......................................... 60 6.6.3 Lessons lost due to a weak evidence base .................................................................... 60

Chapter 7: Building Capacities of Those Closest to the Child ............................................. 62

Sheeran Consulting, Inc. 2008 UNICEF Child Protection Meta-Evaluation Final Report, 15 May 2008 xiii

7.1 Investment characteristics that facilitate recognition of child protection needs ............... 64 7.1.1 Reaching families via adolescent livelihoods interventions .......................................... 64 7.1.2 Reaching families/communities via involvement in programme planning .................. 65 7.1.3 Reaching communities via capacity building programming ......................................... 65 7.1.4 Reaching professionals via training and agent-partner linkages .................................. 65 7.1.5 Reaching all stakeholders via rights awareness training .............................................. 66

7.2 Constraints to greater recognition of child protection needs ............................................. 66 7.2.1 Abusive relatives not being engaged by community-based project ............................. 66 7.2.2 Communities/CSOs need more intense training........................................................... 66 7.2.3 Over-reliance on volunteers ......................................................................................... 66 7.2.4 Weak front-end needs assessments ............................................................................. 67 7.2.5 Lack of community involvement in programming ........................................................ 67 7.2.6 Professionals’ training is inadequate ........................................................................... 67 7.2.7 Poor coordination among institutions and agencies .................................................... 67 7.2.8 Lessons lost due to a weak evidence base .................................................................... 67

7.3 Investment characteristics that facilitate response capacities ............................................ 69 7.3.1 Providing relevant supports (e.g., crèches) to families ................................................. 69 7.3.2 Rooting child protection efforts in the community ...................................................... 69 7.3.3 Community outreach makes a difference ..................................................................... 70 7.3.4 “Indigenized” models are effective ............................................................................... 70 7.3.5 Facilitating agency-community cooperation................................................................. 70 7.3.6 Training is a vital investment for building professionals’ response capacities ............. 70 7.3.7 Specialist capacity building has clear value................................................................... 71 7.3.8 Advocacy and mobilization supports professionals’ response capacities .................... 71 7.3.9 Giving the same trainings to all stakeholders: .............................................................. 72

7.4 Constraints to strengthening response capacities .............................................................. 72 7.4.1 For young people, unsupportive parents are a constraint ........................................... 72 7.4.2 Families’ economic needs are unmet ........................................................................... 72 7.4.3 For communities/CSOs, an over-reliance on unpaid volunteers .................................. 72 7.4.4 For communities/CSOs, an over-reliance on non-professionals ................................... 73 7.4.5 For communities/CSOs, material resource constraints ................................................ 73 7.4.6 For communities/CSOs, inadequate training ................................................................ 73 7.4.7 For communities/CSOs, vertical communication breakdowns ..................................... 73 7.4.8 For communities/CSOs, failure to build public awareness ........................................... 74 7.4.9 Other constraints for communities/CSOs: unmet basic needs, ownership confusion . 74 7.4.10 For professionals, myriad training inadequacies ........................................................ 75 7.4.11 For professionals, too much staff turnover ................................................................. 75 7.4.12 For professionals, lack of human resources and expertise ........................................ 76 7.4.13 For professionals, lack of material resources/transport ............................................ 76 7.4.14 For professionals, failure to integrate approaches .................................................... 76 7.4. 15 Lessons lost due to a weak evidence base ................................................................. 76

7.5 Investment characteristics that facilitate systematic capacity development ..................... 78 7.5.1 Integrating capacities development with awareness raising ........................................ 78 7.5.2. Professionals’ capacity building integrated into long term CP programming ............. 78 7.5.3 Establishing a cohort of trained child protection professionals ................................... 79 7.5.4 Developing local ownership .......................................................................................... 79 7.5.5 Effective cooperation and division of labor between institutions ................................ 80

7.6 Constraints to systematizing capacities development ........................................................ 81

Sheeran Consulting, Inc. 2008 UNICEF Child Protection Meta-Evaluation Final Report, 15 May 2008 xiv

7.6.1 Training is limited to a one-time event (“one off”) ....................................................... 81 7.6.2 No quality control on trainings ..................................................................................... 81 7.6.3 Trainees have no practical work experience ................................................................. 81 7.6.4 Unmet needs and ownership confusion are not addressed ......................................... 82 7.6.5 Dependency syndrome ................................................................................................. 82 7.6.6 Confusion among partners ............................................................................................ 83 7.6.7 Inadequate material support ........................................................................................ 83 7.6.8 Unclear project aims and strategies ............................................................................. 83 7.6.9 Need to grapple with conditions in conflict-affected areas .......................................... 84 7.6.10 Lessons lost due to a weak evidence base .................................................................. 84

Chapter 8: Services Delivery ............................................................................................ 86 8.1 Investment characteristics that facilitate service delivery .................................................. 89

8.1.1 Specific services for specific demographic, with community outreach ........................ 89 8.2 Constraints to service delivery ............................................................................................. 90

8.2.1 No substantive accountabilities .................................................................................... 90 8.2.2 Unassertive management ............................................................................................. 92 8.2.3 Misunderstanding programme contexts ...................................................................... 93 8.2.4 Inattention to girls’ needs, opportunities ..................................................................... 94 8.2.5 Overestimating volunteer capacities ............................................................................ 94

8.3 Investment characteristics that contribute to stronger social safety nets .......................... 97 8.3.1 Context-driven, with long-term outlooks ..................................................................... 97

8.4 Constraints to strengthening social safety nets .................................................................. 98 8.4.1 Skills training in a vacuum ............................................................................................. 98 8.4.2 Implementation pushed ahead of institutional capacity building ................................ 98 8.4.3 Small scale investments and limited time horizons ...................................................... 99

Chapter 9: Monitoring and Oversight ............................................................................ 102 9.1 Investment characteristics that support systematic monitoring, data collection ............. 105

9.1.1 Management or (high level) government commitment ............................................. 105 9.2 Constraints to systematic monitoring, data collection ...................................................... 106

9.2.1 Call for an “off the shelf” system ................................................................................ 106 9.2.2 Management commitments (not capacities) are not evident .................................... 106 9.2.3 The “why” of monitoring—not just the “how”—needs reinforcing ........................... 107 9.2.4 Lessons lost due to a weak evidence base .................................................................. 108

9.3 Investment characteristics of functional participatory, locally based mechanisms .......... 109 9.3.1 Strong intergenerational alliance at the community level ......................................... 109

9.4. Constraints to participatory, locally based mechanisms .................................................. 109 9.4.1 Failing to recognize and make good use of obvious opportunities ............................ 109

9.5 Investment characteristics of reliable reporting systems.................................................. 110 9.5.1 High level government involvement ........................................................................... 110

9.6 Constraints to reliable reporting systems .......................................................................... 110 9.6.1 Programmes implemented without reporting plans .................................................. 110 9.6.2 Lessons lost due to a weak evidence base .................................................................. 111

9.7 Investment characteristics of working community surveillance systems ......................... 111 9.7.1 Strong intergenerational alliance at the community level ......................................... 111

9.8 Constraints to mobilizing community surveillance systems .............................................. 112 9.8.1 Lessons lost due to a weak evidence base .................................................................. 112

Sheeran Consulting, Inc. 2008 UNICEF Child Protection Meta-Evaluation Final Report, 15 May 2008 xv

Chapter 10: Child Protection in Emergencies ................................................................... 113 10.1 Investment characteristics of good rapid assessment capacities .................................... 114

10.1.1 Situation analysis has to ask the right questions ...................................................... 114 10.2 Constraints to rapid assessment capacities ..................................................................... 115

10.2.1 Situation analysis ignored or missing ........................................................................ 115 10.3 Investment characteristics of well managed, holistic responses .................................... 117

10.3.1 Essential: partnership-building in non-emergency times ......................................... 117 10.4 Constraints to management response capacities............................................................ 118

10.4.1 Limited quality assurance mechanisms .................................................................... 118 10.4.2 Unresolved administrative, fiscal, coordination Issues............................................. 118

10.5 Supports to separated/unaccompanied children ............................................................ 120 10.5.1 Pre-planning for family based care ........................................................................... 120

10.6 Investment characteristics of targeted psychosocial interventions ................................ 121 10.6.1 Ability to mobilize quickly ......................................................................................... 121 10.6.2 Providing a safe haven .............................................................................................. 122 10.6.3 Training develops intergenerational cadre ............................................................... 122

10.7 Constraints to psychosocial interventions ....................................................................... 122 10.7.1 Inadvertently excluding out-of-school children ........................................................ 122 10.7.2 Haphazard and non-integrative service delivery ...................................................... 123 10.7.3 Confusion on service provision to adolescents ......................................................... 123 10.7.4 Inconsistent application of the psychosocial framework ......................................... 123 10.7.5 Missing needs assessment ........................................................................................ 123 10.7.6 Over-reliance on teachers ......................................................................................... 124

10.8 Investment characteristics of poorly gender-differentiated needs in emergencies ....... 125 10.8.1 Ignorance of basic socio-cultural definitions ............................................................ 125 10.8.2 Programmatic gender apartheid, no impact analyses .............................................. 125 10.8.3 Failure to advocate for girls’ dignity and equity ....................................................... 126

Chapter 11: Cross-Cutting Issues ..................................................................................... 128 11.1 Effective cost analyses ..................................................................................................... 129

11.1.1 Cost-benefit analysis is extremely limited ................................................................ 129 11.1.2 Two evaluation discuss connections between costs and sustainability ................... 129

11.2 Constraints to costs analyses ........................................................................................... 130 11.2.1 Rigid budget requirements constrain the program .................................................. 130 11.2.2 “Do-more-for-less-itus” ............................................................................................. 130 11.2.3 Lessons lost due to a weak evidence base ................................................................ 130

11.3 Investment characteristics of partnerships arrangements .............................................. 132 11.3.1 Who: Civil Society Organizations and Government Ministries ................................. 132 11.3.2 What: Implementation and advocacy, not enforcement ......................................... 132 11.3.3 Where: Low to middle-level seniority ....................................................................... 132

11.4 Inter-sectorality blind spots ............................................................................................. 133 11.4.1 Child Protection needs to consider socio-economic contexts .................................. 133 11.4.2 Protection must address economic vulnerabilities ................................................... 134 11.4.3 Child Protection should increase synergies with other areas ................................... 135 11.4.4 Poverty negatively affects community participation ................................................ 135

Chapter 12: Conclusions ................................................................................................. 140 Programming and Evaluation Practices ................................................................................... 142

12.1.1 Crippling programming and evaluation practices are taken for granted ................. 142

Sheeran Consulting, Inc. 2008 UNICEF Child Protection Meta-Evaluation Final Report, 15 May 2008 xvi

12.1.2 De facto operations assumptions may be limiting the ambition of Child Protection evaluations ........................................................................................................................... 142 12.1.3 Improving analysis of institutional contexts is a necessity ....................................... 142 12.1.4 Need to re-examine quality control, leverage, at the at the sub-programme level . 142 12.1.5 Difficult to reconcile gender findings with corporate commitments ....................... 143 12.1.6 The askSam database has been an indispensible tracking tool ............................... 143

Strengthening Government Commitments ............................................................................. 143 12.2.1 How the convening organization convenes .............................................................. 143 12.2.2 Disparity in provision of technical information inputs .............................................. 143 12.2.3 Need to clarify the reasons for dis-investments in enforcement ............................. 144

Legislation and Enforcement ................................................................................................... 144 12.3.1 Evaluators do not address institutional contexts, re constraints to enforcement ... 144 12.3.2 A de facto “backdoor” enforcement strategy is in operation ................................... 144 12.3.3 Lopsided supports to duty bearers versus rights holders ......................................... 144

Attitudes, Traditions, Customs, Behaviors, Practices .............................................................. 144 12.4.1 A wholly replicable methodology is not seeing much marketing ............................. 145 12.4.2 Models/theories of societal change may be missing? .............................................. 145

Open discussion/engagement with Child Protection issues.................................................... 145 12.5.1 Need to clarify practical/theoretical differences: “information” and “awareness” . 145 12.5.2 Rational and place for programme communications (“C4D”) are unclear ............... 145 12.5.3 Rights based approaches have an unclear status ..................................................... 145

Children’s Life Skills, Knowledge, and Participation ................................................................ 146 12.6.1 Rights based approaches are not a central emphasis .............................................. 146 12.6.2 Evaluating Life Skills investments has not been a priority ........................................ 146

Building capacities of those closest to the child ...................................................................... 146 12.7.1 Professionals get more professional inputs from UNICEF? ...................................... 146 12.7.2 There is no consistent standard for working with volunteers .................................. 146 12.7.3 The front end information needs of civil society partners/stakeholders are taken for granted ................................................................................................................................. 147 12.7.4 Protective Environment programming theory is unclear about the status of building capacities of families ............................................................................................................ 147 12.7.5 Evaluators do not address the systematization of capacities building ..................... 147 12.7.6 Best investment towards sustainability: training professionals ............................... 147

Delivery of necessary, child friendly, services ......................................................................... 147 12.8.1 Evaluation terms of reference: out of sync with the PEF.......................................... 147 12.8.2 Child Protection vastly understates its successes ..................................................... 148

Monitoring and Oversight........................................................................................................ 148 12.9.1 No explanation for monitoring findings, they are totally inexplicable .................... 148

Child Protection in Emergencies .............................................................................................. 148 12.10.1 Gender and Psychosocial findings merit further study ........................................... 148

Cross-cutting issues ................................................................................................................. 149 12.11.1 Lack of clarity on what makes a cross-cutting issue cross-cutting .......................... 149

Sheeran Consulting, Inc. 2008 UNICEF Child Protection Meta-Evaluation Final Report, 15 May 2008 1

Chapter 1: Introduction

Overview Meta-evaluation purpose and context This meta-evaluation covers UNICEF Child Protection investments between 2002 and 2007. The purpose is to develop “an authoritative basis for confirming effective protection interventions.” Child Protection is one of the five UNICEF programme focus areas and represents 10.2% of the global programme assistance expenditure.1 In 2002, in the context of the Mid-Term Strategic Plan, UNICEF introduced a new strategy for Child Protection, the Protective Environment Framework (PEF). This defines eight elements that are critical to good protection and that allow UNICEF to work toward systemic changes in national and societal support for protective laws, policies, and social conventions. The introduction of the PEF coincided with a management concern to consolidate the evidence base for what works and what does not work. As articulated in the Terms of Reference:

All but one of the 157 UNICEF-supported countries, areas, and territories address child protection issues programmatically. At the same time, not all country offices work confidently in child protection, and the evidence base remains small, including in verifying to what extent and how the elements of the protective environment are understood and addressed in the field by UNICEF and others. 2

In developing an authoritative basis for confirming effective protection investments, the meta-evaluation will (a) provide UNICEF Child Protection with the basis for global lessons sharing with partners and for global, regional, and national programming and advocacy efforts, (b) support management accountability for capturing lessons learned and examining the impact of UNICEF’s work in child protection, and (c) inform any revisions of UNICEF’s Child Protection goals, especially those contained in the Medium Term Strategic Plan 2006-2009.3

1 Source: 2006 UNICEF Annual Report. The other four programme areas and budgets are: Child Survival and Development

(51%), Basic Education and Gender Equality (21.3%), HIV/AIDS and Children (5.5%), and Policy Advocacy and Partnerships for Children’s Rights (11%). 2 Terms of Reference may be found in Appendix One. In 2006, UNICEF cooperated with 155 countries, areas, and territories: 44

in sub-Saharan Africa (ESARO and WCARO); 35 in Latin America and the Caribbean (TACRO); 35 in Asia (EAPRO and ROSA); 20 in the Middle East and North Africa (MENARO); and 21 in Central and Eastern Europe and the Commonwealth of Independent States (CEE/CIS). The scope of child protection programmes and interventions vary according to the magnitude of protection issues in these countries and the set priorities. 3 The Mid-Term Strategic Plan serves as UNICEF’s guiding framework over a four year period. It is developed through a

consultative process with Executive Board Members, United Nations agencies, non-governmental organizations, and UNICEF staff. A copy of this document is available at this web address: http://www.unicef.org/childsurvival/files/05-11_MTSP.pdf

OVERVIEW The Protective Environment Framework defines eight elements of child protection that allow UNICEF to work toward systemic changes in national and societal support for protective laws, policies, and social conventions.


Objectives and scope To address the overall purposes of the meta-evaluation, two basic objectives were set: to identify viable evidence based programming strategies, and to examine programming drivers and constraints. An important subsidiary objective was to explore knowledge-management software options to help index evidence based good practices. Summarizing from the Terms of Reference:

1. “Determine which programming strategies and project interventions have been proven to work. Proven to work means that the combination of good design, monitoring, and quality evaluation allows UNICEF to state that certain approaches—if well executed—are likely to result in measureable improvement in resisting violations of children’s rights to protection, and to enhancing the resiliency and social networks that will foster strong child and community development.”

2. Examine the evidence “against the goals contained in the Protective Environment Framework” to allow Child Protection to contribute to UNICEF and the child protection community broadly on matters such as management accountabilities, training priorities, knowledge management strategies.4

3. Determine the value added of knowledge management software options to help index evidence based good practices.

Although the scope of the meta-evaluation is global, covering all regions and in principle all countries, only English language reports were slated for inclusion. Of 149 Child Protection evaluation reports that were completed during the time period designated for the meta-evaluation, 2002 to 2007, English language reports comprised 117 or 78% of the total.5 The meta-evaluation is based upon 59 evaluation reports. This is 50% of the total in English and 40% of the total for all languages for the period under consideration.

4 The terms of reference for this assignment specify that the meta-evaluation should allow Child Protection to contribute to

UNICEF and the child protection community broadly on matters such as management accountabilities, training priorities, and knowledge management strategies (pp. 2–3 Terms of Reference). These are characterized as “Other direct and indirect expected benefits” (p.3). 5 These figures are estimates provided by the Evaluation Office (January 2008). The time period for the meta-evaluation, 2002-

2007, was selected in light of the coterminous introduction, during those years, of the Protective Environment Framework as a new programming strategy.

PRIMARY OBJECTIVE Determine programming strategies that are proven to work … through the combination of good design, monitoring, and quality evaluation … in building the Protective Environment for children.


Tools and Methods

Reporting Framework

Overview: reporting tools To realize the objectives described above, UNICEF Child Protection and UNICEF Evaluation Office developed two reporting tools:

To report Child Protection activities relating to the eight components of the Protective Environment Framework, emergency child protection, and five cross-cutting issues.

To report “programming evidence of a high quality” according to guidelines established by the UNICEF Evaluation Office for this meta-evaluation.

Both tools contained numerous sub-categories. These provided the possibility of tracking as many as 64 distinct variables across an expected 70 evaluation reports.6 Neither tool was set up to allow for corporate self reflection per se. That is, the tools could collect information about child protection activities and evidence quality, but they could not collect information about elements of UNICEF’s own programming practice such as programme design and execution.7 To fill this gap, the evaluator built a third reporting tool:

To report on programming practice as it may impinge upon or support protective environment investments, such as strategic planning, knowledge management, stakeholder engagement, and context sensitivity.

askSam database The askSam database is a free-form database solution that allows text searching of materials from a variety of source document types. The UNICEF Evaluation Office selected this database and steered its incorporation into the meta-evaluation as an experiment (successful, in the opinion of the evaluator) in knowledge management. Each report was coded according to the Protective Environment Classification System and delivered into askSam. Creating the meta-evaluation data set This section provides further detail on the reporting tools and summarizes the methodology that united them to produce a statistical dataset for the meta-evaluation.

6 The original terms of reference specified that 70 evaluation reports would form the basis of the meta-evaluation. Only 59

were identified as suitable in time to be included. This turned out to be more than a matter of expediency, however: by the 40th

report the level of redundancy across the reports was clear. The additional reports up to 59 helped to confirm these trends. 7 As this report unfolds, readers will see that evaluation report authors linger on programming practices to the detriment of

evaluating outputs. A paucity of longitudinal data constrained the analysis of outcomes, and impacts.

TOOLS The meta-analysis catalogued Protective Environment strategies and cross-referenced them to (a) evidence quality and (b) programme design and execution quality.


1. Protective Environment Classification System This classification system contains nine primary categories—the eight components of the Protective Environment Framework, plus Child Protection in Emergencies—and an additional five cross-cutting categories that are “critical for successful Child Protection strategies”.8 The 14 categories each contained numerous sub-categories, 53 in all, and many contained numerous sub-sub-categories. These various levels of differentiation describe strategies and outcome areas for protective environment investments. The Classification System in its entirety may be found in Appendix Two. The primary categories of the Protective Environment Framework are:

Strengthening Government Commitments

Legislation and Enforcement

Attitudes, Traditions, Customs, and Behaviors

Open Discussion/Engagement on Child Protection Issues

Children’s Knowledge, Life Skills, and Participation

Building Capacities of Those Closest to the Child

Delivery of Necessary and Child Friendly Services

Monitoring and Oversight Each category was filtered through any of five cross-cutting issues: Costs, Partnerships, Intersectorality, Scale, and Prevention or Response. For example, when an investment in, say, delivering necessary and child friendly services, worked well, what were the partnerships in play? As another example, which, if any, of the successful Protective Environment categories seemed to map to larger or smaller scale of investment? 2. Programme Practice Classification System This system tracks eight elements of programming practice that, when in play in a Protective Environment investment strategy, contribute to or constrain success. In other words, this is a system that allows the meta-evaluation to probe more deeply into causation than is possible within the framework of the Protective Environment Classification System. The Programme Practice Classification system asks “why,” for instance:

Did a protective environment investment pay enough attention to community buy-in in developing a community-based child protection network?

Did management drop the ball in putting a monitoring system into place or in facilitating partners to ensure proper quality assurance and oversight?

Consulting Appendix Four, readers will see that the Programming Practice system is organized alphanumerically with letter codes for the programming practice elements and numerics for how well or how poorly they were put into play. The eight components are:

a. Context (In practical terms, how well does the programme design fit the political, cultural, and economic climate?)

8 Excerpted from the Meta-Evaluation Terms of Reference, p. 5.


b. Intersectorality (Reasonable coordination with other related needs?) c. Programme communications d. UNICEF’s use of its facilitative role e. Achieved level of community ownership f. Child participation g. Strategic planning (How the work will get done, by whom, with what resources, etc.) h. Knowledge management

How was the programming practice system developed? During the inception phase of the meta-evaluation:

Step one - The evaluator used the Protective Environment Classification System to conduct a content

analysis of the first 47 reports and was able to identify a recurring pattern of themes and issues, which mostly pointed to programming practice as making or breaking Protective Environment investments.

Step two - On the basis of the recurring thematic patterns, the evaluator converted the Protective

Environment Classification System into a programming practice grid. This grid describes a continuum of programming practices ranging from unacceptable to excellent.

Step three - Based on this continuum, the evaluator then designed a simple numerical ratings scale to

address the appropriateness/effectiveness of the Protective Environment strategies discussed in each report, so that if it appeared (on the subjective basis of the report) that a particular strategy had worked well or was poorly executed or not sufficiently thought out, it would be rated high or low accordingly. The system operated on a scale of zero (unacceptable) to three (excellent).

How was this tool applied? During the in-depth analysis phase, the evaluator:

Step four - Returned to each report in depth, categorized investments according to their best fit within the

Protective Environment Classification System, filtered them back through the programming practices grid, and assigned a quality rating. This process allowed the meta-evaluation to make judgments about quality and effectiveness of investment strategies.

- Meaningful evidence addressing any component of the Protective Environment Classification

System was assigned a code to correspond to the programme practice that was most obviously in play. More than one letter code could be assigned as necessary.

Appendix Four provides more details about the programming practices classification system and where the more highly rated investments were tending to cluster in relation to any of its eight elements.


How does the programming practices system relate to the matter of evidence quality? The evidence quality tool acted as a sort of “backseat driver” to the programming practice classification and measuring system. That is, if a particular finding was unjustified or contradictory on the basis of the evaluator's own evidence, it was ignored. However, the two tools pursued different lines of emphasis. The evidence quality tool and the programming practices system proved to be mutually reinforcing: good quality ratings on the Protective Environment strategies correlated well with good evidence ratings, and vice versa. Moreover, a significant pattern in programming practices also emerged. Among other things, this showed that poorly rated Protective Environment investments shared similar weaknesses in programming practice—for example, neglecting the findings of a situation analysis or not having had a situation analysis to begin with. Ultimately, the system helps to identify a failed or successful investment while simultaneously shedding light on programming practices that were most or least in play. Further information on this system and many more findings than can be summarize in the main report may be found in Appendix Four. 3. Meta-evaluation Evidence Quality Tool This tool was designed by the UNICEF Evaluation Office to determine evidence quality. It uses seven variables, each rated on a scale of zero to three (zero = unacceptable, three = excellent).

Results based management levels - Whether the evaluator reports activities or goes forward to report outcomes or impacts

Use of comparison groups - Selected at the time of evaluation or random assignment

Incorporation of longitudinal data - End evaluation only or baseline, middle, and end

Acknowledgement and reflection on international standards - None or uses globally accepted standards

Analysis of intended and unintended consequences - None or considers intended and unintended consequences

Data analysis/Disaggregation - None or use of compound or single variables

Statistical analysis - None or complex analysis of survey data

Appendix Three provides details on the ratings averages for individual variables. It also comments on areas of similarity and divergence between these meta-evaluation ratings and the ratings given to UNICEF Evaluations generally for the period under scrutiny.

4. askSam database The evaluation reports were coded for content that shed light on the various elements of the Protective Environment Classification System. These reports were then imported into askSam. The coding followed basic text searching protocols for askSam to allow flexible key word and alpha-numeric search queries based on the reporting variables. The real strength of AskSam is the ability to examine actual text from

RESULT Excellent evidence quality indicators included: use of base-middle-end-line time series, random selection of comparison groups, and complex statistical analysis.


the 59 documents that pertain to the same element at the same time. Without the ability to use coded documents the evaluative process would be reduced to more general comparison. askSam reports were generated for each of the Protective Environment Framework characteristics and according to the level of evidence quality (low, medium, high). In Excel, the quantitative assessments of programming practices were captured as scores. These were accompanied by notations on the meta-evaluator’s reasons, giving the project’s subjective evidence base complete transparency. While Excel and AskSam were used for two different ends of the project, they were both structured by the Protective Framework elements. As a result, the two data stores worked synergistically, with the statistical and textual evidence contrasted side by side.

A “ReadMe” file (available on request from UNICEF Child Protection) guides first time users in searching the database.

Limitations and Biases The evaluation uses non-probability purposive sampling based on this criteria: materials were included if they were available, if they were in English, and if they were completed between 2002 and 2007.9 This meta-evaluation is comprised of English-language only evaluation reports. As explained earlier, this excludes about 21% of evaluation reports (mostly in Spanish and French) for the period under consideration. This evaluator notes three limitations of the meta-evaluation process: 1. The meta-evaluation tracks about 64 unique characteristics of the

Protective Environment Framework (including Child Protection in Emergencies and Cross-cutting Issues) across a relatively small number of reports. As such, in some instances a "finding" for any one of these characteristics consists of a single report. Put another way, there is enough redundancy in some evaluation findings to allow a firm conclusion to be drawn, for instance, with respect to monitoring or to the quality of certain programming practices. But the analysis of findings is necessarily constrained by the high level of itemization in the Protective Environment Classification System. In what sense, then, is the meta-evaluation a baseline for Child Protection? Inasmuch as the fundamental requirement of a “baseline” is to show a specific performance measure, findings for the programming practices and evidence quality meet this requirement. The meta-evaluation also provides lessons about how the Protective Environment Framework operates in practice, for instance, that technical work such as situation analyses cluster around governments more than communities.

9 As explained in the Research Methods Knowledge Base http://www.socialresearchmethods.net/kb/sampnon.php , “With a

purposive sample, you are likely to get the opinions of your target population, but you are also likely to overweight subgroups in your population that are more readily accessible.” Readers are reminded that the meta-evaluation tracks information on the basis of the Protective Environment Framework categories, not child protection thematic categories.

LIMITATION With some exceptions, evaluation reports tend to operate along the lines of customer satisfaction surveys, where the likeability of the investment (from the point of view of key informants) stands in for outcomes analyses.

http://www.socialresearchmethods.net/kb/sampnon.php


In terms of individual thematic areas of Child Protection, for instance, child labor or early marriage, findings about what works and what is not working must be scrutinized for their applicability.

2. The level of detail addressing any topic in the Protective Environment Framework Classification

System varied greatly from one report to another. For example, one report may contain in-depth information on partnerships, while another may have only touched up this in favor of focus on other issues. This kind of disparity is a consequence of the meta-evaluation’s retrospective application of the Protective Environment Framework, which for the most part had not yet begun to inform evaluation reporting emphases for the time frame that was selected for the meta-evaluation. Yet,

While the reports are too varied in the level of detail to allow for much comparative depth, and

While the actual universe of reports for any particular characteristic of the Protective Framework Classification System may be as large as 20 or as small as two,

Common themes emerged and provide Child Protection with (1) insights into its investments; (2) some evidence-based good practices; (3) a storehouse of information on programming design and execution lessons learned (and a few lessons lost); and (4) a solid basis to regularize Child Protection evaluation reporting, i.e., “Model Terms of Reference.”

2. The reports that were available for the meta-evaluation provide (a) insufficient analysis of

institutional contexts (e.g. donor financing limitations, host government sensitivities), (b) scarce time series information (e.g. 64% of the reports have no longitudinal data), and (c) an alarming over-reliance on Key Informant Interviews with an under-emphasis on sampling and scale (e.g. 80% lack a sampling plan). With some important exceptions the evaluations tend to operate along the lines of customer satisfaction surveys where the likeability of the investment too often stands in for impact. This situation, together with the high level of itemization in the reporting framework (see Point One, above), means that in many cases the data for identifying good practices and significant trends is largely inferential.

The meta-evaluation compensates for Points One and Four by (a) indicating sample sizes for all themes, and (b) presenting themes and issues in the context of verbatim extracts from the reports. Limitations and biases of the three reporting matrices (Protective Environment Framework Classification System, Programme Practice Classification System, and Evidence Ratings Tool) may be found in their respective appendices.


Profile of the Meta-evaluation Reports

Evidence Quality and Regional Draw

Ratings and rankings In 2004, UNICEF implemented an evaluation quality ratings system that reflects and is complementary to international evaluation standards. This system rates evaluations on 22 variables such as the clarity of the description of stakeholder contributions, the sensitivity to evaluation ethics, concordance with rights based approaches, transparency of methodology, and depth of interpretation of findings. All meta-evaluation reports were initially assessed with this ratings system. A comparison between all UNICEF evaluations versus those for the Child Protection Meta-Evaluation showed no significant differences:

Average UNICEF evaluation quality, 2000–2006, is 2.3410 o 58% of the reports received a “Satisfactory” rating; 26% rated “Poor”

Average Child Protection Meta-Evaluation quality rating is 2.43 o 63% of the reports rated as “Satisfactory”; 22% rated “Poor”

The Meta-evaluation Evidence Quality Ratings focuses exclusively on how the evidence base for each report was generated. According to this tool:

The more highly rated reports utilized longitudinal data, comparison groups, and complex statistical analysis. These reports were also consistently attentive on matters pertaining to results based tools and methods. Approximately 41% of the reports (24) carried good ratings and four were excellent.

The average overall quality was poor with nearly 40% scoring in that range (24); two were rated as unacceptable.

o 64% of the reports have no longitudinal data; 80% had no sampling plan Appendix Five lists all evidence quality ratings alongside each report’s title, country, and date. Regional distribution of meta-evaluation reports The following chart illustrates that the meta-evaluation’s regional draw is relatively equal for the four of seven regions where UNICEF works: Central and Eastern Europe and the Commonwealth of Independent States (CEE/CIS), East Asia and the Pacific (EAPRO), Eastern and Southern Africa (ESARO), and the Middle

10

http://www.unicef.org/evaldatabase/files/UNICEF_Evaluation_Report_Quality_Review_2006.pdf The 2006 review sample

was nearly five times the size of the Child Protection meta-evaluation sample (313 reports to 59). This probably accounts for the small percentage difference in the quality rating averages between the two groups of evaluations. A detailed comparison of Child Protection ratings for individual standards with ratings of the 2006 Quality Review would be expected to map to the Review’s trend analysis which shows, among other findings, that the quality of Child Protection evaluations is increasing annually.

http://www.unicef.org/evaldatabase/files/UNICEF_Evaluation_Report_Quality_Review_2006.pdf


East and North Africa (MENA). It is quite low for the remaining three regions: Latin America and the Caribbean (TACRO), West and Central Africa (WCARO), and South Asia (ROSA). This difference is largely reflective of the meta-evaluation’s concentration on English language materials for TACRO and WCARO and the relatively smaller number of countries in ROSA.

Figure 1: Geographic Distribution of Meta-evaluation Reports

Thematic Overview

The UNICEF Evaluation Database lists all reports by primary and secondary theme codes. For the reports that were available for the meta-evaluation:

The largest two thematic categories are “Multi-thematic Child Protection evaluation” (18 reports) and “Armed conflict/Child soldiers” (10 reports).

o Of the “Multi-thematic” evaluation reports, the mean rating was slightly lower than the overall mean for the meta-evaluation.11 In terms of evidence quality, the reports were split evenly with half having evidence ratings of satisfactory or higher and half scoring low to unacceptable.

11 The overall mean for the meta-evaluation for evidence quality was 1.45; for the “multi-thematic” category, the mean was

1.37.


The next largest thematic categories are “Juvenile Justice” (7 reports) and “Sexual exploitation/trafficking” (6 reports).

“Social protection; social safety nets; conditional cash transfers (CCT)” is a secondary theme in 17 reports. 14 of these come under the category “Multi-thematic Child Protection.”

“Child rights legal and policy/administrative reform; Convention on the Rights of the Child (CRC) CRC compliance; policy and law analysis” is a secondary theme in 10 reports.

Gender based violence and sexual exploitation is a secondary theme in only five reports. In terms of evidence quality, the lowest rated report of the meta-evaluation is one of these.

No report takes child participation as a secondary theme and there is no category in the coding classification system for Child Protection reports to address.

The “programme communications/media analysis” category for Child Protection is not represented in the meta-evaluation. Undoubtedly this reflects what was available for the meta-evaluation; however, a call for information on investments of this nature would help to clarify the status of programme communications (“Communication for Development”, or C4D) in child protection investments to build a protective environment.

Appendix Seven lists all reports by primary and secondary themes. 12

1.1 Characteristics of the Protective Environment Investments

1.1 Most/least frequent Protective Environment investments Figure 9, Appendix Six, shows the frequency of evidence that address characteristics of the Protective Environment Framework and Emergencies in the 59 reports.13 The following points may be made:

Within the universe of this meta-evaluation, evidence addressing Open Discussion on Sensitive Issues and Information on Sensitive Topics is limited.

The highest evidence areas are: Strengthening Capacities (of governments and community/professional stakeholders) and Providing Child Friendly Services.

Monitoring and Oversight is also an area of very high evidence area; however, this reflects the frequency with which evaluators flagged its absence or incompleteness.

Legislation-related activities are less well represented in this meta-evaluation than investments dedicated to developing national plans of action and partnerships.

12 The Evaluation Office classification system for Child Protection has certain constraints. For example, while a very standard

Child Protection investment is the care of children who may have been orphaned by war, the taxonomy breaks this into two separate categories: Armed Conflict, and Children without Caregivers. The meta-evaluation author has systematized these and other classifications through a careful analysis of each report’s primary and secondary areas of emphasis. 13 Fig 9 does not include cross-cutting issues because, as shown in Chapter 11, “Cross-Cutting Issues,” the evidentiary scope for

these was minimal.

MAJOR THEMES Multi-thematic CP evaluations are the largest primary category of the reports (18); "Social Protection" is the largest secondary thematic category (17); Programme Communications (“Communication For Development”) has no showing.


1.1.2 Average Quality, Protective Environment investments In this Average Quality Ratings Table (see Fig. 2), the horizontal axis shows the eight components of the Protective Environment plus Emergencies. The vertical axis shows the average quality rating of the programming practices (their appropriateness and effectiveness) for these nine components. Again, these quality ratings are not measures of evidence quality: they are measures of the quality and effectiveness of the investment strategies, as facilitated or constrained by programming practices.

Figure 2: Average Quality, Protective Environment Elements (Aggregates)

The following observations may be made regarding this chart and its disaggregated version in Appendix Two (Figure 4):

The highest ratings are attached to the central axis of the Protective Environment: Government commitment to fulfilling protection rights of children. The disaggregated version of this chart (Appendix Two, Fig. 4) shows that within this category, efforts to strengthen partnerships, advocacy, and implementation are receiving the highest values for appropriateness and effectiveness.

The lowest quality ratings were seen in the area of monitoring and oversight. This is the weakest link of Child Protection investments as represented in the meta-evaluation reports.

1.2 Programming Practices: Leaders and Laggards


The following chart (Fig. 3) shows the frequency with which Protective Environment investment strategies pointed meaningfully to any of the eight characteristics of effective programming.

Figure 3: Programme Practices: Leaders and Laggards

*Vertical Axis = Number of occurrences (frequency) when the programming variable was flagged as having significant evidence

*Horizontal Axis = Eight characteristics of effective programming

Frequency of evidence is shown on the vertical axis against eight programming practices on the horizontal axis. Appendix Four carries further detail on programming practices leaders and laggards. Anticipating in-depth discussion of these characteristics in relation to Protective Environment elements as presented in the main chapters of this report, four points can be flagged now: 1.2.1 Strategic planning As the chart in Figure 3 makes clear, strategic planning has been flagged more than any other programming practice. Strategic planning is defined as how the work will get done, by whom, and with what resources. The density of evidence around this topic, which is four and one-half times higher than that for child participation, conveys a great deal of information about the professionalization of development practice. How is this so?


Strategic planning-related practices were flagged most often in relation to the weakest link in the Protective Environment Framework: monitoring and oversight. Only 26% of the strategic planning practices in this area had a satisfactory or higher quality rating.14

This is as opposed to good or better quality ratings for 62% of the strategic planning-related practices for service delivery, 72% for building capacities of those closest to the child, and 75% for child participation.

1.2.2 Context-appropriateness Context-appropriateness, that is, the “fit” between programming design and concept and the programming environment (e.g., political, cultural, ethnic, economic), was the second most populated programming practice for the meta-evaluation, as seen in Figure 3. This is no surprise at all: with the bulk of Child Protection investments focusing on building capacities of those closest to the child (within the context of this meta-evaluation, at any rate), the topic of how well or poorly UNICEF understood the environment of potential allies came up regularly. Good or better ratings were given to investments that appear to:

Tap into relevant formal and informal decision-making networks.

Activate new, unusual partnerships. About 58% of the time, the Protective Environment investment’s resonance with its context was rated good or better. 1.2.3 Community ownership Behind programme design and execution issues relating to strategic planning and context-appropriateness, community ownership was the third most populated programming practice area of the meta-evaluation. If the investment helped to facilitate buy-in and/or was community-defined or driven, it was rated good or better. If expectations were unrealistic—for example, evaluators found volunteer fatigue that had not been addressed—this programming practice was rated poorly. “Building capacities of those closest to the child” was the component of the Protective Environment where practices relating to community ownership had the most evidence. About 45% of the times, this programming practice was rated good or better. On a related note, as stated earlier, the “Multi-thematic Child Protection” theme is the largest category of Child Protection investments with an even split, nine for nine, in reports with satisfactory versus unsatisfactory evidence ratings. The findings are not conclusive as to whether these –as opposed to more narrowly defined—investments stand out in any way. Multi-themed programs are successful in some instances, and must be evaluated on a case-by-case basis. The community-based programs are most likely to succeed when there is maximum input from the community, i.e. ownership (stakeholders and target population). Yet these are the same cases that caution against over-use of (community) volunteers. 1.2.4 Facilitative role (UNICEF’s practical and moral persuasion)

14 All disaggregated values may be found in Fig. 8 (Appendix Four)


As shown in Figure 3, above, UNICEF’s use of its role as a convening organization (“facilitative role”) is the fourth most populated programming practice in the meta-evaluation. Essentially, did UNICEF know who to lobby and how to end programming bottlenecks, and proceed accordingly? Significantly, when the answer is “yes” to the above question, it is usually with respect to UNICEF’s work with government partners at the national level:

The closer to centralized political power, the more “clout” UNICEF appears to find and utilize (at least in the universe of reports that were available for the meta-evaluation).

With good results: strengthening government commitments and Legislation/Enforcement had good or better ratings in this area of programming practice in 68% and 64% of the instances, respectively.

Alternatively, there are practically no instances of UNICEF using its comparative “clout” to press for monitoring and oversight, for civic engagement with Child Protection issues, or with respect to service delivery.

Further information may be found in Appendix Four, Programme Practices.

1.3 Theme and Gap Analysis

The Terms of Reference for this project call for a gap and thematic coverage analysis. Three topics carry the weight here:

Is there a de facto orientation in programming and evaluation that gives gender analysis a secondary emphasis?

Where are the effective instances of programme communications (Communication For Development, C4D) supporting Protective Environment efforts?

How does evaluation practice limit what’s known and knowable? Does it shed light on practices that could be specific to Child Protection?

1.3.1 de Facto gender bias Since the universe for this evaluation report addresses only the available reports that are in English, it is not feasible to make an overall assessment of efforts to strengthen the protective environment for girls. Within the scope of these evaluation reports, however, the situation appears worrisome:

Six of 59 reports carry gender as a secondary thematic emphasis. These come under four general categories: Sexual Exploitation and Trafficking (2), Violence (2), Street Children (1), and Harmful Traditional Practice (1). One of the smaller and worst rated reports addresses harmful tradition practices.

GENDER BIAS? Most (42 of 59) reports find gender analysis to have been insufficient at the level of program design and implementation.


Supporting the impression of a secondary status for gender analysis, the meta-evaluation finds that gender analysis is insufficient at the level of programme design including with respect to emergencies.

1.3.2 Limited Communication for Development (C4D) investments The overall place of advocacy and the media as tools in strengthening the protective environment cannot be determined from the reports that were available for the meta-evaluation. Although this is likely a robust area of investment, it is not well represented in this context. In fact, no reports carry a secondary thematic emphasis addressing any aspect of programme communications or advocacy. This is in spite of the fact that the biggest category of investments—training and capacity building—would seem naturally to align with efforts to nurture a broad public dialogue. Indeed, the analysis shows that for capacity building there is a heavier relative degree of emphasis on capacity building to respond rather than to recognize child protection needs. Where programme communications were most frequently present (Developing Open Discussion/Engagement), in 41% of the instances (39 times) the ratings were satisfactory or higher. 1.3.3. Missing quality assurance accountabilities Monitoring capabilities as represented in these reports do not differ significantly from the global norm in that they are processes in need of support. 90% of the reports that were available for the meta-evaluation contained complaints about monitoring having been insufficient or absent altogether. The meta-evaluation finds this area of programming practice to be critically sub-standard. The situation regarding baselines also reflects the global norm in that in the majority of the reports, evaluators say that there were no baselines for the programmes being evaluated. Yet, the meta-evaluation also notes a critical absence of dialogue by evaluators about the institutional drivers of this situation, such as donor financing requirements. 15

1.3.4 Non-systematic sampling plans and small scale evaluations The validity of evaluation evidence is typically measured in relation to the evaluation’s plan for surveying a subset of the programme’s target population. This is the sampling plan: of the universe targeted for an intervention, what portion of that universe can be tapped for research purposes to get a generally accurate picture of what is going on? In the reports for the meta-evaluation, evaluators did not provide guidance on which findings could be generalized beyond the stakeholders that were consulted for the evaluations. For example:

15 The status of impact evaluation in global development aid has been addressed in a recent study by the Center for Global

Development Evaluation Gap Working Group, When Will We Ever Learn? Improving Lives through Impact Evaluation (May, 2006) http://www.cgdev.org/content/calendar/detail/7829/ . According to this report, “An “evaluation gap” has emerged because governments, official donors, and other funders do not demand or produce enough impact evaluations . . .” (p.2). The NONIE network (Network of Networks Impact Evaluation Initiative) also lists resources on the problems and issues relating to impact evaluations and development aid.

EVIDENCE BASE The meta-evaluation evidence base is largely atemporal and ad hoc: approximately 64% of the reports have no longitudinal data and 80% lack sampling plans.


Only 50% of the reports reflect on the number of stakeholders consulted for the evaluation in relation to the number of stakeholders targeted by the programme.

Of this subset, 20% have a stated criteria for which respondents were selected for the evaluation (sampling plan).

The overall absence of stated criterion for how respondents were selected for the evaluations also created difficulties in interpreting the scale of the evaluations in which Child Protection is investing:

Whereas in four (known) instances, the evaluators consulted with more than one thousand stakeholders,

In as many as twenty (known) instances, the number of stakeholders consulted for the evaluation is negligible. For example,

o 52 stakeholders consulted for a programme targeting 11,000 (Bosnia and Herzegovina 2003).

o 28 . . . for 1,733 (Zambia 2002). o 109 . . . for 8,000 (Eritrea, 2004).

It is unnecessary, of course, to have a huge sample size to get relevant statistical data. A sample size of a thousand can give high confidence over a population of several million (say 95% confidence level and +/-5% confidence interval). The point here, however, is not about sampling plans or confidence intervals per se, but to ask whether Child Protection evaluations are conceived and funded at too small a scale. Further information on the matters of programme and evaluation scale may be found in Appendix Eight. Fig. 10, Appendix Eight, is a simple bar graph showing the number of stakeholders that have been targeted by a programme (when known) versus the number of stakeholders consulted for the evaluation (when known). 1.3.5 Broad donor-political contexts are under-analyzed Even in the stronger evidence-based reports, there is no analysis of donor requirements as a potential factor in programming design and evaluation emphases and timelines. There is no information, for instance, as to whether donors pressed the case for baseline studies, or made ample funding and time available to ensure that these were properly developed. There is likewise a signal drop off in analyses of political considerations between UNICEF and host governments that may constrain or otherwise shape programming and evaluation. How is the balance of power between UNICEF and host governments (or among UNICEF, governments, and donors) a factor in the disposition by Child Protection to place a relative de-emphasis on the mechanics of enforcement?

EVIDENCE BASE How might the balance of power between UNICEF and host governments be a factor in Child Protection’s de-emphasis on enforcement-related investments? Evaluation reporting does not address this question.


Chapter 2: Government Commitment to Fulfilling Protection Rights of Children

UNICEF tends to be chosen as a preferred partner by governmental counterparts. Such credibility is based, among other things, on the organisation’s capacity of coordinating work with numerous partners at the same time, as well as its proven ability to act as a bridge between governments and civil society.

Evaluation of the UNICEF Impact on the Juvenile Justice System Reform (Serbia, Romania, Tajikistan 2006)

Introduction and Overview Government commitment to fulfilling protection rights of children is the cornerstone of the Protective Environment. Government interest in, recognition of, and commitment to child protection is an essential element for a protective environment. This includes ensuring that adequate resources are made available for child protection. It also includes political leaders being proactive in raising protection on the agenda and acting as advocates for protection. Within the universe of this meta-evaluation, the “Government Commitments” component of the Protective Environment is not the largest direct category of investment; however, it is the most highly rated for the quality of its programming practices.16 Evaluation report contents were analyzed on the basis of two general sub-categories17:

National plans, policies, and budgets

Partnerships, advocacy, and implementation

Investments to strengthen partnerships, advocacy, and implementation were twice as common among the reports used in this meta-evaluation as efforts addressing national plans, policies, and budgets per se. However, the latter was 8.5% stronger in terms of the quality of its programming practices than the former. 16 Reminder: Appendix Two contains disaggregated investment quality ratings for the Protective Environment Framework. 17 The askSam database was set up to track a third category, “Commitments emphasize prevention, response, or remediation;”

however, the level of detail to ensure accuracy on this point was not available in the majority of the reports.


Twenty-one reports contain evidence on this area of the Protective Environment. In the aggregate:

UNICEF advocates for child protection at the national level through three technical outputs:

o Bringing partners to the table.

o Providing a good situation analysis to sensitize partners to the issues.

o Investing in pilot projects.

UNICEF does not advocate for outputs to address:

o Results-based tools and methods.

o Technical details of implementation.

Essentially, efficacy flows from UNICEF’s use of its status as a convening organization. On matters with high visibility—HIV and AIDS, juvenile justice reform, trafficking—reports show that UNICEF Child Protection is strong in facilitating policy development. This indeed was described as an area of “best investment” for the cause of juvenile justice reform in Serbia, Montenegro, Romania, and Tajikistan. Consistent with this strong facilitative role, the mine risk education programme evaluation found that UNICEF’s comparative strengths were more salient at the level of advocacy, not implementation. The Programming Practice Leaders and Laggards table (Fig. 3, previous chapter) shows that UNICEF’s use of its facilitative role was the fourth most common element of programming practice to come out in the meta-evaluation reports, running behind strategic planning, community ownership, and context relevance. However, with the exception of the Protective Environment component addressing building capacities of those closest to the child, use of UNICEF’s leveraging capacities was almost ten times more dominant in “Government Commitments” and “Legislation” than in any other area of the Protective Environment.

What works — supports to strengthening government commitments

Technical assistance in (a) establishing inter-ministerial bodies, (b) policy development, and (c) developing experimental or pilot projects.

What is not working — constraints to strengthening government commitments

De-emphasis on monitoring and oversight, weaknesses in the area of sub-programme coordination, and a tendency to underplay follow through.


National Plans and Policies

2.1 Investment characteristics that strengthen government commitments

’best investments’…: Advocacy and support for the establishment of inter-ministerial, inter-sectoral bodies, which have been established in Serbia, Montenegro, and Tajikistan and provide UNICEF with opportunities for a dialogue with all its partners in juvenile justice reform. (emphasis in original)

Evaluation of the UNICEF Impact on the Juvenile Justice System Reform (Serbia, Romania, Tajikistan) (2006)

As this epigram suggests, a “best investment” for building national political will is establishing inter-ministerial bodies, which provide stakeholders with an otherwise non-existent context for dialogue and planning. Within the universe of materials that was available for the meta-evaluation, this is one of three areas of Child Protection practice that show efficacy. The other two are: Policy Development, and Investments in Pilot Projects.

2.1.1 Technical assistance, coalition building As indicated already, the meta-evaluation finds efficacy where the Child Protection focus was on mobilizing partners’ political will through strategies including building national plans of action and ministerial-level monitoring bodies (seven reports; first six with good evidence base ratings).

Evaluation of the Street Children and Street Mother, Partnership Programme (Ethiopia 2004)

Creating the national political will:

o Of utmost importance in terms of contribution has been the huge strides made in consciountization of the plight of street children and mothers among the different strata of Ethiopian society. Although it has yet to reach the critical mass whereby it translates into the enactment of policies and the allocation of public budget, the awareness created has been instrumental in drawing national attention to the phenomenon. The government has formulated a national action plan and the media has begun publicizing the plight of street children and mothers.18

Evaluation of the Protecting Street Girls and Ending Child Migration Programme (Ghana 2003)

Identifying significant decision makers from the top on down to the community level:

o The RPP [Rights Protection and Promotion] programme also has a strong advocacy component at the national level. ‘Operation End Child Kayayee’ was a campaign initiated at the beginning of 2002 and engages advocates from among Ministers of State, Parliamentarians, traditional rulers, and opinion leaders at community level to fight child migration. Beneficiaries themselves are also actively engaged in the planning and implementation of the programme and managers pay special attention to issues of

18 Unless otherwise noted in the main text, all narratives attached to the evaluation reports are direct excerpts. This holds for

all content in all chapters of the meta-evaluation.

FINDINGS Activities like coalition building show good outcomes such as the formation of sustainable inter-ministerial working groups and programme monitoring groups (six reports).


sustainability in building linkages and synergies between the government and NGO sectors.

Evaluation of the Mahzel Project for Social Reintegration and Protection of Disadvantaged Children (Eritrea 2004)

Bringing partners to the table:

o The Programme Review and Monitoring Group (PROREMO) meetings were a managerial tool to periodically allow all partners to monitor progress in execution and implementation and to regulate project implementation. UNICEF played a crucial role in the overall coordination of the project through PROREMO meetings.

Evaluation of UNICEF’s Support to Mine Action (Global 2005)

Creating a framework or context:

o What UNICEF has done better is create platforms of dialogue, raise standards, and engage others in an inclusive collaborative forum at an international level.

Evaluation of the Psychosocial Care for Children Traumatized by Terrorist Violence Programme (Algeria 2003)


o In response to the needs, UNICEF has launched a project for psychosocial rehabilitation of children traumatized by terrorist violence. The project has developed through several stages from an initial response to the crises to a partnership in development with public sectors and non-governmental organization.



o …’best investments’…: Advocacy and support for the establishment of inter-ministerial, inter-sectoral bodies, which have been established in Serbia, Montenegro and Tajikistan and provide UNICEF with opportunities for a dialogue with all its partners in juvenile justice reform. (emphasis in original)

o UNICEF tends to be chosen as a preferred partner by governmental counterparts. Such credibility is based, among other things, on the organisation’s capacity of coordinating work with numerous partners at the same time, as well as its proven ability to act as a bridge between governments and civil society.

Mid-Term Evaluation of the “Children’s Chance for Change” Project, a Juvenile Justice Initiative in Serbia and Montenegro (2006) — This evaluation is not highly rated for evidence quality, but it shows at least one evaluator’s sense of the magnitude of UNICEF’s contributions:

… the project attracted attention of professional and wider community . . . The most reputable experts and social and political figures of GO and NGO sector were included in the project activities through the work of Juvenile Justice Committee of the Ministry of Justice of the Government of Republic of Serbia.

"The Programme Review and

Monitoring Group (PROREMO) meetings were a managerial tool to periodically allow all

partners to monitor progress in execution and implementation

and to regulate project implementation. UNICEF played

a crucial role in the overall coordination of the project

through PROREMO meetings."

Eritrea 2004

EXCERPT


2.1.2 Technical assistance, policy development Seven reports describe UNICEF’s technical supports to national-level policy development. The evidence quality is viable in only two and these offer slightly differing findings on the matter of technical assistance.

Education as a Preventive Strategy Against Child Labour (Global 2003) — UNICEF was found to be technically deficient, but still effective in terms of advocacy:

o Policy advice and advocacy require UNICEF to have a more comprehensive understanding of and play an active role in existing and planned policies and strategies of governments and other partners.

o In general, the Programme was considered relevant by participating countries in a variety of ways: activating their political commitment to the CRC; increasing awareness and understanding of child labour in relation to education …

Evaluation of the UNICEF Impact on the Juvenile Justice System Reform (Serbia, Romania, Tajikistan) (2006) — Here, technical proficiency and vision projected UNICEF into a leadership role:

o On the whole [regarding Tajikistan], it would seem that the need for a comprehensive approach was not being met by any other international actor, except UNICEF.

The following five reports have a less certain evidence base. Together with the two cited just now, these reports show UNICEF working on the ground floor, so to speak, providing technical assistance in areas that are new and unfamiliar. This is a feature of UNICEF’s work as a convening organization that is well leveraged in the service of Child Protection at the national level.19

Mid Term Review UNICEF Romania Report on Child Protection Programme (Romania 2002)

Good credibility in defining the course of action:

o Support for Child Protection Reform: … This sub-project is possibly one of the most important parts of the UNICEF Child Protection Programme. There are other actors providing technical assistance and training but it is difficult to think of another actor with the authority concerning policy development that UNICEF has. For the following years the policy part of the project will be of high relevance when the revised legislative framework shall be implemented.

Evaluation of UNICEF Support to Juvenile Justice System (Yemen 2004)

A responsible partnership:

o The main purpose of this programme is to assist the Government in developing a comprehensive national polices and strategies on all child protection issues …

Evaluation of UNICEF Project on Preventing Trafficking of Women and Children, Gender-Based Violence, and HIV/AIDS in Bosnia and Herzegovina (2003)

19

Two additional reports imply that UNICEF has been a central “mover and shaker” in supporting the developing of national plans and policies, but the information is not enough to go on: - Institution Building & Mainstreaming Child Protection in Indonesia UNICEF supported Child Protection Bodies (LPAs),

(Indonesia 2003) - Preliminary Assessment of the Use of Family and Community Conference Group as an Alternative Approach in Juvenile

Justice, Evaluation Report (Thailand 2007)

FINDINGS Technical assistance for policy development was found to be insufficient for Child Labor; for Juvenile Justice Reform, UNICEF showed technical vision and leadership. Because of poor evidence quality there are more lessons lost here than lessons learned.


Leadership in brokering inter-sectoral coordination:

o UNICEF BiH has been working to support the Ministry of Human Rights and Refugees in the development and implementation of their National Plan of Action on Trafficking … UNICEF was a bridge from government organisations to NGOs and now the door was open to mutual understanding and a different relationship based on a multi-sectoral approach. UNICEF ‘s role had also opened the possibility of cooperation with other international agencies…

Lessons Learned Review on the Law Enforcement Against Sexual Exploitation and Trafficking of Children Project (Cambodia 2006) — A partner agency (World Vision) steered this effort:

o LEASETC has very effectively harnessed high level commitment and support, ensuring that LEASETC’s efforts are regarded as priority activities in the MoI. Policy development at the national level has been slow. However, this was to be expected with such a new concept, the benefits of which needed to be demonstrated through the successful implementation of the pilot projects.

Evaluation of the Deinstitutionalisation Process in the Republic of Tajikistan (2005) — This is an example of evaluation findings that are difficult to harmonize:

o On the macro level, this project has been a catalyst for change. The development of the new Children’s Rights Departments has sparked the beginnings of a national debate on the need for replication of these departments across the country and the need for systemic reform of the child protection system both at local and national level.

o The lack of a clear direction by the State has resulted in a similar lack of an integrated response from the international donor community …

2.1.3 Technical assistance, pilot projects Four reports describe foundational, preliminary, or experimental investments, including pilot projects and situation analyses, which have helped to define or elaborate national policies. The first three have good evidence quality ratings.

An Assessment of the Impact of Implementing the MVC Programme and the Operation of Funds and the Potential for Scaling Up (Tanzania 2007)

Good stewardship of pilot project lessons learned:

o … the national guidelines and monitoring and evaluation system which resulted from the pilot phase of the programme are now used by collaborating partners who support programmes for most vulnerable children with resources from the Global Fund and from PEPFAR [President’s Emergency Plan for AIDS Relief]. A national implementing partners’ forum meets regularly, chaired by the Department of Social Welfare, to review progress and discuss any problems which may arise.20

20 The author also writes that a “national framework for social protection is under development” with an inference that UNICEF

is involved in that process.

FINDINGS

Because of poor evidence quality there are more lessons lost regarding technical assistance for policy development than lessons learned.

FINDINGS Using pilot projects to complement work toward national plans and policies shows good outcomes (three reports).


Evaluation of Lihlombe Lekukhalela Child Protectors (Swaziland 2005)

Good stewardship of a situation analysis:

o The DPM’s (Deputy Prime Minister’s) office coordinated the survey involving community youth assessments … UNICEF in collaboration with Save the Children, Deputy Prime Minister’s Office, National Emergency Response Council on HIV and AIDS (NERCHA), Care Nakekela, Swaziland Action Group Against Abuse (SWAGAA) has been able to set up child protection committees in 188 communities. A total of 5,690 child protectors have been trained on child protection.


More good stewardship of a situation analysis:

o …’best investments’…: A good situation analysis. The situation analysis not only can be, through its dissemination and presentation to government’s bodies, an effective instrument of sensitization on the social importance of juvenile justice reform, but can also, more specifically, contribute to a keen appreciation on the part of governmental counterparts of the value of data collection, evidence based planning, and monitoring. (emphasis in original)

o Last but not least, one of the main strengths of UNICEF’s programmes in support of juvenile justice reform is represented by the important focus given to the creation of community based alternatives. These initiatives vary from country to country, but all of them have an important element of innovation and are not only having a positive effect on the lives of children, but also by the force of example strengthening support for reform amongst professionals and decision makers.

External Evaluation of the Day Care Center “TISA” for Children with Disabilities (Montenegro 2005)

Working in a non-traditional area to deliver a pilot project that is designed for replication:

o Tisa is an innovative project … *it+ represents the first community setting for children with disabilities in Montenegro … designed to be replicated in Montenegro, through a network of day care centres, for children and teenagers with disabilities. …The results and lessons learned in the first year of functioning will contribute also to the analysis and propositions related with the quality standards for this type of service in Montenegro.

"… one of the main strengths of

UNICEF’s programmes in support of juvenile justice

reform is represented by the important focus given to the

creation of community based alternatives … all of them have

an important element of innovation and are not only

having a positive effect on the lives of children, but also by the force of example strengthening

support for reform amongst professionals and decision

makers."

CEE/CIS 2006

EXCERPT


Implementation

2.2 Constraints to strengthening government commitments

Activities supporting national plans and policies are not strongly oriented toward delivery. This finding is exemplified by a de-emphasis on monitoring and oversight, weaknesses in the area of sub-programme coordination, and a tendency to underplay follow through. National plans may be in place, as one evaluator found, without the requisite policy frameworks to see them through, or basic financial commitments may not have been mobilized. 2.2.1 Limited follow through mechanisms and accountabilities Constraints of this nature are described in the ten reports excerpted here. The first six have good evidence quality ratings.

Rehabilitation/Re-integration of Mine Victims and People with Disabilities (Cambodia 2005) — Suggests deficits in attention to follow through and missed advocacy opportunities:

o There is a definite need for more strategic level support to partners to work through programming issues with government services implementing other UNICEF funded responses. Equally there is a need for UNICEF to undertake high-level advocacy to government on issues affecting impact of Child Protection Programme supported projects.

o There seems to be a need across UNICEF to examine inter-programme and sub-programme coordination.

External Evaluation of the Child Protection Network (Cambodia 2004) — Suggests that the programme was put into place ahead of the necessary and enabling policy and fiscal measures:

o There is a lack of policy development from the part of the ministry. Monitoring and evaluation conducted by the national level are superficial and lack analysis. The national trainers who were supposed to take the role of linking the provincial and national level have not been able to fill this role.

o From a structural point of view, the (CBCP) Community Based Child Protection programme lacks a policy framework that would provide it with the necessary legitimacy and pressure to mobilise stakeholders from national to community level.

o There is a lack of long term, sustained support from communities and government. … As long as the government does not commit itself to contribute to the safety nets by allocating appropriate budgets to provincial social affairs, there is little hope that the situation of children improves substantially. Virtually all longer term solutions are provided through service providers outside the community, notably NGOs.

Education as a Preventive Strategy Against Child Labour (Global 2003) — Suggests a potentially missed advocacy and fundraising context to assure the investments’ future sustainability:

o The fact that the Programme has triggered the use of regular

FINDINGS

Investments in national plan- and policy-building are not strongly oriented toward follow through (six reports).


and other resources for child labour-related activities in several Country Programmes is a remarkable sign of growing commitment, but needs to be complemented by a more massive mobilisation of resources for the fight against child labour by governments, civil society (including the private sector), and external support agencies.

Evaluation of the UNICEF Impact on the Juvenile Justice system Reform in Three Countries in the Regions (Serbia, Romania, Tajikistan) (2006) — The findings here speak to general findings of the meta-evaluation itself:

o … main challenges … Reinforcement of results-based management within the UNICEF Country Offices, with greater efforts to be deployed in order to develop: well-defined long term strategies; more precise and concrete objectives; relevant and quantifiable indicators; tools for data collection and monitoring systems. More emphasis should then be given to the evaluation of impact or outcomes, rather than outputs. Last but not least, greater synergy and links between UNICEF’s work on juvenile justice and its work on other areas should be created. (emphasis in original)

An Assessment of the Impact of Implementing the MVC Programme and the Operation of Funds and the Potential for Scaling Up (Tanzania 2007) 21 — Suggests a failure to create ownership to facilitate implementation:

o … the problems plaguing the implementation of the pilot programme persist: in many areas, the programme continues to be viewed as an externally driven programme with consequent lack of local commitment among stakeholders and facilitators at all levels and mobilisation of community participation is limited

Impact Assessment of the Most Vulnerable Children (MVC) Community Based Care, Support, and Protection in Musoma Rural (Tanzania 2004) — Shows more evidence of inattention to implementation:

o The programme implementation process at all levels is constrained by poor coordination; lack of commitment by the government leaders; lack of integration of the programme to other development efforts; under-resourcing the programme; poor MVC data management and lack of follow-up and lastly, lack of integration of the non-state actors in the implementation of the programme (i.e., NGOs, FBOs).

21 On the face of it, a substantive counter example here would be the Tanzanian government’s establishment of a national

monitoring and evaluation system for the Most Vulnerable Child (MVC) programme. This appears however to have resulted from a United Nations agreement on the “Three Ones … one coordination mechanism, one national action plan, and one evaluation and monitoring system” (Tanzania 2007).

"The creation of networks and

structures is necessary, but not sufficient to ensure improved

child protection. Systems need to be functional, supported by

the actors involved, and oriented towards achieving

results. This requires in addition to a planning framework, a

commitment from government to develop and apply policy and

legislation." Cambodia 2004

EXCERPT


2.2.2 Lessons lost due to a weak evidence base

These four reports are among the weakest across the whole meta-evaluation for evidence quality. Excerpts are provided here to highlight lessons lost:

Midterm review - Gender and Child Protection Programme (Ethiopia 2004) — Without a strategy for implementation, the policy’s value is probably chiefly rhetorical:

o Absence of a national strategy, however, is indicated to be lacking for them22 to strongly and exhaustively pursue their effort along this line. The national policy (on Ethiopian Women), thus, needs to be accompanied by strategy to effectively address gender concerns as well as mainstream it.

Evaluation of Alternative Care Programmes (Sierra Leone 2003) — Missing data collection methods:

o Beginning in 1999, The National Commission for Disarmament, Demobilisation, and Reintegration (NCDDR) began a district by district systematic process of demobilizing these children who were attached to the fighting forces. Upon demobilization they were placed in a nation-wide Child Protection Network (CPN) Interim Care Centre (ICC) system while their families were traced … An accurate data collection system should be developed and updated regularly.

Final evaluation of the Empowerment of Children in Need of Special Protection Project (Bangladesh 2005) — Suggests the concept and design were shallow:

o Finally, any future project must articulate in more detail what is meant by an enabling environment for adolescents, and more systematically develop the local and national interventions required to create it.

Evaluation Report: Promoting the Deinstitutionalisation Process in the Republic of Tajikistan (2005) — Unclear where UNICEF’s efforts helped to address the State’s apparent inaction:

o The lack of a clear direction by the State has resulted in a similar lack of an integrated response from the international donor community and frequently there are examples of various ‘projects’ and initiatives that could be far more effectively dovetailed in order to give them a greater potential for sustainability.

22 The author is discussing “legal provisions observed in the Constitution, National Family Law as well as the drafting and

adoption and popularisation of the regional Family Law.”

FINDINGS

Four reports address investments relating to national plan- and policy-building and show that these are not strongly oriented towards follow through. However, poor evidence quality constrains the usefulness of these findings.


Chapter 3: Legislation and Enforcement More training on rights of MVC (Most Vulnerable Children) like inheritance rights should be organized for the caretakers, MVC, and the community. The idea of rights and legal rights is not well understood in villages. This is an area where impact has been minimal. . . .

Impact Assessment of MVC Support Programme Implementation in Karagwe, Kisarawe, and Magu (Tanzania 2004)

Introduction and Overview Legislation and enforcement is a second element of the Protective Environment. An adequate legislative framework, its consistent implementation, accountability, and a lack of impunity, are essential elements of a protective environment. Within the universe of this meta-evaluation, the “Legislation and Enforcement” component of the Protective Environment ranks fourth for the quality of its programming practices. Evaluation reports were analyzed on the basis of two sub-categories:

Legal development/reform reflecting child rights principles

Strengthened enforcement

The frequency of evidence on these sub-categories is minimal, ranking second to last (just ahead of “Monitoring and Oversight”) across all the meta-evaluation materials. Legislative investments are more highly rated than enforcement-related investments by 15% even though enforcement-related investments (which are primarily about advocacy for enforcement, not enforcement mechanics per se) outnumber legislative investments by 25%. The bulk of legislative emphasis is in juvenile justice reform and alternatives to institutionalization. Enforcement themes range from early marriage to community based re-integration, to preventing gender-based violence and trafficking of children.

In the aggregate, this component of the Protective Environment is

Strong on networking and advocacy

Weak on strategic planning and management

In the context of legislation and enforcement, UNICEF is most active as a resource or facilitator for bringing child protection issues onto the table. Conversely, in the implementation or trickle down arena —for instance, getting district and regional partners to coordinate — UNICEF is either not seeking or is


not able to procure impacts. This aligns to findings presented in the previous chapter under “What doesn’t work,” where efforts to cultivate national political will were found to be effective on the “mobilization” side but lacking where the apparatus of implementation pertained. What works — supports to legislation and enforcement

Knowledge of the big picture; community level stakeholders making use of the threat of legal sanctions.

What is not working — constraints to legislation and enforcement

De-emphasis on monitoring accountabilities and enforcement mechanisms at the state level; a disposition to prioritize advocacy for enforcement instead of literal enforcement mechanisms at the state level.

Legislation

Synopsis

Among the 12 reports addressing legislative efforts explicitly, the average evidence quality rating is about 16% lower than the average evidence quality for all the meta-evaluation reports.23 Only four reports had evidence quality of good or better. Findings here are thus constrained by a relatively weak evidence base as well as by the extremely limited sample size that was available for the meta-evaluation. Two themes occur commonly in legislation-related reports:

Within the universe of this meta-evaluation, legislative effort is concentrated in juvenile justice reform and institutionalization.

A relative de-emphasis on strategic planning and coordination weakens these efforts.

23 Average for the 12 reports is 1.21; for all reports the average quality rating is 1.44. The more poorly rated reports shared two

drawbacks: they were end-point evaluations only and had no or limited comparison groups. As discussed in the introductory chapter and as seen in evidence quality ratings for individual reports (Appendix Five), these are the two weakest links in evaluation practice for the meta-evaluation as a whole. To convey the magnitude of lessons lost through poor evidence quality, here is a tally of reports containing information about legislative efforts: Good evidence quality (four reports)

o Legal reform/juvenile justice (CEE/CIS Zambia), alternatives to institutional care (Belarus, Bosnia, and Herzegovina). Poor evidence quality (eight reports)

o Legal reform/juvenile justice (Thailand, Yemen), legislation to support social assistance (Romania), legislation and advocacy for a regional “family law” (Ethiopia), alternatives to institutional care (Moldova, Serbia, and Montenegro), child protection law (Indonesia [2]).


3.1 Investment characteristics that support legal reform

3.1.1 Technical supports that grasp the big picture

The common denominator here is UNICEF’s grasp of the big picture and where the investments would move the knowledge-practice nexus forward:

Evaluation of the Deinstitutionalisation Project (Belarus 2004) — Tangible outcomes:

o … Development of the regulatory and legal frames for implementation of the policy on deinstitutionalisation of orphaned children was one of three “direct outcomes” of the project.

o … An impact of this legislative reform is “Growth of the number of children placed into families (in 2003, in the majority of the regions 60% to 70% of children deprived of parental care were placed into families).”

Evaluation of the UNICEF Impact on the Juvenile Justice System Reform (Serbia, Romania, Tajikistan 2006) — Systematic contributions:

o On the whole, activities supported by UNICEF have been having a satisfactory degree of influence on the overall reform and the development of system in most of the countries.

Evaluation Report on Child Justice in Zambia (2005) — Investing in a quality situation analysis:

o Law reform: The current legislation is antiquated24 and is increasingly an impediment to transformation and improved service delivery. It is therefore recommended that law reform be embarked upon as a matter of priority … special attention should be given to the definitions of child, age of criminal capacity, custodial sentencing, remand period, diversion, and certain offences.

3.2 Constraints to legal reform

3.2.1 Legislative efforts are not addressing accountabilities for enforcement

UNICEF is engaged in activities such as financing of reports and situation analyses, technical assistance in reviewing existing legal frameworks and developing new ones, lobbying, and convening inter-sectoral coordinating entities. There is no evidence of investments related to monitoring and oversight, a gap that evaluators cite in several reports.

Three reports have good evidence quality. Excerpts from them illustrate evaluators’ concern about coordinating matters such as national–local linkages and developing management protocols to track and follow up on the investments:

Evaluation Report on Child Justice in Zambia (2005) — Observes that awareness of the issues is greater but, practically speaking, things are staying the same, and highlights the matter of accountability:

24 Juvenile Act, 1956

FINDINGS • Improved legal and regulatory frameworks for juvenile justice reform and institutionalization are validated outcomes in three reports. •However, in two of these, evaluators find no monitoring or quality assurance mechanisms.


o On a general level it can be concluded that an increased awareness and sensitivity towards children’s rights in the criminal justice system were observed in nearly all the sites visited during the fieldwork. This has, however, not translated into a general improvement in service delivery (especially outside of Lusaka) and children are still subjected to nearly all of the ills, misuses, and delays that were observed in 2000.

o Firstly, good governance principles need to be complied with, with specific reference to management skills, ability to deliver, and quality control.

Evaluation of the UNICEF Impact on the Juvenile Justice system (Serbia, Romania, Tajikistan 2006) — Working in unfamiliar areas, regular accountability protocols were lost:

o UNICEF’s culture of accountability is one of its strengths, but juvenile justice reform is a relatively new area for the organization and limited in-house experience and expertise have had a negative impact on strategic planning, the development of indicators, the monitoring and evaluation of implementation. In general terms, results-based management needs to be strengthened.

Evaluation of the Program “Improved Mechanisms of Child Protection" (Bosnia and Herzegovina 2003) — A program in place before the legal framework:

o The following factors did influence outcomes, although they could have been foreseen at program start: Legal framework / lack thereof on entity, state but also on cantonal level.


Enforcement

Where there are social workers, you can see the difference — she’s living in the community and they know the structures where they can report.

Child Protection Advocate, Community Based Reintegration: Programme Evaluation (Sierra Leone 2006)

Synopsis

Two themes dominate:

1. UNICEF Child Protection invests more in enforcement advocacy than in enforcement strategies and mechanisms. UNICEF invests:

o Almost exclusively in using its credibility and convening power for advocacy and mobilization.

o Substantially in building the child protection literacy of national governments and professionals.

o And substantially less in promoting a culture of monitoring and systematized reporting with regard to enforcement, or in specific enforcement mechanisms.

There is almost no instance of UNICEF leveraging its credibility and role as an “honest broker” to press for monitoring and oversight regarding enforcement.

2. Meanwhile, community level stakeholders use the threat of enforcement to bring about compliance.

o Stakeholders persuade others to follow the rules by pointing to the consequences of not following the rules.

o But only insofar as government and donors maintain good visibility and communicate commitments.

FINDINGS Investments cannot show results without directly addressing two enforcement challenges: creating (a) the mechanics, and (b) the political possibility, of compliance with legal mechanisms (four reports).


3.3 Investment characteristics that support enforcement

3.3.1 Community stakeholders build on the threat of enforcement

Although muted, a theme can be noted in several reports wherein the expectation of enforcement is a tool to be leveraged for the sake of child protection. This idea is reported from the point of view of community stakeholders for whom the symbolic and pragmatic possibilities of enforcement are meaningful.

Early Marriage Prevention Program, Endline Report on Kishori Abhijan: An Intervention Research Study on Adolescent Girls’ Livelihood (Bangladesh 2005)

Government “buy in” has to be recognized as such by members of the community:

o The collaboration of government representatives such as the Union Parishad Member, Chairman, in addition to the involvement of respected community members is especially effective … in that people should be obliged to respect decisions given by an official they, themselves elected to office.

Community Based Reintegration: Programme Evaluation (Sierra Leone 2006)

Government “buy-in” has to be palpable in the communities. The following are excerpts from members of child protection networks who were interviewed by the evaluator:

o People now know that there are people in charge who will prosecute you. They will think twice before doing anything to children.

o If they don’t see action from the Ministry, they say, why do we bother? Ministry needs to be strong to support them. If they go back and say, the kid you identified, this is what happened, they will be motivated to continue. Where there are social workers, you can see the difference — she’s living in the community and they know the structures where they can report.

o Since IRC left us, we only met once … Once they left, everything died down and fell apart. We need IRC. When we had problems we consult them. No sooner than IRC left us with the promise that the Ministry will follow suit. As of now, they have not followed up at all.

Lessons Learned Review on the Law Enforcement Against Sexual Exploitation and Trafficking of Children Project (Cambodia 2006)25

Well situated to press for enforcement:

22

World Vision appears to have been the lead agency in this multi-partner effort beginning in the late 1990s. UNICEF’s investments are unclear, however, there is a note that support is to be extended as of 2006. World Vision managed and funded this evaluation with one objective being “to make recommendations for UNICEF’s end-of-project evaluation.” This evaluation is poorly rated for evidence quality.

FINDINGS The data suggest that UNICEF de-emphasizes investments in the mechanisms of enforcement while simultaneously investing in supports to community level Child Protection advocates. Some good outcomes are seen in spite of this discontinuity.

"People now know that there are people in charge who will prosecute you. They will think

twice before doing anything to children."

Stakeholder, Sierra Leone 2006

EXCERPT


o … is ultimately a government responsibility, requiring long-term commitment to technical support to build capacity. Therefore, establishing the project within the MoI [Ministry of the Interior] has been invaluable. … There is excellent ownership of LEASETC within the MoI … has enabled the project to impact directly on high level government decisions, while working closely with the police responsible for the issue.

3.4 Constraints to enforcement

3.4.1 Enforcement is largely at the level of moral persuasion

Four reports here have good or better evidence quality ratings. These support findings that a stand out challenge for legislative investments is creating (a) the mechanics, and (b) the political possibility, of compliance with legal mechanisms.

Protecting Street Girls and Preventing Child Migration (Ghana 2003) — Shows limited attention to creating a broad “enabling environment” to sustain service delivery:

o The majority of Accra-based beneficiaries were victims of gender-based violence. At the same time, RPP [Rights Protection and Promotion Programme] had no explicit intervention to address this issue. There were no explicit interventions to make the legal, community, or service environment more responsive to the needs of victims. (p. 49)

Evaluation of Grassroots Peace-Building Project (Sudan 2004) — Suggests an over-reliance on legislation at the expense of how laws get enforced:

o The project could help children and young people monitor how customary law governing children and women’s rights in South Sudan is developed and implemented. Understanding and influencing legal practice is usually much more important than influencing legislation.

External Evaluation of the Child Protection Network (Cambodia 2004) — Is UNICEF investing more in form than function when it comes to putting laws into practice?

o The creation of networks and structures is necessary, but not sufficient to ensure improved child protection. Systems need to be functional, supported by the actors involved and oriented towards achieving results. This requires, in addition to a planning framework, a commitment from government to develop and apply policy and legislation.

Evaluation of the UNICEF Impact on the Juvenile Justice system (Serbia, Romania, Tajikistan 2006) — Call to strengthen accountability:

o The following are the main challenges that UNICEF should face in order to make its current and future programmes in support of juvenile justice reform more effective and successful:

Reinforcement of results-based management within the UNICEF Country Offices, with greater efforts to be deployed in order to develop: well-defined long term strategies; more precise and concrete objectives; relevant and quantifiable indicators; tools for data collection and monitoring systems. More

"...More emphasis should then be given to the evaluation of

impact or outcomes, rather than outputs."

CEE/CIS 2006

EXCERPT


emphasis should then be given to the evaluation of impact or outcomes, rather than outputs.


Chapter 4: Attitudes, Traditions, Customs, Behavior, and Practices The prevalence of FGC reported among daughters aged 0 – 10 years decreased significantly among women directly and indirectly exposed to the program.

The TOSTAN Program, An Evaluation of a Community Based Education Program in Senegal (2004)

It is the quintessential pressure group—an intergenerational alliance to prevent early marriage.

Bangladesh Early Childhood Marriage Prevention Program (part of the Adolescent Girls’ Livelihoods Program, 2005)

Introduction and Overview Addressing harmful attitudes, customs, behavior, and practices is a third component of the Protective Environment. In societies where attitudes or traditions facilitate abuse—for example, regarding sex with minors, severe corporal punishment, the application of harmful traditional practices, or differences in the perceived status and value of boys and girls—the environment will not be protective. In societies where all forms of violence against children are taboo, and where the rights of children are broadly respected by custom and tradition, children are more likely to be protected. Three evaluation reports26 are relevant to this component of the Protective Environment:

Early Marriage Prevention Programme, Endline Report on Kishori Abhijan: An Intervention Research Study on Adolescent Girls’ Livelihood (Bangladesh 2005),

Protection of the Girl Child and Eradication of Malpractices (Egypt 2002)

Community-Based Child Protection Programme Evaluation and Review (Somalia 2006)

Because of this paucity of data, UNICEF Child Protection obtained a Population Council evaluation report on UNICEF investments in programmes to stop female genital mutilation:

The TOSTAN Program, An Evaluation of a Community Based Education Program in Senegal (2004)

26 The askSam database tracks evidence on changing attitudes and practices across 35 other reports. This material falls under

other elements of the Protective Environment such as legislative reform, service delivery, and raising capacities of those closest to the child.


In the aggregate, the reports represent opposite ends of the evidence quality spectrum:

The Bangladesh report is the third most highly rated for evidence quality of the entire meta-evaluation. The Senegal report is also well validated. But only two reports out of the 59 that were rated for the meta-evaluation are rated more weakly than the Egypt and Somalia reports.

Good evaluation practice as well as good programme practice (see below) is a common element in the Bangladesh and Senegal investments. Both have large sample sizes, comparison groups, access to longitudinal data.

4.1 Investment characteristics that facilitate changed attitudes, practices

4.1.1 Intergenerational alliances, with peer counseling and health education

Early marriage

In the Bangladesh report, children’s and young people’s participation constituted the basis of a distinctive intergenerational alliance to prevent early marriages. Elders in the community relied on the young people as “eyes and ears” and followed up on news of a pending marriage negotiation with dialogue with the girls’ parents. This programme included peer counseling and health education.

Early Marriage Prevention Programme, Endline Report on Kishori Abhijan: An Intervention Research Study on Adolescent Girls’ Livelihood (Bangladesh 2005)

o Baseline and follow up studies (2001 and 2003) involving 5,024 adolescents found:

Project members paid less in dowry than non-project members.

Greater awareness of STDs.

Half of the negotiations attempted under the Early Marriage Prevention Programme were successful in delaying marriage.

To “avoid detection,” marriage negotiations were taking place with increasing secrecy27:

Advocating against marriage before the age of 18 years is a simple yet potentially effective community-based intervention undertaken by the adolescent members of CMES (Centre for Mass Education in Science). The adolescents, consisting of unmarried girls and boys, ally themselves with Adult Support Groups composed of respected elders in the community, both male and female. Initially a part of CMES’s Social Action program, advocacy against early marriage through this program was given a prime focus. … This intervention uses the presence and knowledge regarding marriages held by peer adolescents and the members’ networks to address the issue of early marriage while in progress.

27 In the previous chapter, this programme was presented as an example of stakeholders’ use of the threat of enforcement as a

tool of advocacy.

FINDINGS

Two reports represent the best in evidence practice with large sample sizes, comparison groups, and longitudinal data.


Female genital cutting/mutilation

Like the Bangladesh Early Marriage Prevention Programme, the Senegal programme to address and curtail female genital cutting situated girls’ involvement as peer educators within a sustained education and advocacy framework.

The TOSTAN Program, An Evaluation of a Community Based Education Program in Senegal

Baseline (2000) and Endline (200428?) studies involving about 800 men, women, and girls, found:

o The prevalence of FGC reported among daughters aged 0–10 years decreased significantly among women directly and indirectly exposed to the program. Life table analysis confirmed this change in the intervention group, but also that the girls who were cut were being cut earlier than before.

Methodologically,

o The basic education program consists of four modules: hygiene, problem solving, women’s health, and human rights. Through these four themes, emphasis was placed on enabling the participants, who were mostly women, to analyze their own situation more effectively and thus find the best solutions for themselves.

o Since 1997, TOSTAN has organized a public declaration by a large group of villages that have agreed to abandon FGC as a strategy to enable the people themselves to renounce a traditional practice without fear of social stigma.

4.2 Constraints to changing attitudes, practices


The primary constraint with the two remaining reports is their weak evidence base. The Somalia report says advocacy on sensitive issues like FGM had led to “some instances of changed practices around FGM” but this is not possible to verify.29 The evaluation further says that that community members “pick and choose” issues to engage with and there is no information on why efforts to encourage different thinking about female genital cutting might or might not be reaching people.

The Egypt report paints a picture of an essentially intractable set of attitudes about female genital cutting. It suggests that the project training lacked focus and needed to have been supported better by a

28 Timelines are unclear. 29 Child protection advocates in Somalia encountered resistance in three communities because of the lack of services being

offered by the CPCM [Child Protection and Community Mobilization] programme. Communities who rejected the programme stated that such activities were not a priority, and that they were capable of looking after their children. Alternatively, communities commonly stated that they had to focus on food, shelter, water, and education to assist their children, and there was a perception that child protection programming was irrelevant given that the community was lacking basic services and could not provide for children in their care.

Sixteen reports (27%) include reference to child protection needing to be embedded in baseline survival efforts such as water and food. See Chapter 11, “Cross-Cutting Issues,” for further details.

FINDINGS Good outcomes: an excellent evidence base supports findings of effective investments that embed youth-to-youth outreach within community-wide education efforts addressing sensitive topics (two reports).


substantive media campaign. These are generally stated findings, however, and it becomes unclear exactly where the trainings so terribly missed their targets.

Community-based Child Protection Programme Evaluation and Review (Somalia 2006)

o In instances where the CPAs [Child Protection Advocates] had discussed culturally sensitive information such as FGM, the evaluation found that many communities had generally been prepared to engage. Communities commonly referred to the use of relevant Koranic verses and cultural teachings which had changed perceptions, and in some instances changed practices, around FGM.

o In villages where communities continued to practice FGM, there was still a level of interest in other issues such as child labour and violence against children. Communities seemed to pick the issues from the CPS [Child Protection Study] that were of interest to them and would pay less attention to other issues presented. In most instances, the evaluation found that community disagreement with certain issues in the CPS did not discourage the community from engaging in the programme overall.

Protection of the Girl Child and Eradication of Malpractices (Egypt 2002)

o Some physicians, nurses, and health workers stated that the training duration (two days) … was not sufficient to address all aspects related to circumcision.

o Some trainers said that some girls require circumcision.

o Although they denied practicing circumcision, nine of a total of ten daya30s said that “circumcision is essential for some girls.”

o Two-thirds of the nurses (60 of a total of 91) said “circumcision is necessary for a small percentage of girls.” The rest of the nurses (31) said “it was not necessary for any girl.” (p. 21)

o The majority of physicians chose the phrase “circumcision is necessary for a small percentage of girls” (69 of a total of 82 physicians). Only 13 physicians said “circumcision is not necessary at all for any girl.” (p. 21)

30 This term is often translated as “traditional midwife”

FINDINGS A poor evidence base limits the instructional value of findings regarding outputs, for example, of trainings on FGC/M that missed their targets or of village advocacy efforts that are acknowledged and selectively ignored (two reports).



Chapter 5: Open Discussion and Engagement … level of public awareness about the policy of deinstitutionalisation is low.

Evaluation of the Deinstitutionalisation Project (Belarus 2004)

None of the programmes included a component aimed at “changing public attitudes towards children in conflict with the law and juvenile justice system reform.”

Juvenile Justice System Reform (Serbia, Romania, Tajikistan 2006)

The activities within the service have a high level of difficulty, because the child deinstitutionalisation culture with the purpose of returning them into the stable family environment does not exist yet.

Evaluation of the Development of Integrated Community Based Child Protection Services (Moldova 2004)

Introduction and Overview Open discussion and engagement with child protection issues is a fourth element of the Protective Environment. At the most basic level, children need to be free to speak up about protection concerns affecting them or other children. At the national level, both media attention to and civil society engagement with child protection issues contribute to child protection. Partnerships among actors at all levels are essential for an effective and coordinated response.

Within the universe of reports that constitute this meta-evaluation, this theme is one of the more meagerly represented elements of the Protective Environment. Readers may refer to the Figure 9, Appendix 6, to see the relative frequency of evidence on this theme.31 Information on what works and what does not has been tracked on the basis of five sub-categories:

Increase in public knowledge about the issue

Open discussion on sensitive issues

Use of media for advocacy work

Formal linkages to CRC are visible, part of the programme

Children feel able to speak and/or act more freely about protection concerns Judging from the manner in which knowledge/advocacy activities are typically situated within programme strategies and the generally lackluster manner in which impacts are monitored and evaluated, “open discussion” is not so much a by-product of other elements of the Protective Environment as, it would seem, a poor relation to them. Even the high visibility investments in juvenile

31 As previously stated, “relative frequency” was determined on the basis of the number of instances that any element of the

Protective Environment Framework Classification system received a programming practices quality rating.


justice reform in Central and Eastern Europe failed to include a component aimed at “changing public attitudes.”32 In the aggregate, this component of the Protective Environment:

Needs to distinguish among knowledge, awareness, and advocacy, and map strategies and expectations accordingly.

Would benefit from an assessment of good practices in programme communications, also known as C4D, Communication for Development.

Might consider that evaluation practice is a barrier to the identification of actual impacts. There is a general tendency to list activities and outputs and outcomes and impacts—suggesting among other things that impact analysis is not a high management priority.33

Much work is being done on the matter of harmful traditional practices such as early marriage and female genital cutting. However, for the meta-evaluation only four evaluation reports were available specifically on those topics (see previous chapter). At the same time, many reports describe broad “sensitization efforts” across issues like child protection; protection of orphans (orphaned because of war, HIV/AIDs, or natural disasters); street working girls/children; disabilities. Several standout examples of what works have been identified. That said, this is not a category where outcomes are much available. It must be emphasized that evaluation practice is a significant factor in this poor visibility.

Increase in Public Knowledge About the Issue

Synopsis

A finding for this sub-category of “Open Discussion” is that evaluators do not delineate what they mean by terms such as “knowledge” as distinct from “advocacy” or “awareness.” Reading either on or in between the lines, it is probable that these distinctions were not material to the project concept/design either. Why should this matter?

In order to advocate, the need is to bring new arguments and ideas and create awareness.

Raising awareness is a strategy for increasing advocacy.

Knowledge and awareness come together through information.

32

Juvenile Justice System Reform (Serbia, Romania, Tajikistan 2006). Chapter 4, “Attitudes, Traditions, Customs, Behavior, and Practices,” was limited to customarily difficult topics such as female genital cutting and early marriage. 33 The evaluator of the Eritrea report (2004), for example, says that “on the National Child’s Day in December 2000, advocacy

and sensitization on CRC was carried out; children were made aware of their rights and adults were sensitized on the same.” This report is highly rated for evidence quality but this is almost a comment made in passing. The issue here is not only a confusion of outputs with outcomes and impacts, it is that “public knowledge” was positioned as more or less a by-product of other activities.

FINDINGS Evaluators do not delineate what they mean by terms like “knowledge” as distinct from “advocacy” or “awareness.”


Evaluation reports with good evidence of increases in public knowledge possess at least a tacit understanding of the knowledge/information/advocacy nexus. There are five such reports in the meta-evaluation. In addition, these reports draw attention to the fact that what works:

Is not only the provision of information, but

Doing so in a context where related child protection issues are already on the table, and

Working through administrative channels that have gone some way toward maturation already.

5.1 Supports to increased public knowledge

5.1.1 Leveraging already mature networks/administrative mechanisms


The community based “child protectors” programme in Swaziland focused on protecting orphans and vulnerable children from abuse. This evaluation is well rated for evidence quality. According to the evaluator:

o The general awareness of children’s rights and child abuse is generally high and this is seen as a result of the initiative. … It is positive too that the areas where most abuse cases are reported are LL [Lihlombe Lekukhalela Child Protectors] areas (up to 100%). This is the one indicator that puts all but one LL area ahead of areas where the initiative has not been implemented.

The child protector initiative was essentially a sub-strategy of a larger effort coordinated through the Deputy Prime Minister’s office to address the needs of children in poverty and orphaned by HIV/AIDS. Child protector networks were thus born into a framework of already regularized partnerships and administrative pathways for community networking in support of this demographic. Two additional reports indicate a similar convergence among administrative capacities and standing networks, with the resulting potential for impacts in public knowledge. The first is well rated for evidence quality:

Evaluation of Disability Project (Cambodia 2005)

This programme operates within a context where the state has already been strongly involved in disabilities issues that also have a relatively high international visibility. According to the evaluation authors:

o There is ample evidence that all projects have had measurable effect in … reducing discrimination and promoting understanding of the rights of disabled people.

FINDINGS Good outcomes: three reports find a “measureable effect" of rights promotion efforts in communities with pre-existing child protection networks.

“It is positive too that the areas where most abuse cases are

reported are LL [Lihlombe Lekukhalela Child Protectors]

areas (up to 100%). This is the one indicator that puts all but

one LL area ahead of areas where the initiative has not

been implemented.”

Swaziland 2005

EXCERPT


Thao Dan Street Children Program (Vietnam 2002)

This project has been operating for ten years and is known for “sustained” educational outreach and communications addressing health and exploitative child labor. According to the evaluation authors:

o Our survey showed … higher awareness of needs and rights of street children in communities where Thao Dan has opened houses for street children.

5.2 Constraints to expanding public knowledge

5.2.1 Investments in awareness-raising without service or policy follow up In three reports, evaluators cite improvements in awareness without reciprocal changes as yet in budgets or policies. It would appear that the administrative apparatus necessary to transform awareness into action is still evolving. The first two reports here are well rated for evidence quality:

Evaluation of Street Children and Street Mother Programme (Ethiopia 2003)

Unclear whether plans are in place to address the gap between talk and serious action here:

o Of utmost importance in terms of contribution has been the huge strides made in consciountization [sic]of the plight of street children and mothers among the different strata of Ethiopian society. Although it has yet to reach the critical mass whereby it translates into the enactment of policies and the allocation of public budget, the awareness created has been instrumental in drawing the national attention to the phenomenon. The government has formulated a national action plan and the media has begun publicizing the plight of street children and mothers.

Report on Child Justice in Zambia (2005)

This situation—talk without action—has been reported already with respect to Legislation and Enforcement, and will come up again with respect to Service Delivery:


Mid Term Review UNICEF Romania Report on Child Protection Programme (2002)

Unclear about the plan to transform the talk into action:

o Institutionalised children and families and children at risk have received much attention and have become a high priority of the government. The hopes for system changes and better work-methods could be regarded as a positive outcome of all the efforts of UNICEF, official agencies, and NGOs during the last years. It is now important that these expectations will be met and transformed into sustainable programmes of community-based approaches to the welfare of socially vulnerable children and families in order to avoid a back-lash that could seriously slow down the process of improvements.

FINDINGS Constraints to outcomes: increased "awareness" without corresponding budget commitments or service delivery improvements (three reports).


Open Discussion on Sensitive Issues

Synopsis

Of immense importance to UNICEF, this is a poorly represented topic within the universe of reports that are available for this meta-evaluation: only five reports have been identified as having content that evaluators have flagged as sensitive. Three reports (see below) fall into the category of harmful traditional practices and thus were presented in the previous chapter. Fortunately, also as discussed in the previous chapter, one of these reports does describe a stand-out methodology that Child Protection may consider promoting as a good practice. With a call for evaluation reports addressing this topic, a fuller picture of investments could be obtained. The topics in this category are:

Early marriage (one report)

Female genital cutting (two reports)

Incest (one report)

Institutionalisation (one report)

5.3 Supports to facilitating open discussion

5.3.1 Intergenerational alliances, with peer counseling and health education

As reported in the previous chapter (Section 4.1.1), two reports with good evidence provide (a) replicable methodology and (b) a demonstrated good practice in giving voice to critical stakeholders across the age spectrum of a community (Bangladesh, Early Marriage Prevention, 2005; Senegal, TOSTAN Community Education, FGC, 2004). The Bangladesh report is the third most highly rated report for the meta-evaluation in terms of evidence quality.34

5.4 Constraints to facilitating open discussion

5.4.1 Lessons lost due to a weak evidence base As discussed in the previous chapter (Section 4.2.1), two reports with weak evidence contained information about investments on the topic of female genital cutting (Somalia, Community-Based Child Protection, 2006; Eradication of Malpractices, Egypt, 2002). In these reports, findings either cannot be verified or point to a weakly coordinated and conceived advocacy plan.

34 The first and second highly rated reports are Report on Child Justice in Zambia (2005), and Evaluation of the Street Children

and Street Mother Partnership Programme (Ethiopia 2003)

FINDINGS The meta-analysis finds a good outcome addressing “open discussion on sensitive issues”—built upon the use of a strong inter-generational alliance in conjunction with community education programmes. This is a methodology that Child Protection will find widely applicable.


Two additional reports describe investments on sensitive topics (incest and institutionalization) but both possess an uncertain evidence base:

Institution Building & Mainstreaming Child Protection (Indonesia 2003)

Incest is described as a “taboo” topic but details remain at the level of perception, for instance:

o The number of cases brought up in public and investigated by the police has considerably increased. The cases in 2004 include many more issues considered taboo (like incest) than before 2000. It is perceived by LPA [Child Protection Group] and their partners that public opinion regarding child rights has changed considerably during the last six years.

Evaluation of the Development of Integrated Community Based Child Protection Services Project (Moldova 2004)

“Child institutionalization” is described as a “delicate subject” and the author’s observation points to the need for open discussion where one apparently does not exist:

o The activities within the service have a high level of difficulty, because the child de-institutionalization culture with the purpose of returning them into the stable family environment does not exist yet.

Use of Media for Advocacy Work

Synopsis

This is nearly the leanest area of the whole “Open Discussion” component of the Protective Environment. In terms of the quality of the programming practices associated with it, “use of media for advocacy work” also ranks among the lowest in the meta-evaluation. Only six reports address media-related activities explicitly and evaluation reporting is a definite question mark: either the investments were not a focus so were mentioned only in passing, as outputs, or good practices for measuring impacts were not known to the evaluators.

5.5 Constraints to media/advocacy


Four of the least well rated reports for evidence quality are also those where media-related activities are flagged. The constraint these reports pose is that for whatever reason the evaluators focus on activities or outputs with no concrete analysis or evidence base addressing outcomes or impacts:

Evaluation of the UNICEF-supported Chikankata CBOSP & OVC Training projects (Zambia 2002) — Descriptive, without analysis of impacts:

FINDINGS Media-related investments are mostly limited to descriptions of activities and some outputs (six reports).


o In Lusaka, participants from Anglican Children’s Project are engaged in community mobilization and child abuse advocacy campaigns such as presentation of drama, theatre, and group discussions especially in markets in nine communities.

UNICEF Project on Preventing Trafficking of Women and Children, Gender-Based Violence, and HIV/AIDS (Bosnia and Herzegovina 2003) — Descriptive, without analysis of impacts:

o Two television videos on HIV/AIDS prevention were also produced for publicity.

Evaluation of District-based Initiatives on the Prevention of CSEC and Child Labour (Indonesia 2006) — Descriptive, without analysis of impacts:

o There is excellent evidence of advocacy and awareness raising of children’s rights (including a child’s right to education), the Child Protection Law, and the harm caused by child labour, through extensive media campaigns and community education initiatives … Although the consultant touched on this topic with local villagers, there was not enough time for the evaluation to fully assess the relevance or effectiveness of the media campaigns. However, from conversations at the local village there was evidence that the campaigns have had a direct impact on community, employer, and parental attitudes to child labour, which in turn has led to a decrease in the incidence of child labour (particularly the most hazardous forms), as well as a decrease in the number of children dropping out of school—and this is to be commended.

Institution Building & Mainstreaming Child Protection in Indonesia UNICEF Supported Child Protection Bodies (LPAs) (Indonesia 2003) — Unclear how conclusions were reached:

o Awareness has been raised through public campaigns, research and publications, rallies, and radio discussions. And to a large extent the greater awareness of the press has been achieved by including the media as stakeholders in LPA structures. Additionally, the training of media representatives has proven to be a very effective tool.

5.5.2 Viewer/reception expectations were misguided In two reports, the evaluators find that investments that were intended to produce greater awareness had contradictory impacts (Guyana), or the place of the media in the whole training process had not been fully thought through in advance (Egypt). These are two poorly rated reports for evidence quality:

Participatory Evaluation for the “Women and Children at Risk” Project of St. Francis Xavier (Guyana 2002) — An ineffective communications effort:

o The public was not as aware of the project as SFX initially thought. Most persons seemed to be more aware of the personalities they saw on TV than the message being sent out.

Protection of the Girl Child and Eradication of Malpractices Evaluation Report (Egypt 2002) — This investment seems to have been unable to put its finger on the roadblocks to changed attitudes (see previous chapter) yet calls for media investments:

o Recommendations … Activating the role of the media, particularly the television because of its influence on public opinion, and encouraging central and regional channels to raise the issue for discussion through drama and programs. Such measures would help raise awareness and create anti-circumcision social climate and public opinion.

"The public was not as aware of the project as ... initially

thought. Most persons seemed to be more aware of the

personalities they saw on TV than the message being sent

out." Guyana 2002

EXCERPT


Formal Linkages to CRC Are Visible, Part of the Program

Synopsis

If the field of reports that has been available to the meta-evaluation is anything to go by, the incorporation of the United Nations Convention on the Rights of the Child (CRC) is another facet of building a protective environment that lacks a clear status.

Five reports describe impacts that can be verified, where the CRC has been an intended mainstay of the program.

Nine reports describe the CRC as an intended program mainstay but with mixed impacts or impacts that are difficult to verify.

Three reports describe programs where the CRC should have been a mainstay, but wasn’t. As discussed in the introductory chapter, it is unfair to find evaluation reports wanting simply because the meta-evaluation has defined a set of priority topics and themes in retrospect. What is necessary here is to carry findings and observations forward into model Terms of References so that future evaluations address matters issues that Child Protection would most like to track.

5.6 Supports to facilitating rights awareness

5.6.1 Using the CRC to a programming lingua franca Impacts where the CRC has been a centralized component of the project are found in five reports, of which four are well rated for evidence quality. The findings show that the CRC helped to legitimate certain programming emphases, that it acted as a common framework in building advocacy efforts, and that it set the context and rationale for facilitating young people’s civic and political participation. 35

Protecting Street Girls and Preventing Child Migration (Ghana 2003) — CRC used to legitimate and organize investment emphases:

o … providing vocational training in support of Article 28 (right to education), primary healthcare through outreach clinics in support of Article 24 (right to health care).

Street Children Program (Peuan Mit) (Laos 2006) — CRC used to facilitate civic engagement and education:

o Peuan Mit facilitated inputs by children and young people into the drafting of a new Children’s Law, which is now being reviewed by the National Assembly.

35 However, it is impossible to say how much the evaluators’ own attentiveness to the CRC allows readers to see the

investments through the prism of the CRC, and whether, as a result, such realities are present but invisible in other programmes. A Model Terms of Reference might ask evaluators to take the measure of the Convention on the Rights of the Child in terms of its practical and rhetorical place in the programming design.

FINDINGS Where the CRC set the context and rationale of CP investments—explicitly—outcomes were good: activating partners' political commitments to children's rights and facilitating young people’s political participation (five reports).


Bam Emergency Programme (2003–2006) (Iran [Wiles et al] 2007) — CRC used to raise the visibility of children’s issues:

o An extremely positive achievement for UNICEF in their Bam earthquake response is that they succeeded in reminding policy makers and government of the importance of children’s rights in the response. Several senior civil servants told the evaluation team that UNICEF reminded the authorities that children’s issues were important … keeping children and their right to assistance following the earthquake on the policy agenda was an achievement.

Education as a Preventive Strategy Against Child Labour: Evaluation of the Cornerstone Programme of UNICEF's Global Child Labour Programme (Global 2003) — Helped to frame child labor issues around a common theme:

o In general, the Programme was considered relevant by participating countries in a variety of ways: activating their political commitment to the CRC …

Evaluation of emergency child protection programs in Bam (Iran [Lazo and Balanon] 2006) — Helped to introduced a rights-based approach:

o The experience promoted a paradigm shift within SWO from an orientation that separated and orphaned children should be immediately sent to orphanages to a rights-based approach that promoted fostering with extended family members.

5.7 Constraints to facilitating rights awareness

5.7.1 Investments are framed without reference to the CRC In three instances (the first two with good evidence quality ratings), evaluators say explicitly that mapping investments explicitly to the Convention on the Rights of the Child had not happened: in the monumental CEE/CIS juvenile justice reform process, in the peace building project in Sudan, and in Egypt’s urban child protection projects.

Report of the Organizational Development Assessment and Capacity Building of UNICEF Child Protection Partners in Somalia/land (Somalia 2005) — No buy-in around the CRC as an advocacy tool:

o … most (of the Child Protection Network partners) … confessed to have come together at the behest of UNICEF and not in order to address a need they had perceived to exist in the Children’s or Human Rights Sector and which required collective action. None of the Networks was found to have taken part in any advocacy or openly activist activities necessary for the challenge of impunity in the violation of children’s rights.

Evaluation of the Program “Improved Mechanisms of Child Protection” (Bosnia and Herzegovina 2003) — The civics of the CRC (participation, rights to information, non-discrimination) could have been leveraged but weren’t:

o Gender and human rights issues should be highlighted better and incorporated accordingly to reflect local requirements.

"An extremely positive achievement for UNICEF in their

Bam earthquake response is that they succeeded in

reminding policy makers and government of the importance

of children’s rights in the response."

Iran 2007

EXCERPT


Community-based Child Protection Programme Evaluation and Review (Somalia 2006) — An interesting challenge here to translate universal commitments in local cultural currency:

o CPC [Child Protection Committee] and community members stated that prior to the presentation of the CPS [Child Protection Study]36 they had not viewed issues such as child labour and other harmful practices as a problem; rather they were seen as common practice and a normal way of life … Whilst a handful of communities had claimed an increased understanding of children’s rights which they credited to the CPS presentation, the majority of communities viewed information on the Convention on the Rights of the Child and other human rights frameworks as ‘removed’ from their lives or as a ‘foreign’ concept.

Two other reports indicate that there was an intention to highlight or leverage the CRC to support children orphaned by war or HIV/AIDS, but that the degree of integration of the CRC was “minimal” (Tanzania 2004; Zambia 2002) and describes the position of the CRC in the programming as essentially ceremonial (Eritrea 2006).

5.7.2 Lessons lost due to a weak evidence base CRC-related ”sensitization” activities are claimed in three reports but impacts are difficult to verify due to the weak evidence base (Bosnia and Herzegovina 2003; Indonesia 2003; Indonesia 2006).

Children Feel Able to Speak and/or Act More Freely About Protection Concerns The status of child participation and involvement as represented in the reports that were available for the meta-evaluation may be found in the next chapter, “Children’s Life Skills, Knowledge, and Participation.”

36 UNICEF 2003


Chapter 6: Children’s Life Skills, Knowledge, and Participation The principle that children have the right to be heard, and to have their views taken into account, is one of the basic principles that underlies the CRC. However, thus far children have had little or no role in the design and implementation of the juvenile justice reform projects.

Evaluation: Juvenile Justice System Reform (Serbia, Romania, Tajikistan 2006)

Children surveyed at the “Drop in center” rated “Life skills trainings” below “Games,” “Dancing,” “Sport,” and “Education” in terms of services they like, but above “Clinic,” “Counseling,” and “Haircut.”

Street Children’s Progamme Evaluation (Laos 2006)

Introduction and Overview Children’s life skills, knowledge, and participation constitute a fifth element of the Protective Environment. Children are less vulnerable to abuse when they are aware of their right not to be exploited, or of services available to protect them. With the right information, children can draw upon their knowledge, skills, and resilience to reduce their risk of exploitation. Children also need to be provided with safe and protective channels for participation and self-expression. Within the universe of this meta-evaluation, the “Life Skills” component of the Protective Environment ranks fifth in size and in the quality of programming practices associated with it. Evaluation reports were analyzed on the basis of three sub-categories:

Insights into what children/young people know/need to know.

Relevant information/services are provided (e.g., rights trainings, free counseling, condom distribution, school curricula, child media, peer-to-peer activities).

Children/young people are facilitated to get involved (e.g., establishment of youth networks).

The latter category, participatory processes, carries about 25% less programming practice value than the first two categories. In the aggregate, this component of the Protective Environment:

Has room to grow where participatory design, implementation, and monitoring are concerned. Would benefit from assessing robust good practices in life skills delivery, monitoring, and

evaluation, and from higher standards of evaluation practice.


Can deliver good practices that will be valuable across a range of sectors. Should probably be reviewed for its relative degree of priority in Child Protection investments.

Insights into What Children and Young People Know and Need to Know

Synopsis

This category was set up to track information that would shed light on how program designers figured out what young people know and need to know. The results are extremely sparse:

Only three evaluation reports imply or are explicit in describing life skills investments as guided by direct consultations with children and young people.

Another three reports indicate that a proper needs assessment or outreach mechanism was missing.

This is an alarming finding. Two possible interpretations are:

Consultative processes with younger stakeholders are relatively de-emphasized at the design stage in building protective environment investments in reports that are available for the meta-evaluation, or

Evaluators and evaluation managers are not prioritizing analysis on this topic.

The remainder of this section and the rest of the chapter suggest that both interpretations are probably correct.

6.1 Finding out what young people know and need to know

Findings can be grouped into two themes:

Physical proximity to and regular contact between service providers and stakeholders sharpen programme relevance.

Nevertheless, top-down planning led Child Protection to miss the mark in several instances. This is the mirror opposite of the bullet point above, suggesting that in all probability a standard of programming practice has not been set or monitored effectively.

FINDINGS Regular contact between the service providers and the young people’s networks serves as a rolling needs assessment, but examples are few and no set standard of practice seems to be in play (six reports).


6.1.1 Regular contact acts as a rolling needs assessment In the three reports excerpted here, regular contact between the service providers and the young people’s networks acts like a sort of rolling needs assessment. The localized geographic settings (especially in the investments addressing street working children) may be significant in making this feedback loop work. The relevance of geographic proximity between young people and programme design and budget focal points for a program’s relevance and insightfulness would be a fruitful topic for further research. Of the following three reports, only the first report has a good evidence quality ranking:

Review of Street Children’s Project (Laos 2006) — Strong outreach efforts in a mostly localized area:

o The team maintains an intense schedule of work, regularly visiting 50 specific locations around the city. The outreach team was found to have good knowledge about individual street children, knew many by name and had developed trusting relationships.

Thao Dan Street Children’s Program (Vietnam 2002) — Strong outreach efforts in a mostly localized area:

o … To do this work, Thao Dan volunteers and staff rely on their network of street children in various locations to assist them in locating new and vulnerable children on the street so they can explore the possibility of reuniting the child with his/her parents before the street life truly takes hold.

Report on the Evaluation of the UNICEF-Supported Chikankata CBOSP & OVC Training Projects (Zambia 2002) — Suggests good networking with young people but without describing the terms of reference between programme designers and young people:

o Through drama and discussions, 19 cases of various types of child abuse have either been identified or reported to the outreach workers. Of the 19 cases, 10 are sexual abuse cases involving all female children ranging from 6 months to 18 years of age.

6.2 Constraints to finding out what young people know and need to know

6.2.1 Top-down programming missed vital details The excerpts here mostly speak for themselves: missing needs assessments limited service providers’ ability to be proactive in designing relevant supports. The first two of the following three reports have good evidence quality ratings:

External Evaluation of the Child Protection Network (Cambodia 2004) — Programme designers were operating more on instinct and less on fact:

o There is some concern that the CBCP [Community Based Child Protection Network] was established without undertaking a substantial assessment of child needs and protection issues in the target areas. As a result there is a lack of baseline data and information on children’s issues, which would ensure that interventions are focused and relevant.

"The [outreach] team maintains an intense schedule

of work, regularly visiting 50 specific locations around the

city." Laos 2006

EXCERPT

FINDINGS Poor outcomes resulted from top-down planning, bypassed needs assessments, and missed opportunities (three reports).


Impact Assessment of MVC Support Programme Implementation in Karagwe, Kisarawe, and Magu (Tanzania 2004) — Suggests a lack of contact between service providers and stakeholders who were supposed to benefit from the services:

o It seems that MVC are still not aware of their legal rights. The idea of legal rights seems to be new and unknown for most caretakers.

Evaluation of Psychosocial Rehabilitation of Children (Thai-Burma border 2006) — In the haste to provide services, a broad one-size-fits-all mentality meant that programme designers never had access to essential facts:

o Psychosocial programming … was made even before the Baseline Needs Assessment was completed. There is no indication that adjustments in the programming were made later. Some data gathered, particularly of coping techniques of children in the context of their culture … were not incorporated in the programming. For example, children were found to prefer to remain silent, to forget and refrain from expressing emotions related to their experiences as refugee children.

6.2.2 Lessons lost due to a weak evidence base Several additional reports describe investments in outreach workers or mobile teams who act as “eyes and ears” and deliver services. Unfortunately, the extent and timing of young people’s inputs was missing or impossible to infer. Because all of the reports for the meta-evaluation have been entered into a keyword-searchable database, “askSam” (see “Tools and Methods, Introductory Chapter) readers may use mobile* to tap into this state of play in the askSam database.

Relevant Information and Services Are Provided

Synopsis

"Life skills"… refers to a large group of psycho-social and interpersonal skills which can help people make informed decisions, communicate effectively, and develop coping and self-management skills that may help them lead a healthy and productive life. Life skills may be directed toward personal actions and actions toward others, as well as actions to change the surrounding environment to make it conducive to health. 37

The meta-evaluation finds one report addressing “relevant information and services” where the evidence quality is reasonable. 38 The overall picture, however, is perplexing:

37http://www.unicef.org/lifeskills/index_7308.html

38A boolean search in the askSam database facilitates access life skills-related information.

"…the community based child protection network was

established without undertaking a substantial

assessment of child needs and protection issues in the target

areas. As a result there is a lack of baseline data and

information on children’s issues…"

Cambodia 2004

EXCERPT


Whereas 17 evaluation reports contain information about life skills-related investments,

16 of these cannot deliver information on impacts because of poor evidence quality (seven reports) or for a variety of other reasons (nine reports).

It is impossible to know whether this situation stems from a relative de-emphasis within Child Protection on the impact analysis of life skills investments (and if so, why that would be the case)—the Terms of Reference were not available in too many instances to make this determination. In the opinion of this evaluator, life skills investments were generally treated in a taken-for-granted way in the evaluation reports: not exactly decorative, but not strategically worth much of a second glance.39

6.3 Supports to delivering relevant life skills

Findings can be grouped into two themes:

Relevance is facilitated by the ongoing contact between service providers and young people’s networks.

Life skills investments are unmonitored, missing, or un-ambitious. 6.3.1 Regular contact acts as a rolling needs assessment

Early Marriage Prevention Program, Girls Livelihoods Program (Bangladesh 2005) — The practical applicability of the life skills investments were important here (see also 4.1.1):

o Members were significantly more likely to report awareness concerning STDs and pregnancy.

o … the programs comprising Kishori Abhijan hoped to instill ‘life skills’ in the form of improved negotiating ability, increased information regarding reproductive health, etc. that can potentially improve the situation of new brides in their marital homes.

o Some of the qualitative data suggest that members were able to positively utilize their experience in programs to cultivate good relations in their marital home.

6.4 Constraints to delivering relevant life skills

In the following nine reports, the common elements are that life skills trainings were missing (two reports), were not monitored for impact (two reports), were constrained by service providers’

39 In a model Terms of Reference, Child Protection may consider the level of priority to be placed on impact analysis of life skills

investments, with supporting information on good practices in monitoring and evaluating them.

FINDINGS Whereas 17 evaluation reports contain information about delivery of life skills-related outputs, only one has a clear outcome with a clear evidence base to support it.

“… the programs … hoped to instill ‘life skills’ in the form of improved negotiating ability,

increased information regarding reproductive health,

etc. that can potentially improve the situation of new

brides in their marital homes.”

Early Marriage Prevention Program, Bangladesh 2005

EXCERPT


unfamiliarity with the materials (two reports), could have been taken to scale but weren’t (one report), or described in terms that are too generic to be informative (two reports). In reality, the elements described just now run through all the reports to some extent. The shared element in common is that, very clearly, they were not the evaluator’s main area of focus. Except for the last report in this section, all the reports excerpted here possess a good evidence quality rating. 6.4.1 Life skills component is missing

Evaluation Report on Child Justice in Zambia (2005) — Are life skills trainings “elective”?

o A planned training programme for 2003 that would have focused on the life skills programme did unfortunately not materialise due to funding constraints.

Report of projects aimed at the prevention of child abandonment in maternities (Romania 2003) — Trainings would have been vital:

o According to her own experiences, the social worker noticed that the family planning staff did not have sufficient patience to explain to their customers the mode of employment of the various family planning methods … One of the mothers, who chose contraceptive pills, took all of them at once, because she didn’t know what was the correct way of taking these pills. Also, many mentally retarded or mentally ill women were given contraceptive pills.

6.4.2 Training deficiencies constrain delivery

Evaluation: Protecting Street Girls and Preventing Child Migration (Ghana 2003) — Is there a de facto assumption that anyone can teach life skills?

o “… life skills is a new area … many staff … do not have the knowledge or skills to teach life skills or health education …”

o Life skills has been identified as a weakness in the programmes. While RPP (Rights and Protection Programme) is making progress toward the objective with regard to basic services, it is not making headway in enhancing life skills.

Community Based Reintegration programme (Sierra Leone 2006) — Trainings were insufficient and narrowly conceived:

o Unfortunately, by and large, the trainers had little pedagogical know-how and little experiences in teaching.

o … while there are some important life skills emerging from the *children’s+clubs, such as leadership and public speaking, the main activities are recreational.

6.4.3 No impact monitoring

Review of the Street Children Project (Laos 2006) — No impact monitoring: o A “’mobile school” includes trainings on life skills which have not been monitored for

impact. o A “Drop in” center offers life skills trainings but “sessions were not taking place at the

time of this review therefore no assessment of quality is possible.”

FINDINGS Across nine reports (nearly 10% of the meta-evaluation content), life skills trainings were needed but missing (two), were not monitored for impact (two), outcomes were constrained by service providers’ unfamiliarity with the materials (two), limited in scale (one), or described too generically (two).


o Children surveyed at the “Drop in center” rated “Life skills trainings” below “Games,” “Dancing,” “Sport,” and “Education” in terms of services they like, but above “Clinic,” “Counseling,” and “Haircut.”

Evaluation of the UNICEF Impact on the Juvenile Justice System Reform (Serbia, Romania, Tajikistan 2006) — No monitoring of life skills:

o No attempt has been made to measure any impact of the activities, projects, and strategies used by UNICEF on the empowerment of children and families, in any of the countries covered by this evaluation.” Examples of program models include:

o A non-custodial, community based programme (district level, Tajikistan) for the rehabilitation of juvenile offenders was established in 2004 … The services offered consist of a package that includes remedial education, vocational training, life skills training, family counseling, and free medical care … official statistics indicate that offending by juveniles has decreased … since this programme began.

o Community based alternatives (Serbia): there is ample anecdotal evidence that the life skills of some and perhaps many of the children in this facility have improved.

6.4.4 Not taken to scale

BAM Emergency Programme 2003–2006 (Iran [Wiles et al]2007)

o It seemed that expansion of life skills training to all children, which UNICEF started in 16 junior schools and high schools in 2006, would have been very relevant.


Education as a Preventive Strategy Against Child Labour (Global 2003) — The description is too generic:

o … it appears that several participating countries are incorporating life skills trainings (e.g., Cambodia, Panama, West Bank and Gaza), however:

o … critical dimensions linked to child labour, such as girls’ education, life skills education, HIV/AIDS, ECCD, health, nutrition, and water, environment, and sanitation (WES), did not receive adequate attention in the timeframe of this programme with the exception of a few countries.

Evaluation of the project on reintegration and rehabilitation of children associated with militia (Somalia 2006) — A generic and uninformative description:

o … non-formal training, combined with the acquisition of generic skills (applied non-formal education and life skills) and vocational skills is most appropriate and should be continued.


Children and Young People Get Involved

Synopsis

Of 59 reports that constitute the universe of this meta-evaluation:

Seven reports contain evidence about the efficacy of children’s and young people’s participation; the evidence quality is good in just four of these reports.

Six reports take note of participation without analysis of impacts.

Eight program evaluations found that child participation had not been incorporated but should have been.

Given the corporate emphasis on children’s participation, these statistics are alarming. Still, from the point of view of the meta-evaluation, is this glass half empty or half full?

Half full: at least 21 reports—one-quarter of all reports in the meta-evaluation—carry some mention of child participation.

Half empty: in just seven reports, child protection stands to obtain information on what works and what doesn’t regarding child participation in the service of building a protective environment. Only four of these reports have good or better evidence quality ratings.

6.5 Investment characteristics that support child participation

Findings can be grouped into three themes:

In three instances, the proximity of service providers and stakeholders facilitated participatory processes.

Nevertheless, in four instances children and young people were excluded at the level of programme design.

And in eight reports, children’s and young people’s participation is said to be missing altogether. 6.5.1 Close proximity of service provides and young people Three evaluation reports contain good practices for child participation, the first two with evidence quality ratings. It should be emphasized, however, that only in the first case (the intergenerational alliance to contest inappropriate early marriages [Bangladesh 2005]) were evaluators specifically looking at participatory methodologies.

FINDINGS 21 reports mention child participation but only 4 contain validated evidence and of these, most are output, not outcome, analyses.


Bangladesh Early Childhood Marriage Prevention Program (part of the Adolescent Girls’ Livelihoods Program, 2005) — The service providers are in regular touch with and close geographic proximity to the young people’s networks, with a very specific mandate:

o Girls and boys as “Peer Leaders” participate in an inter-generational alliance with respected elders and parents to advocate against early marriage.

o “It is the quintessential pressure group”

Street Children Program (Peuan Mit) (Laos 2006) — Regularized opportunities for feedback, with close geographic proximity between service providers and young people:

o Peuan Mit facilitated inputs by children and young people into the drafting of a new Children’s Law, which is now being reviewed by the National Assembly.

o At the Drop-in Centre, weekly meetings are organized between children and project staff to discuss ideas for improving services; children have elected a youth representative to act as a focal point for staff-child dialogue and to follow up on key issues.

Thao Dan Street Children’s Program (Vietnam 2002) — This program is well known for its outreach efforts:

o Outreach at Thao Dan is a huge effort led by ten student-age volunteers and directed by the “volunteer coordinator.”

6.6 Constraints to child participation

6.6.1 Exclusions at the program design and planning stages Four reports cite participatory processes that did not go far enough. The first two reports are well rated for evidence quality:

Sierra Leone Community Based Reintegration Program (2006) — Weak buy-in to leadership training and gender exclusivity are not explained:

o … while there are some important life skills emerging from the *children’s+ clubs, such as leadership and public speaking, the main activities are recreational and “Girls are grossly underrepresented.”

"At the Drop in Centre, weekly meetings are organized

between children and project staff to discuss ideas for

improving services; children have elected a youth

representative to act as a focal point for staff child dialogue

and to follow up on key issues."

Laos 2006

EXCERPT

"A weakness in almost all of the programmes is that children do not participate in the planning, or running of the programme."

Ghana 2003


Protecting Street Girls and Preventing Child Migration (Ghana 2003) — A typical model of top-down planning that is participatory only during the implementation phase:

o Child migrants returning from Accra have also been mobilized as “Anti Child Migration Peer Educators” in order to raise awareness on the risks and harmful effects of rural-urban migration.

o A weakness in almost all of the programmes is that children do not participate in the planning, or running of the programme.

Indonesia Psychosocial Interventions (2003) — Another design phase exclusion:

o The strengths of this design [Psychosocial Training Project] in West Timor were that it was comprehensive and multi-levelled—parents, teachers and youth volunteers trained by the project provided social support to beneficiaries, while counsellors and social workers also provided counselling for those needing more specialised assistance.

o Future projects should involve children in their design and implementation.

Indonesia Evaluation of district-based initiatives on the prevention of CSEC and child labour (2006) — Marginalized stakeholders are twice marginalized since participatory opportunities are not reaching them:

o In Indramayu children are being trained as facilitators and are training their peers on Children’s Rights, and protection from abuse of exploitation. However, it is observed that the network of this forum needs to be expanded to include children who are no longer in school and who are involved in, or at risk of being involved in, prostitution.

6.6.2 Investments are not framed with respect to participation Eight reports describe investments where child participation should have been built in. These include three community-based child protection programs and the very sizeable investments in juvenile justice reform in Central and Eastern Europe:

Community based child protection programs (Somalia, Cambodia, Egypt)

Iran earthquake emergency programming

Juvenile justice efforts in the CEE/CIS and Thailand

Peace building in the Sudan


At least 21 reports—one-quarter of all reports in the meta-evaluation—carry some mention of child participation. In just seven reports, child protection stands to obtain information on what works and what doesn’t regarding child participation in the service of building a protective environment. Only four of these reports have good or better evidence quality ratings.

EXCERPT

FINDINGS Eight reports describe investments where child participation should have been in-built. These include three community-based child protection programs and the very sizeable investments in CEE/CIS juvenile justice reform.



Chapter 7: Building Capacities of Those Closest to the Child

Community frustration and desperation to access basic resources proved to be the biggest barrier.

Evaluation: Community Based Child Protection (Somalia 2006)

An important outcome of all activities has been the establishment of a professional community of specialists who have special knowledge and skills to use modern technologies of work … Today in Belarus, the existence of the substitution families does not depend on the UNICEF funding.

Deinstitutionalisation and Alternative Placement of Children (Belarus 2004)

Introduction and Overview

Building the capacities of those closest to the child to recognize and respond to child protection issues is a sixth element of the Protective Environment. Parents, health workers, teachers, police, social workers, and many others who care for and live, deal, and work with children need to be equipped with the skills, knowledge, authority, and motivation to identify and respond to child protection problems.

Within the universe of this meta-evaluation, the “Capacities of Those Closest to the Child” component of the Protective Environment is the third most highly rated for the quality of its programming practices. It is also an area of dense aggregate activity evidence (see Fig 9, Appendix 6). Evaluation reports were analyzed on the basis of these three sub-categories:

a) Major actors are facilitated in recognizing child protection needs. o Family o Community/Civil Society Organizations o Professionals

b) Major actors are facilitated in responding to child protection needs. o Family—parenting, etc. o Community/CSOs o Professionals

c) Capacity development is becoming systematized and integrated, not ad hoc. Among these sub-categories:

The meta-evaluation tracks 31% more activity in the area of responding to child protection issues than in recognizing these issues.


Across both “response” and “recognition” categories, professionals’ capacity building saw the highest programme practices quality, with 19% more overall quality than investments in communities’ capacity building.

Capacity building of families was not an area of much emphasis in the reports that were available for the meta-evaluation, carrying about half the evidentiary information as the other two categories.

As a whole, efforts to strengthen recognition and response capacities of families, communities, and professionals have a higher programming practices quality by 11% than efforts to systematize capacity development.

Reminder: 6a, 6b, 6c40 refer respectively to enhancing stakeholders’ capacities

To recognize child protection issues (6a)

To respond to child protection issues (6b)

To have steady (as opposed to ad hoc) capacity building opportunities on both recognizing and responding to child protection issues

In the aggregate, this component of the Protective Environment:

Points to the importance of training programme content and delivery, such that length and/or quality of training and support make the difference, not just a difference. Combining results for families, communities, and professionals,

o Training is touted as reason for success in (a) 4 of 17; (b) 11 of 18; and (c) 4 of 14 reports.

o Inadequacies in training are said to impede success in (a) 5 of 16; (b) 13 of 24; and (c) 10 of 20 reports.

Is probably subject to some degree of selection bias. Certain data findings—the relative difference, for instance, in the number of reports (35 of 58) addressing subcategory 6(a); and 6(c), more information on what doesn’t work than on what does work—may be the result of what’s available for the meta-evaluation as opposed to the actual state of play across the whole of the CP investment effort in building capacities.

Ownership issues are most frequently reported as affecting project sustainability, and also impinge on recognition of and response to CP needs (6a and b), and occur at the level of the community and professionals as well as government institutions.

40 The numerics here refer to the codes that were assigned to this area of the Protective Environment Classification System (see Appendix Two).

FINDINGS • The meta-evaluation tracks 31% more activity in the “response” category than in the “recognition” category. • Across both “response” and “recognition” categories, professionals’ capacity building saw the highest programme practices quality, with 19% more overall quality than programming design and execution relating to communities’ capacity building. • Capacity building of families was not an area of much emphasis in the reports.


Major Actors Are Facilitated in Recognizing Child Protection Needs

Programmes like the Child Protection Teams are well-designed community based programmes using various community members to educate on the risks of child migration as well as other aspects of child rights.

Protecting Street Girls and Preventing Child Migration (Ghana 2003)

Synopsis

Data addressing this theme, “Major actors are facilitated in recognizing child protection needs,” suggest that raising awareness about rights and needs of vulnerable children is often an afterthought, secondary to actionable objectives of strengthening the capacity to respond to needs (see 6(b), next section). A total of 35 reports, 18 of which are rated credible in quality of evidence, address recognition of CP needs among major actors. The majority of the 18 projects target recognition on the part of communities and CSOs; smaller numbers target professionals and, fewer still, families. One might question, however, whether evaluators actively sought data in this category; is evidence of “recognition” less easily observed and measured than “capacity to respond” (investigated in 6b)? All 6a reports (total 35) are separated into two groups by quality of evidence (18 are good to excellent; 17 are poor to unacceptable).

7.1 Investment characteristics that facilitate recognition of child protection needs

7.1.1 Reaching families via adolescent livelihoods interventions

Promising approaches to adolescent livelihoods interventions are seen in two studies:

Adolescent Girls’ Livelihood (Bangladesh 2005), Early Marriage Prevention Program


FINDINGS Six reports show good outcomes signaling the importance to awareness- raising of community involvement in programme planning and implementation. Four reports show good outcomes linking rights awareness and community participation.


7.1.2 Reaching families/communities via involvement in programme planning

The importance of community involvement in programme planning and implementation is shown in six studies: the Community-Based MVP (Most Vulnerable Children) programmes in Tanzania (2004, 2004 and 2007); Reintegration and Rehabilitation of Children Associated with Militia (Somalia 2006); Protecting Street Girls and Preventing Child Migration (Ghana 2003); Adolescent Girls’ Livelihoods/Early Marriage Prevention Program (Bangladesh 2005); and Psychosocial Rehabilitation of Children Traumatized by Terrorist Violence (Algeria 2003).

Algeria 2003: the difficulty of earning the trust of the community is acknowledged together with the importance of doing so.

Ghana 2003: “Programmes like the Child Protection Teams are well-designed community based programmes using various community members to educate on the risks of child migration as well as other aspects of child rights.”

7.1.3 Reaching communities via capacity building programming

A positive feedback loop whereby needs recognition and capacity-building programmes reinforce each other is seen in:

Psychosocial Rehabilitation of Children Traumatized by Terrorist Violence (Algeria 2003): “Developing and supporting spaces for expression and rehabilitation for children and women has invited better community participation and involvement and more learning about community and culture.”

The Improved Mechanisms of Child Protection programmes (Bosnia and Herzegovina 2003): “… promoting wider civic involvement; including the involvement of vulnerable population as implementer rather than the recipient of these projects.”

Education as a Preventive Strategy against Child Labour (UNICEF/Global 2003): “Some countries were successful in gaining community support on eliminating child labour and child trafficking through members’ participation in village committees (Benin), forums (Indonesia), and community funds (the Philippines).”

7.1.4 Reaching professionals via training and agent-partner linkages

Training programs are cited as improving needs recognition in several studies: Improved Mechanisms of Child Protection (Bosnia and Herzegovina 2003); Social Work Training Programme (Cambodia 2005); and Mahzel Project for Social Reintegration and Protection of Disadvantaged Children as Future Development (Eritrea 2004).

Linkages among agents and participating partners facilitated Lihlombe Lekukhalela (LL) Child Protectors (Swaziland 2005): “Where strong linkages have been achieved with the police, Rural Health Motivators, clinic staff, and local schools, there is a high level of reporting and referral of cases to the relevant partners.”

“Programmes like the Child Protection Teams are well-designed community based programmes using various

community members to educate on the risks of child

migration as well as other aspects of child rights.”

Ghana 2003

EXCERPT


7.1.5 Reaching all stakeholders via rights awareness training

Four studies, all grounded in community participation, address the importance of rights awareness among all stakeholders: Street Children/Street Mother Program (Ethiopia 2003); Impact Assessment of MVC Support Programme Implementation (Tanzania 2004); Project for Social Reintegration and Protection of Disadvantaged Children (Eritrea 2004); and LL Child Protectors (Swaziland 2005).

Ethiopia 2003: “The stakeholders have become more conversant with the nature, extent, and ways of addressing the phenomenon of street life in the Ethiopian context. The project has been instrumental in creating awareness among teachers, students, and administrators about the plight of street children and mothers.”

7.2 Constraints to greater recognition of child protection needs

7.2.1 Abusive relatives not being engaged by community-based project

Abusive relatives not being engaged by community-based project, LL Child Protectors program (Swaziland 2005): “*The+ pattern is that abuse mainly takes place at home in the form of physical abuse most often at the hands of brothers, fathers, and stepmothers. To this end there appears to be a need to engage with these groups (fathers, brothers, stepmothers, and absentee parents) in particular.”

7.2.2 Communities/CSOs need more intense training

The need to improve or intensify training was cited in two community-based studies: MVC Support Programme Implementation (Tanzania 2004) and LL Child Protectors program (Swaziland 2005).

LL Child Protectors program (Swaziland 2005): “Training should be pre-planned as a series of interventions in each community rather than one or two contacts only. More sessions of shorter periods are required.”

7.2.3 Over-reliance on volunteers

Over-reliance on volunteers was cited in three studies: volunteer workers - Protecting Street Girls and Preventing Child Migration (Ghana 2003); and/or trainers - LL Child Protectors (Swaziland 2005); or services - Child Protection Network (Cambodia 2004).

FINDINGS Constraints to awareness raising include over-reliance on volunteers, "one-off" trainings, inaccurate situation analyses, and failure to engage the actual offenders.

"Experience in peer education programmes, which are largely

voluntary, suggests that volunteerism may be more

costly than anticipated." Ghana 2003


7.2.4 Weak front-end needs assessments

Inadequate assessments interfere with needs recognition in two studies:

o Child Protection Network (Cambodia 2004): Failure to carry out assessment of child needs and protection issues: “…there is a lack of information on children’s issues which would ensure that interventions are focused and relevant.”

o MVC Programme and Potential for Scaling Up (Tanzania 2007): Failure to follow up initial assessments.

7.2.5 Lack of community involvement in programming

Lack of community involvement in programming impedes CP awareness in three studies: Improved Mechanisms of Child Protection (Bosnia and Herzegovina 2003); Protecting Street Girls and Preventing Child Migration (Ghana 2003); and Community Based Reintegration (Sierra Leone 2006).

Community Based Reintegration (Sierra Leone 2006): “There was little or no consultation with the communities in terms of program design.”

7.2.6 Professionals’ training is inadequate

Inadequate training programs:

o Social Work Training Programme (Cambodia 2005): “A significant percentage of trainees are coming out of the course still believing that domestic violence is the fault of the woman. More training materials need to be developed.”

o Prevention of Child Abandonment in Maternities (Romania 2003): “Prior to the official launch of the project, the maternity staff should be trained on issues concerning child development, the rights and needs of the child, in order to better understand the sense and importance of the child abandonment prevention activity.”

7.2.7 Poor coordination among institutions and agencies

o Thematic Evaluation of UNICEF’s Contribution to Juvenile Justice System Reform (CEE/CIS 2006): “The good cooperation established with the relevant university departments should be further enhanced.”

o Street Children and Street Mothers (Ethiopia 2003): “Advocacy and social mobilization activities were not sufficient and sustained due to the pulling out of many of the implementing partners; inadequate coordination on advocacy and social mobilization among partners…”


All “recognition capacities” reports (total 35) are separated into two groups by quality of evidence (18 are good to excellent; 17 are poor to unacceptable).

EXCERPT

“Training should be pre-planned as a series of interventions in each

community rather than one or two contacts only. More

sessions of shorter periods are required.”

Swaziland 2005

EXCERPT


Strengthened Capacities for Responding to Child Protection Needs

Village committees, who provided bottom-line support for the most vulnerable children … “reported cases which they could not manage; this arrangement proved to be cost effective and also empowered communities in addressing the issue of orphans at a grass root level.”

Project for Social Reintegration and Protection of Disadvantaged Children (Eritrea 2004)

Synopsis

Data addressing “Strengthened capacities for responding to child protection needs” suggest that “top-down” (vs. “bottom-up”) strategies predominate in capacity-building CP programmes. Among reports showing instances of facilitation of capacities to respond (“What works,” shown below), 4 programmes—all dealing with street children—target families. In comparison, 12 programmes target capacity of communities and 13 target professionals. A similar pattern in “What doesn’t work” (families are targets of 1 programme, communities of 17, and professionals of 21) indicates that families are rarely targeted, despite the fact that those closest to the child are generally family members.

Community involvement in various phases of program planning, design and implementation appears to be a facilitating factor for recognition of and response to child protection needs by families and communities. Yet six studies suggest that over-reliance on volunteer workers, trainers, and/or services impedes CP recognition and response (6(a) and (b) combined). How can project planning and funding balance the efficacious effects of community involvement with volunteer fatigue?

All 6b reports (total 52) are separated into two groups by quality of evidence (25 good to excellent; 27 poor to unacceptable).

FINDINGS Data addressing “Strengthened capacities for responding to child protection needs” suggest that “top-down” (vs. “bottom-up”) strategies predominate in capacity-building CP programmes. Families are rarely targeted; most investments target professionals.


7.3 Investment characteristics that facilitate response capacities

7.3.1 Providing relevant supports (e.g., crèches) to families

Building a Sustainable Street Children Project (Laos 2006): “Substantial support has been provided to the families of street children in an attempt to address the factors that led children onto the streets.”

Protecting Street Girls and Preventing Child Migration (Ghana 2003): “…one of the most innovative forms of programming in the RPP portfolio are the crèches. These programmes are not high cost, but have the potential for ongoing contact with a large number of women and children.”

Bam Emergency Programme (Iran [Wiles et al] 2007): “UNICEF supported a family reunification programme with the Social Welfare Office of the government which was relevant, appropriate, and effective and resulted in the majority of children without primary care givers living in extended family-based care… This was probably one of UNICEF’s greatest achievements in its work in Bam.”

7.3.2 Rooting child protection efforts in the community

Evidence of success in four programmes where child protection is community-based:

o Most Vulnerable Children (MVC) Support Programme Implementation (Tanzania 2004): “MVC committees operate in villages which have organizational structures in the form of village governments. Generally the village government works well with the MVC committee in the village… Many objectives of the programme have been met because of efforts done by these committees.”

o Potential for Scaling Up MVC Programmes (Tanzania 2007): “Response systems successfully established at all levels, and particularly at community level, to facilitate provision of services to most vulnerable children.”

o Project for Social Reintegration and Protection of Disadvantaged Children (Eritrea 2004): Village committees, who provided bottom-line support for MVCs “…reported cases which they could not manage; this arrangement proved to be cost effective and also empowered communities in addressing the issue of orphans at a grass root level.”

o Community Based Reintegration (Sierra Leone 2006): Child Welfare Committees perform “…house to house sensitization of child rights violations in the community, registering vulnerable children and identifying a few who were able to get assistance to go to school.”

FINDINGS What works for CP response capacities: relevant supports (e.g., creches, basic needs), village-level CP committees, cross-departmental/sectoral trainings, hands-on social work experience (16 reports).


7.3.3 Community outreach makes a difference

Success related to community outreach is demonstrated in three programmes:

o Social Work Training Programme (Cambodia 2005): Four projects engaging disabled people in developing greater self-efficacy. “Irrespective of the approach adopted, the key is that the work is clearly grounded at community level and involves substantial capacity building of local actors.”

o Child Soldier Rehabilitation and Reintegration (Somalia 2006): Reaching out to local community through outreach and a focus on training of trainers (TOT) and outreach workers.

o Prevention of Child Abandonment in Maternities (Romania 2003): Mobilizing community resources for the prevention of child abandonment and for keeping the child together with his parents.

7.3.4 “Indigenized” models are effective

Two studies suggest “indigenized” models are more effective in local communities:

o Child Justice (Zambia 2005): “It is not sufficient to expose the functionaries in the system to information and examples. It is required that the model be ‘indigenised’ with its own set of local performance procedures and indicators.”

o LL Child Protectors (Swaziland 2005): “…communities request help in forming networks that can help to protect children more effectively using existing structures such as the traditional leadership.”

7.3.5 Facilitating agency-community cooperation

Benefits of cooperation between agencies and community workers: Prevention of Child Abandonment (Romania 2003): “The cooperation with the CDPRC (County Department for the Protection of the Rights of the Child) and the community healthcare worker supports the sustainability of the interventions even after the departure of the mothers…”

7.3.6 Training is a vital investment for building professionals’ response capacities

Positive effects of UNICEF-funded training are cited in numerous studies:

o Child Justice (Zambia 2005): “A body of knowledge and common understanding between magistrate and social welfare officers developed over time regarding appropriate sanctions. This was particularly the case where the role players concerned participated in the same training.”

o Psychosocial Rehabilitation of Children Traumatized by Terrorist Violence (Algeria 2003): “A large number of psychologists, psychiatrists, teachers, school counselors, and general practitioners have been reached by the training component. The outcome is the

“Irrespective of the approach adopted, the key [to the

programme's success] is that the work is clearly grounded at

community level and involves substantial capacity building of

local actors.” Cambodia [Social Work Training Programme] 2005

EXCERPT


emergence of an informal network of professionals active in the area of psychosocial care for children in general and with those affected by violence in particular.”

o Improved Mechanisms of Child Protection (Bosnia and Herzegovina 2003): “The MMBs [Municipality Management Boards] had by the time of the Key sources interviews in autumn 2004 been established in all five core municipalities, which is a significant achievement… This work was facilitated by the management training provided … and is in itself a novelty for the municipal planning systems.”

o Thematic Evaluation of UNICEF’s Contribution to Juvenile Justice System Reform (UNICEF 2006): “UNICEF has persuasively demonstrated its comparative advantage in providing valuable capacity building of the actors involved in juvenile justice reform. Most capacity building measures have consisted either of training, or the constant exchange of information and ideas that can be seen as a sort of informal mentoring.”

o Building a Sustainable Street Children Project (Laos 2006): “Through a combination of training, mentoring and on-the-job experience… the level of capacity achieved within Peuan Mit is particularly impressive given the lack of professional social expertise in the country.”

o Psychosocial Interventions in Gujarat (India 2002): There were myriad inconsistencies, but “Most teachers who were interviewed remembered the training process as unique in its delivery and content. The key session or process that was remembered was the “anxiety sharing” session wherein they were able to express what they had faced in the wake of the earthquake.”

7.3.7 Specialist capacity building has clear value

Social work experience gained in several programmes contributed to capacity-building:

o Deinstitutionalisation and Alternative Placement of Children (Belarus 2004): Establishment of a professional community of specialists and a network of institutions for prevention of social orphanage.

o Child Protection Network (Cambodia 2004): Specialist capacity building in sexual abuse cases.

o Improved Mechanisms of Child Protection Programmes (Bosnia and Herzogovina 2003): “As a result of the MMBs *Municipal Management Boards+ and other outputs, relevant stakeholders know one each other better and are more ready to work together.”

7.3.8 Advocacy and mobilization supports professionals’ response capacities

Thematic Evaluation of UNICEF’s Contribution to Juvenile Justice System Reform (UNICEF 2006): “…growing support for and participation in juvenile justice reform by national and local NGOs is due in very large part to UNICEF’s advocacy and mobilization efforts.”

“The MMBs *Municipality Management Boards] had …

been established in all five core municipalities, which is a

significant achievement… This work was facilitated by the

management training provided … and is in itself a novelty for

the municipal planning systems.”

Bosnia and Herzegovina 2003

EXCERPT


7.3.9 Giving the same trainings to all stakeholders:

Bam Emergency Programme (Iran *Wiles et al+ 2007): “The effectiveness of the project is largely due to the basic techniques that were taught to people in the counseling sessions. Teachers interviewed said that they had also used the techniques for relaxation and coping with difficulties themselves. Many parents believed that the briefing sessions helped them better understand the behaviour of their children.”

7.4 Constraints to strengthening response capacities

7.4.1 For young people, unsupportive parents are a constraint

Protecting Street Girls and Preventing Child Migration Programmes (Ghana 2003): “Almost all of the programmes mentioned lack of parental commitment or unsupportive parents, as the main barriers in implementing programmes.”

7.4.2 Families’ economic needs are unmet

Reintegration Programme for Children Associated with Fighting Forces (Liberia 2007): “Based upon the lessons learned 10 years ago … where many children left their parents again after placement due to economic constraints, it was advised to provide income generating support to the parents of the younger children. This recommendation was not taken. Funding could not enable this to happen.”

7.4.3 For communities/CSOs, an over-reliance on unpaid volunteers

Three studies show over-reliance on unpaid volunteers:

o Child Protection Networks (Cambodia 2004): One district working group said that in Cambodia “a volunteer is a person that volunteers to do something, but still expects some incentives for time spent on the tasks he or she is undertaking.” As a consequence the assessment found unpaid work (interviewing of children) to be of low quality.

o Potential for Scaling Up of MVC Programme (Tanzania 2007): “Members of the committees reported problems of the size of their villages and the workload—work for which they were considered volunteers and therefore without compensation.”

o Social Work Training Programme (Cambodia 2005): “Ability to sustain motivation and commitment among volunteers, as well as maintain and strengthen their technical skills and capacity, poses an on-going risk to projects relying on volunteer input. In this

FINDINGS Constraints to CP response capacities: disabling economic environment, inability to meet basic needs (e.g., water); over-reliance on unpaid volunteers and on non-professionals.

"Members of the committees reported problems of the size of their villages and the workload

—work for which they were considered volunteers and

therefore without compensation."

Tanzania 2007

EXCERPT


context, provision of allowances and training need to be considered essential core project costs.”

7.4.4 For communities/CSOs, an over-reliance on non-professionals

Three studies sound cautionary notes about relying on non-professionals:

o Child Protection Networks (Cambodia 2004): “The sensitive nature of sexual abuse requires special support and this expertise is not widely available. The evaluation team discovered that the majority of the girls are visited by men for follow up.”

o LL Child Protectors (Swaziland 2005): “In some areas the LL members do not know the limits of their powers nor the extent to which they are protected in intervene in case of abuse.”41

o Assessment of Potential for Scaling Up the MVC Programme (Tanzania 2007): “This form of programme delivery puts much of the onus on local committee structures which themselves frequently need strong oversight.”

7.4.5 For communities/CSOs, material resource constraints

Material resource constraints are reported by two studies:

o Community Based Reintegration Programme (Sierra Leone 2006): “All CWCs [Child Welfare Committees] visited complained that the absence of mobility, funding, and supplies significantly hampered their ability to perform their duties.”

o Assessment of Potential for Scaling Up the MVC Programme (Tanzania 2007).

7.4.6 For communities/CSOs, inadequate training

Inadequate training (three reports):

o Assessment of Potential for Scaling up the MVC Programme (Tanzania 2007): “…difficulty maintaining trained facilitators.”

o Building a Sustainable Street Children Project (Laos 2006): “Less training has been conducted with local government officials and capacity for understanding and responding to street children issues was found to be correspondingly lower.”

o Community Based Reintegration Program (Sierra Leone 2006).

7.4.7 For communities/CSOs, vertical communication breakdowns

Vertical communication and/or coordination breakdown between administrators or professionals and local agents (three reports):

o MVC Programme Assessment (Tanzania 2004): “The MVC coordination/facilitation team has not been receiving or provided with support from the district leadership due to communication breakdown. Communication breakdown resulted from among others, improper handing over of the MVC program from the previous District Development Officers (PDOs) to the incumbent.”

o Assessment of Potential for Scaling Up the MVC Programme (Tanzania 2007): Lack of coordination.

41 The writer probably meant to say “…extent to which they are expected to intervene in case of abuse” (underlining added).


o Community Based Reintegration Program (Sierra Leone 2006): Coordination between two implementing partners was weak “…when it came to distributing items and managing caseload… Children were confused as to which agency was carrying out what component and often came to the wrong one...” Also, community groups who received less attention from a social worker tended to lose interest and motivation.

7.4.8 For communities/CSOs, failure to build public awareness

Failure to build public awareness of services (two reports):

o Community Based Reintegration Program (Sierra Leone 2006): “Interviews with children demonstrated a lack of involvement of CWCs [Child Welfare Committees] in the day-to-day lives of children. Only 38% of children surveyed (both beneficiaries and controls) reported knowing of the existence of the CWC. It is clear that there is a disconnect between children’s awareness of the CWC and its intended functions.”

o Prevention of Child Abandonment Programme (Romania 2003): Evaluator suggests marking “…the launch / beginning of the project with a ceremony in order to increase awareness of the intervention model used, the values it is built upon, and the content of the services provided.”

7.4.9 Other constraints for communities/CSOs: unmet basic needs, ownership confusion

Other serious constraints reported by single evaluations:

o Communities express the need for basics (water, food) preliminary to embracing child protection issues: Impact Assessment of MVC Support Programme Implementation (Tanzania 2004): “The welfare of the MVC has worsened in the past two years because even the neighbors who were giving them food cannot do so now because they have food shortage themselves.” The same assessment also indicated that communities were impeded from responding to MVC needs because of the HIV/AIDS epidemic.

o CSOs exist only to collect UNICEF money: Child Protection, Capacity Building, Monitoring and Advocacy (Somalia 2005): “Though they were reported to have transacted some activities, most of them [CSOs] confessed to have come together at the behest of UNICEF and not in order to address a need they had perceived to exist in the Children’s or Human Rights Sector and which required collective action. None of the Networks was found to have taken part in any advocacy or openly activist activities necessary for the challenge of impunity in the violation of children’s rights.”

o Ownership issues: Assessment of Potential for Scaling Up the MVC Programme (Tanzania 2007): “…in many areas, the programme continues to be viewed as an externally driven programme with consequent lack of local commitment among stakeholders and facilitators at all levels and mobilisation of community participation is limited.”


7.4.10 For professionals, myriad training inadequacies

15 studies show myriad training inadequacies:

o One-off training inadequate: Psychosocial Rehabilitation of Children Traumatized by Terrorist Violence (Algeria 2003); Mahzel Project for Social Reintegration and Protection of Disadvantaged Children (Eritrea 2004); Social Work Training Programme (Cambodia 2005).

o Need opportunities to apply theory acquired during training: Social Work Training Programme (Cambodia 2005); Mahzel Project for Social Reintegration (Eritrea 2004).

o Specific training needs: Building a Sustainable Street Children Project (Laos 2006)—need for more general training and specific request to address drug use; Improved Mechanisms of Child Protection programmes (Bosnia and Herzegovina 2003)—Municipality Management Board members wanted “…training specially designed for them, more focused on management and development issues.”

o Psychosocial Interventions in Gujarat (India 2002)—“the training process for teachers was riddled with difficulties.”

o Lack of relevancy of HIV/AIDS education: Protecting Street Girls and Preventing Child Migration (Ghana 2003)—“Simply describing the modes of transmission does not take into account the reasons why young women are having sex. A large proportion of girls interviewed had experienced forced sex, an experience that is likely under-reported.”

o Training of trainers inadequate: Deinstitutionalisation and Alternative Placement of Children (Belarus 2004); Social Work Training Programme (Cambodia 2005); Reintegration and Rehabilitation of Children Associated with Militia (Somalia 2006).

o Language barriers in training: Psychosocial Rehabilitation of Children Traumatized by Terrorist Violence (Algeria 2003)—“…young psychologists complained that they could not understand some of the training material and lectures given in French. They were embarrassed however to show that to the trainer, since language is a marker of social status.”

7.4.11 For professionals, too much staff turnover

Staff turnover impedes capacity to respond to CP needs (three reports):

o Child Justice Reform (Zambia 2005): “Staff constraints and turnover continue to limit the ability and consistency of a child-friendly court.”

o Street Children/Street Mother Program (Ethiopia 2003): High level of turnover affecting advocacy efforts, also insufficient monitoring and evaluation of performance and impact.

o Child Protection, Capacity Building, Monitoring, and Advocacy Project (Somalia 2005): “… it is expected that all the relevant cadres of Child Protection Officers specifically and UNICEF generally would be familiar with all the elements and stages of the project. This however was not always the case on the ground.”

FINDINGS Training-related problems are cited as constraints to outcomes in at least 15 well-validated reports.

“Less training has been conducted with local

government officials and capacity for understanding and

responding to street children issues was found to be

correspondingly lower.”

(Laos 2006

EXCERPT


7.4.12 For professionals, lack of human resources and expertise

Lack of human resources and expertise (four reports):

o UNICEF’s Support to Mine Action (Global 2005): Noted need for specific expertise in Victim Assistance.

o Deinstitutionalisation and Alternative Placement of Children (Belarus 2004): “The number of employees is determined by local authorities; therefore, the workload on specialists is very high and varies by the districts.”

o Prevention of Child Abandonment (Romania 2003): “The social workers from the child abandonment prevention cabinet in the maternity units do not have the competencies needed in order to intervene in so-called ‘decided child abandonment’ cases.”

o Education as a Preventive Strategy against Child Labour (UNICEF/Global 2003): “The proposed internal capacity-building plan, which appeared to be practical and necessary, never got off the ground, as it required human and financial resources and commitment.”

7.4.13 For professionals, lack of material resources/transport

Lack of material resources, transport (three reports:

o Deinstitutionalisation and Alternative Placement of Children (Belarus 2004): Financial resources cannot provide technical support to the centres or ensure “mobility of specialists to travel to remote rural areas.”

o Community Based Reintegration (Sierra Leone 2006): “All CWCs *Child Welfare Committees] visited complained that the absence of mobility, funding, and supplies significantly hampered their ability to perform their duties.”

o Prevention of Child Abandonment (Romania 2003): Social workers have no financial means to check on the situation of children after birth: “It is not known whether the child is well taken care of, or whether he or she is neglected. The social worker does not have the financial means to check on the situation of the child, even by telephone, through the clinic or the city hall.”

7.4.14 For professionals, failure to integrate approaches

Failure to integrate approaches to the implementation of goals and tasks:

o Deinstitutionalisation and Alternative Placement of Children (Belarus 2004): “There was no uniform understanding of the principles of selection of children for social shelters, nor there was any integrated approach to the implementation of goals and tasks set before such institutions.”

o Education as a Preventive Strategy Against Child Labour (UNICEF/Global 2003): “No systematic capacity-building component was included in the strategic guidance to countries…”

7.4. 15 Lessons lost due to a weak evidence base

All “Response capacities reports” (total 52) are separated into two groups by quality of evidence (25 good to excellent; 27 poor to unacceptable).


Capacity Development Is Becoming Systematized and Integrated, not Ad Hoc

A body of knowledge and common understanding between magistrate and social welfare officers developed over time regarding appropriate sanctions. This was particularly the case where the role players concerned participated in the same training.

Child Justice Reform (Zambia 2005)

… the problems plaguing the implementation of the pilot programme persist: in many areas, the programme continues to be viewed as an externally driven programme with consequent lack of local commitment among stakholders and facilitators at all levels and mobilisation of community participation is limited.

MVC Impact/Scaling Up Assessment (Tanzania 2007)

Synopsis

Out of 59 reports, 41 addressed systematization and/or integration of the programme under evaluation. Note therefore, that nearly 20% of the evaluations failed to even address the sustainability of capacity development. And, of the 41, only 24 are rated with credible quality of evidence.

Twenty of the credible reports described some instance of failure to systematize (or some impediment to sustainability). Fourteen reports suggested some positive instance of systematization and/or integration leading to sustainability in capacity development. 42 These figures seem to indicate that significantly more of the projects evaluated are failing to become systematized than those that are. Is this truly the case, or are the figures artifacts of an evaluation methodology or culture that emphasizes shortcomings rather than factors leading to success?

Not surprisingly, evaluations suggesting sustainability of capacity development frequently attribute the facilitation of recognition of CP needs among stakeholders (factors in 6a) and the strengthening of capacity of major actors to respond (6b, usually professionals). Other factors said to be contributory include clear project aims and strategies and effective communication and linkages between participants and partners. All 6c Reports (total 41) are separated into two groups by quality of evidence (24 good to excellent; 17 poor to unacceptable).

42 Some of the evaluations report both a positive (“Works”) and negative (“Doesn’t Work”) instance. The same is true for 6a

and 6b.

FINDINGS The data suggest that significantly more investments remain ad hoc than those that are systematic. Is this truly the case, or is it an artifact of negative expectations as a culture of evaluation reporting?


In summary: What works—positive impacts on sustainability

Effective CP awareness raising, establishing a cohort of trained Child Protection professionals, and local ownership of programs (found in 14 reports, credible evidence ratings).43

What is not working—negative impacts on sustainability

Inadequacies of training programs (9 projects), ownership issues (4), lack of commitment to funding, poor coordination, unclear goals and strategies (in total, 20 reports with credible evidence).44

7.5 Investment characteristics that facilitate systematic capacity development

7.5.1 Integrating capacities development with awareness raising Successful building of awareness and recognition of CP needs is linked to systematization and/or integration of capacity development (three reports):

Rehabilitation/Re-integration of Mine Victims and People with Disabilities (Cambodia 2005): Concerted efforts to ensure the sustainability of project outcomes have involved “…raising awareness of disability and the rights of disabled people, building self-efficacy among primary care givers and groups of disabled people, mobilizing local authorities and commune councilors in support of responses, and strengthening referral linkages with specialized services.”

Evaluation of Street Children and Street Mother Program (Ethiopia 2003): “Huge strides made in consciountization *sic+… the awareness created has been instrumental in drawing the national attention to the phenomenon.”

Thematic Evaluation of UNICEF’s Contribution to Juvenile Justice System Reform (UNICEF 2006): “Within the relevant Ministries and other sectors, there are often departments or groups of professionals that do believe that juvenile justice needs to be recognized as a priority. Working with them can lead higher authorities to recognize this and make a commitment to juvenile justice reform.”

7.5.2. Professionals’ capacity building integrated into long term CP programming Strengthening of capacities of professionals is linked to systematization and/or integration of CP programming:

43 Another 12 reports proffer evidence of positive systematic capacities building efforts, however, the evidence base in these

was insufficient. Investments addressing the needs of street working girls, life skills, and harmful traditional practices, were among these 12 reports.

44Another 15 reports proffer evidence of negative impacts on sustainability but these have an insufficient evidence base.

FINDINGS What works—building blocks of consistency: effective CP awareness raising, establishing a cohort of trained Child Protection professionals,local ownership of programs (14 reports).


Long-term results of capacity-building of professionals in the juvenile justice reform projects reviewed in Thematic Evaluation of UNICEF’s Contribution to Juvenile Justice System Reform (UNICEF 2006):

o “The ultimate test of any programme on the rights of the child is how it impacts the lives of children and there is some evidence that interventions that have taken place in Romania, Serbia, and Tajikistan have had a positive impact on the lives of hundreds of children.”

o “Even in Montenegro, where very few changes in law and policy have reached the point where they are being implemented, there is evidence that the involvement of juvenile justice and law enforcement professionals in the process of reform has had a positive effect on the way they deal with children.”

o “UNICEF has persuasively demonstrated its comparative advantage in providing valuable capacity building of the actors involved in juvenile justice reform. Most capacity building measures have consisted either of training, or the constant exchange of information and ideas that can be seen as a sort of informal mentoring.”

7.5.3 Establishing a cohort of trained child protection professionals

Importance of establishing a cohort of trained CP professionals:

o Deinstitutionalisation and Alternative Placement of Children (Belarus 2004): “An important outcome of all activities has been the establishment of a professional community of specialists who have special knowledge and skills to use modern technologies of work… Today in Belarus, the existence of the substitution families does not depend on the UNICEF funding.”

o Psychosocial Rehabilitation of Children Traumatized by Terrorist Violence (Algeria 2003): “What is emerging as a result of interventions in repeated crises in the Algerian context is an informal network of professional mental health workers.”

o Thematic Evaluation of UNICEF’s Contribution to Juvenile Justice System Reform (UNICEF 2006): “Training on child rights has been institutionalized for judges, prosecutors, social workers who work in juvenile justice, and police officers.”

7.5.4 Developing local ownership Local sense of ownership is linked to integration of CP capacity development (two reports):

Protecting Street Girls and Preventing Child Migration (Ghana 2003): “This (success) is perhaps because the community itself has played an integral role in the development of the programme and sees it as consistent with community goals and values. Similarly, the Child Protection Teams are a community-based initiative, largely designed by the communities themselves…”

Rehabilitation/Re-integration of Mine Victims and People with Disabilities (Cambodia 2005): “Strategies for ensuring sustainability … are contingent of underlying assumptions about community or government buy-in… sustainability will be determined by how

“This (success) is perhaps because the community itself has played an integral role in

the development of the programme and sees it as

consistent with community goals and values. Similarly, the

Child Protection Teams are a community-based initiative,

largely designed by the communities themselves…”

Ghana 2003

EXCERPT


successful the partners are in guiding and supporting the transition of project activities to either community or government actors.”

7.5.5 Effective cooperation and division of labor between institutions Effective cooperation and division of labor between institutions (two reports):

Deinstitutionalisation and Alternative Placement of Children (Belarus 2004): “Each Oblast of Belarus has developed its own mechanism of interdepartmental cooperation for prevention of family problems and social orphanage… Socially important outcomes: Sustainable development of the foster families; and Cooperation between concerned departments, institutions and organisations at the municipal and regional levels in the field of the organisation of prevention of social orphanage.”

Mahzel Project for Social Reintegration and Protection of Disadvantaged Children (Eritrea 2004): “The clear understanding of the roles and responsibilities of each ministry and the smooth “transfer” of activities … has facilitated the implementation of Mahzel activities without disrupting the existing infrastructure.”


7.6 Constraints to systematizing capacities development

7.6.1 Training is limited to a one-time event (“one off”) A recurrent theme is that training programs need to be extended, with follow-up to ensure sustainable success (six reports):

o Reintegration Programme for Children Associated with Fighting Forces (Liberia 2007): “There is an urgent need to build a sustainable system that will ensure that business trainers remain in their contracts to provide follow-up training and mentoring…”

o Psychosocial Rehabilitation of Children Traumatized by Terrorist Violence (Algeria 2003): “A one-week training course on psychotrauma is hardly sufficient to answer all needs of mental health workers and provide them with necessary tools to intervene efficiently with traumatized children.”

o Grassroots Peace-Building Project (Sudan 2004): “Workshops are not usually enough to give people and organisations the skills, connections, and resources to do new things… The one-off training on mainstreaming is a hostage to staff turn-over: none of the people trained in peace-building in the Upper Nile MOE [Ministry of Education] are still there.”

o Training inadequacy is also reported in: Community Based Reintegration (Sierra Leone 2006); Bam Emergency Programme (Iran 2007); Social Work Training Programme (Cambodia 2005).

7.6.2 No quality control on trainings

Lack of quality control:

o Reintegration and Rehabilitation of Children Associated with Militia (Somalia 2006): “Training manuals were designed by each trainer and no system could be identified how these manuals were prepared, counter-checked, and how they were ultimately approved… “

7.6.3 Trainees have no practical work experience

New trainees require work experience in order to connect theory with practice (two reports):

o Social Work Training Programme (Cambodia 2005): “Longer-term impact on competence depends on the Social Workers regularly practising their skills (and having the logistical and material support to be able to do so)… Professionally, there is need for closer, long-term technical supervision and monitoring of Social Workers…”

FINDINGS Constraints to sustainable capacity development: trainings without follow up, ownership confusion, dependency syndrome, unpredictability of material supports, poor programme clarity, instability of conflict-affected areas.

FINDINGS Nine reports find training-related constraints, e.g., “The one-off training on mainstreaming is a hostage to staff turn-over."


o Community-Based Child Protection Programme (Somalia 2006): “The current approach of mentoring and ‘on the job training’ lacked the structure and resources to develop a satisfactory foundation in child protection to be able to carry out their tasks.”

7.6.4 Unmet needs and ownership confusion are not addressed45 Ownership issues impact negatively on sustainability (four reports):

Evaluations of the community-based Most Vulnerable Children Programmes in Tanzania (2004, 2004, 2007) note the persistence of perceptions that MVC Support is externally-driven:

o MVC Support Programme Implementation (Tanzania 2004): “…when it comes to ownership there is mixed feeling and many people believe that the MVC programme is a “UNICEF project”… Maximum participation of all community members should address the problem of sense of ownership and increase impact of the process.”

o MVC Impact/Scaling Up Assessment (Tanzania 2007):” …the problems plaguing the implementation of the pilot programme persist: in many areas, the programme continues to be viewed as an externally driven programme with consequent lack of local commitment among stakeholders and facilitators at all levels and mobilisation of community participation is limited.”

o Community-Based Child Protection Network (Cambodia 2004): “All participants in the CPN receive per diems for each activity conducted. While this is common in Cambodia, it raises concerns regarding the sustainability of the approach as well as the ownership of the programme.”

o Bam Emergency Programme (Iran *Wiles et al+ 2007): “It is believed that the recent changes in the management and staffing of MoE [Ministry of Education] in different levels can affect the “ownership” of the education programme by the government negatively since the new people need time and motivation to feel attached to the achievements of the project and follow up actions to fill in the gaps.”

7.6.5 Dependency syndrome Dependency and lack of commitment to funding post-UNICEF (two reports):

Social Work Training Programme (Cambodia 2005): “While MoSVY [Ministry of Social Affairs, Veterans and Youth Rehabilitation] commitment to the Social Work Training Programme has been encouraging, e.g., by providing human resources and the offer of a site for a Training Centre, there has been little financial commitment and the programme remains heavily dependent on UNICEF.”

Improved Mechanisms of Child Protection Programmes (Bosnia and Herzegovina 2003): “Ownership of the method/model must be passed to the local stakeholders but with it must come the responsibility of financing it, at least partially.”

45 This topic is addressed in depth in Chapter 11, Section 4, “Cross-cutting Issues/Intersectorality.”

FINDINGS Six reports find ownership confusion and dependency syndrome as constraints to good outcomes, e.g., "in many areas, the programme continues to be viewed as an externally driven…"

"... a regular request was made to provide incentives to the CSH

[Community Social Helper] in the form of rice or cash. One

district working group said that in Cambodia “a volunteer is a person that volunteers to do something, but still expects

some incentives for time spent on the tasks he or she is

undertaking.” Cambodia 2004

EXCERPT


7.6.6 Confusion among partners Division of labor and coordination problems between departments and institutions, as well as failure to involve critical partners, may decrease likelihood of sustainable success (four reports):

Lihlombe Lekukhalela (LL) Child Protectors (Swaziland 2005):

o “In some areas lack of coordination between various child protection interventions may cause confusion as people may duplicate activities or leave gaps in the belief that others are responsible…”

o “There is some criticism from national level stakeholders that their field officers have not been properly integrated into the initiative so they can work hand in hand with the LL. This will inevitably lead to a duplication of resources or competition for resources.”

o “The partnership with the police has to be strengthened… also partnerships with clinic and health center staff.”

Deinstitutionalisation and Alternative Placement of Children (Belarus 2004): “…further attention should be paid to differentiation of powers between the services and specialists, clear identification of the organisation which orders work with every specific family; development of criteria for the completion of the specific work by various specialists.”

Psychosocial Interventions in Gujarat (India 2002): “The focus is on capacity building without significant involvement of the Department of Primary Education for implementation.”

Thematic Evaluation of UNICEF’s Contribution to Juvenile Justice System Reform (UNICEF 2006): “In none of these countries has a concerted effort to involve the legal profession (i.e., lawyers) been made. The only exception can be found in Tajikistan, where UNICEF supported a little project providing legal services to juvenile offenders through a national NGO.”

7.6.7 Inadequate material support Inadequate material support (two reports):

Improved Mechanisms of Child Protection programmes (Bosnia and Herzegovina 2003): Most of the CSW directors “…reported in interview that the equipment received did not really meet their requirements and therefore can not lead to enhanced capacities. In some cases request for particular materials/supplies, which would have had a better chance of enhancing capacity, were not granted because they were not in line with UNICEF procurement/administration rules.”

Child Protection Networks (Cambodia 2004): “Material assistance … was provided on a one-off basis only. There is a lack of institutionalising support for children that are in need of long term support and assistance. The impact of follow up services provided are not well documented, but there are indications that the assistance provided through these visits is not effective.”

7.6.8 Unclear project aims and strategies

“All participants in the *Child Protection Network] receive per

diems for each activity conducted. While this is

common in Cambodia, it raises concerns regarding the

sustainability of the approach as well as the ownership of the

programme.” (Cambodia 2004

EXCERPT

FINDINGS Outcomes are constrained by unclear project aims and strategies, e.g., “The evaluation recommends the Programme adopt a series of guiding principles and indicative strategies that will not only help better define its strategic framework, but provide clearer guidance to potential partners as to what is expected in terms of project design (Cambodia [Disabilities] 2005).


Lack of clarity of project aims and strategies (four reports):

UNICEF 's Support to Mine Action (UNICEF/Global 2003): “UNICEF’s mine action support is not fully tailored to match UNICEF’s priorities, strategy, or programmes.”

Rehabilitation/Re-integration of Mine Victims and People with Disabilities (Cambodia 2005): “The evaluation recommends the Programme adopt a series of guiding principles and indicative strategies that will not only help better define its strategic framework, but provide clearer guidance to potential partners as to what is expected in terms of project design. Equally, resolving issues with how country level programming is translated into complementary and coordinated responses on the ground will significantly enhance program impact.”

Improved Mechanisms of Child Protection Programmes (Bosnia and Herzegovina 2003): “The best test to the effectiveness and efficiency of these boards will be the revision of the 2yr plans. …can *they+ revise the development plans so they really reflect, not only the local needs but also the changes in the political and legislative environment of social protection sector. This element of the program, given stronger guidance and improved linkage with policy bodies (cantonal, entity, and state) could ensure sustainability of the mixed welfare methodology.”

Child Justice Projects (Zambia 2005): “…it is therefore recommended that in order to kick-start a reform process that is explicitly aimed at improved service delivery, that an action plan with not more than ten key deliverables is widely consulted upon and that this be communicated thoroughly to all levels of government and civil society.”

7.6.9 Need to grapple with conditions in conflict-affected areas Two reports comment on difficulties of sustainability in conflict-affected areas:

Child Protection Networks (Somalia 2005): “…unless urgent steps are taken to mitigate their current weaknesses, the CPN’s *Child Protection Networks+ do not represent the best avenue for the protection of Children’s rights in Somalia. This is especially so since many of the violations are perpetrated in the name of religion or traditions which are powerful social forces. This reality is complicated further by the prevailing state of violence, which puts at risk any persons, or institutions that dare raise their voices against the violations.”

Reintegration and Rehabilitation of Children Associated with Militia (Somalia 2006): “Partners in conflict-affected countries, as the Somali example shows, often lack adequate organisational capacity since most of the capable people have left the country. Capacity enhancement of partner organisations and their staff should therefore from the outset be a key program component.”


Although 12 reports proffer evidence of positive systematic capacities building efforts, the evidence base in these was insufficient. Investments addressing the needs of street working girls, life skills, and harmful traditional practices were among these 12 reports.

Another 15 reports proffer evidence of negative impacts on sustainability but these have an insufficient evidence base.



Chapter 8: Services Delivery The skills training programme has managed to engage large numbers of children in learning vocational and business skills, but it has not always increased their employability. The apprenticeship programme however, seems to book good results.

Reintegration Programme Evaluation (Liberia 2007)

Capacity for program innovation limited by activity-based funding; little scope to respond to emerging needs and deviate from planned activities … Activity-based funding and reporting tends to fragment global strategies.

Cambodia Disabilities Project (2005)

The situation of girls confined in facilities for adult women has received scant attention in these projects … On the whole, the evaluation team believes that the attention paid by UNICEF to gender issues and gender sensitivity has been limited both during planning and implementation. (emphasis in original)

Evaluation: Juvenile Justice System Reform (Serbia, Romania, Tajikistan 2006)

Introduction and Overview The delivery of necessary and child friendly services is a seventh element of the Protective Environment. Child victims of any form of neglect, exploitation, or abuse are entitled to care and non-discriminatory access to basic social services. These services must be provided in an environment that fosters the health, self-respect, and dignity of the child. The services ultimately must become integrated into the overall social welfare system. Within the universe of this meta-evaluation, the “access to services” component of the Protective Environment is the second most highly rated for the quality of its programming practices. Evaluation reports were analyzed on the basis of three sub-categories:

Provision of necessary and child-friendly services

Child victims of violence, abuse, and exploitation facilitated to access these services

Strengthened social welfare systems

Across the whole of the meta-evaluation, no element of the Protective Environment saw more activity than “Access to necessary and child friendly services.” The quality of its programming practices was 26% greater than “Strengthened social welfare systems,” where the evidence across the meta-evaluation was about 20% less. Yet, while not well represented in the meta-evaluation, contributions to stronger social safety nets are substantial. A striking feature of the “Access to necessary and child friendly services” materials is that in spite of the relative density of evidence addressing this topic, evidence quality is a significant constraint: of 27


reports possessing information on service delivery efforts, only 15 rated credibly in terms of their evidence base. The category “Child victims of violence, abuse, and exploitation, facilitated to access these services” was de-limited originally to evaluations of programs addressing the needs of orphans and unaccompanied children. Roughly 10 evaluation reports fit this category, clustering in these countries: Eritrea, Iran, Ghana, Somalia, Swaziland, Tanzania, and Zambia. In three instances (Eritrea, Iran, and Tanzania) investments are found to be contributing to strengthened social safety nets and/or substantive policy shifts. Because of the many overlaps with the “Access to necessary and child friendly services” category, these two categories are treated as one in this chapter. In the aggregate, this component of the Protective Environment:

Demonstrates better impacts when the program focus is narrow (e.g., thematically discrete).

Can be improved by more sophisticated needs assessments, for instance, mapping vocational training to actual labor market requirements, and more focus on quality assurance monitoring.

May need to consider whether services are specified equally for the needs of girls as well as boys: a constant theme across the evaluation reports is that programs are paying insufficient attention to gender.

Lastly, evaluation reporting for this component of the protective environment can be a constraint because what actually got provided and to whom is often the least of an evaluator’s concern: disentangling who was supposed to do what, while meeting the many requirements of current global evaluation reporting protocols, means that reports can be unfocused.


Provision of Necessary and Child-Friendly Services

The decision to work on pilot projects while simultaneously working on law reform was a good one, because the success of pilot projects contributed to support for the draft law.

Juvenile Justice System Reform Impact Evaluation (CEE/CIS 2006)

The appropriateness of the training package was not based on training needs assessment and mapping of employment opportunities of the local market to absorb would-be graduates.

Evaluation: Reintegration and Rehabilitation of Children Associated with Militia Program (Somalia 2006)

Synopsis

Data addressing “Provision of necessary and child-friendly services” is not exactly an afterthought in the evaluation reports that were available for this meta-evaluation. However, as suggested above, evaluation reporting around this topic can be intensely cluttered and the bottom line is not always accessible. Delivery of services, even if the programme intended to produce them, is sometimes so constrained that the constraints become the evaluator’s focus rather than who accessed what services to what impact. The askSam database offers an excellent window into this state of play. As a result, in the universe of materials that were available for this meta-evaluation there is substantially more information on what doesn’t work—up and down the lines of coordination especially—than on what does work. Taking the foregoing into account, then, the meta-evaluation findings for this element of the Protective Environment are necessarily circumscribed. What works—supports to service provision

Specific themes in combination with targeted demographics (six reports with credible evidence; themes noted were street children, orphans and vulnerable children, and juvenile justice). 46

What is not working—constraints to service provision

Narrowly framed needs assessments with limited quality assurance mechanisms, overestimating volunteer capacities, and underestimating professionalization requirements (nine reports with credible evidence).47

1. 46 Effective service delivery investments are also seen in the context of natural disasters, a topic that is covered in

Chapter 10, “Child Protection, Emergencies.” 47

Poor or unacceptable evidence of investments that did not meet their intended targets was found in 12 reports.

FINDINGS Evaluation reporting on this topic can be intensely cluttered … constraints to service delivery can become so distracting that who accessed what services to what outcome can be hard to trace. The askSam database offers an excellent window into this state of play.


8.1 Investment characteristics that facilitate service delivery

The reports in this section share two feature in common: (a) the services to be delivered have either a single discrete demographic in a relatively localized geographic setting (e.g., urban street working children, orphans in a village community, youth offenders in targeted communities) or the thematic emphasis of the services is narrow (e.g., child abuse, housing, education, victim-offender mediation), and (b) either of these is combined with community outreach and awareness-raising. 8.1.1 Specific services for specific demographic, with community outreach

Street Children/Street Mother Program Evaluation (Ethiopia 2003)

This program had five primary components: education, health, micro-credit, housing, and awareness raising. Education supports were implemented in two specified localities:

o Number of street children and youth enrolling in formal education with the support of the project has reached 2,282, which is almost double of the originally envisaged plan.

o The program’s “more profound achievement” has been in raising awareness about street children … The introduction of the idea of partnership into the project has paved the way for opportunities to come together and co-create visions, strategies, and programs for joint action to create synergistic impacts [for street children and mothers].

Street Children, Program Evaluation (Laos 2006)

Project targets children who are “at risk” for entering street life:

o Community outreach activities … reaching approximately 450 children and young people each week at the time of this review.

o …most “at risk” children supported by [the program] to return to school did so after completing remedial classes with the mobile school.

FINDINGS What works in service delivery: an on-the-ground presence, a topic with wide inter/national visibility, steady/sturdy community outreach, and partners with a good track record (six reports).

"One of the main findings was that improvement in housing

has begun … this is a great innovation. Few programmes (in the history of the country) have ever been able to bring

changes in housing. The success in building and repairing some

MVC’s houses is a clear indication that it is possible to

address the problem of housing for MVC at village level.”

Tanzania [Magu] 2004


Impact Assessment of MVC Support Programme Implementation in Karagwe, Kisarawe, and Magu (Tanzania 2004)

Housing improvements are a singular success but the evaluator does not say why:

o One of the main findings was that improvement in housing has begun … this is a great innovation. Few programmes (in the history of the country) have ever been able to bring changes in housing. The success in building and repairing some MVC’s houses is a clear indication that it is possible to address the problem of housing for MVC at village level.”


Although service delivery here faced some constraints (see next section), UNICEF worked through community networks with external funding for education, clothing, food, medical supplies, and soap:

o The UNICEF/Most Vulnerable Child (MVC) programme provided support to an increasing percentage of children over time. By 2005 and 2006, almost 30 per cent of surveyed children reported support from the programme.

Lihlombe Lekukhalela (LL) Child Protectors (Swaziland 2005)

Greater gains where community members are involved in efforts to reduce child abuse:

o Generally LL [Child Protector] areas fair [sic] better in terms of reporting of abuse.

Juvenile Justice system Reform Impact Evaluation (CEE/CIS 2005)

The evaluator finds that “there is some evidence that interventions that have taken place in Romania, Serbia, and Tajikistan have had a positive impact on the lives of hundreds of children, or perhaps thousands.” Pilot projects offering specific types of services in localized settings helped to “win acceptance” for new, community-based alternatives to incarceration. Taking Serbia as an example:

o The decision to work on pilot projects while simultaneously working on law reform was a good one, because the success of pilot projects contributed to support for the draft law. The part of the project focused on community based alternatives provided considerable support to two quite different approaches: the mobile child rights teams designed to support preventive and rehabilitative services that focus on the adolescent and his/her family, and victim-offender mediation (VOM). VOM was a new concept in Serbia and is, by nature, an approach that is difficult to sell to the public and authorities, who often see it as being soft on offenders and failing to understand the importance of mediation to the victim and the utility of VOM in preventing recidivism. The decision to support these two complimentary approaches, and to make a long-term investment in winning acceptance of VOM in a carefully chosen pilot community, was a wise one.

8.2 Constraints to service delivery

8.2.1 No substantive accountabilities

Zambia Child Justice Program Evaluation (2005)

EXCERPT

FINDINGS


More awareness without tangible changes:48


Street Children and Street Mother (Ethiopia 2003)

Partners mobilized without administrative or institutional accountabilities:

o Had it not been for the numerous bottlenecks entangling the implementation process, the performance of the program could have ridden on a higher plane. The most prominent bottleneck with a pervasive effect on the implementation process had been the form of the partnership, which was characterized powerless [sic]entity and weak internal process. The non-existence of rules and regulations has preempted the practice of transparency and accountability casting very long shadows on the emergence of good governance that could have otherwise prevailed. As procedural and operational guidelines were not prepared, the performance of duties became hostage to the rule of thumbs; or at best, personal caliber and goodwill. To sum up, it is above all the absence of the major institutional elements in the partnership forged that underlie the curtailment of performance.


Piecemeal programme planning:

o There seems to be a need across UNICEF to examine inter-program and sub-program coordination. In this respect, the Child Protection Programme needs to take a closer look at how its supported activities dovetail with broader UNICEF programming, particularly with its education and social development sections.

UNICEF’s Support to Mine Action (Global 2005)

Limited accountability for outcomes:

o The effect that UNICEF support to mine action has had on the lives of mine/UXO affected populations remains insufficiently verified and UNICEF can do much more to develop adequate performance and quality management systems. … Quality control mechanisms are used more than quality assurance.


Training is not aligned to market realities:

o The appropriateness of the training package was not based on training needs assessment and mapping of employment

48 Also reported in Chapter Three, “Legislation and Enforcement.”

Constraints to service delivery: weak outcome accountabilities (eight reports).

“On a general level it can be concluded that an increased

awareness and sensitivity towards children’s rights in the

criminal justice system were observed in nearly all the sites

visited during the fieldwork. This has, however, not

translated into a general improvement in service delivery

(especially outside of Lusaka) and children are still subjected

to nearly all of the ills, misuses, and delays that were observed

in 2000.” Zambia 2005

EXCERPT


opportunities of the local market to absorb would-be graduates.

o The fact that no critical examination of the project proposal was made before approval raises questions as to how the project rational [sic] was identified, whether the various stakeholders have actually been involved in the identification and definition of the core-problem of children associated with militia. This question is directly linked to the issue of ownership.

Psychosocial Care for Children Traumatized by Terrorist Violence (Algeria 2003)

Limited outcomes accountabilities:

o Although most critical, the question about the outcome of activities and their impact in improving the wellbeing of children traumatized by terrorist violence remains to be answered.

8.2.2 Unassertive management

Reintegration Programme Evaluation (Liberia 2007)

Evaluator does not say whether management had an exit strategy or whether management was aware of this concern:

o The majority of schoolteachers have received psychological training that they found to be useful in their work with ex-CAFF and other children; they are concerned about the continuity of educational and psychosocial care and support activities after the project end.


Giving information without the means of acting upon it:

o The HIV/AIDs/STI information given in the context of the RPP programme was inadequate … the failure in the programmes was that promotion of condoms was weak or non-existent.

o …vocational programs reach a very small # of people … need for management skills to run a business

Child Protection Networks Program Evaluation (Cambodia 2004)

No explanation given for the neglect of a needs assessment:

o …There is some concern that the CBCP *Community Based Child Protection Programme] was established and expanded without undertaking a substantial assessment of child needs and protection issues in the target areas. As a result there is a lack of information on children’s issues, which would ensure that interventions are focused and relevant.


Fiscal rigidity weakened project’s responsiveness to new opportunities:

o Capacity for program innovation limited by activity-based funding; little scope to respond to emerging needs and deviate from planned activities … Activity-based funding and reporting tends to fragment global strategies.

FINDINGS More constraints to service delivery: weak response to emerging evidence of design flaws including an over-reliance on volunteers and an assumption that girls’ needs/ opportunities are the same as boys’ (eight reports).


Community Based Reintegration (Sierra Leone 2006)49

Singular, or “one-off “ investments with no long term plan:

o Many children stopped attending once CEIP [Community Education Investment Programme] had ended. The only reason that children cited for not attending was that they were unable to afford school fees.

8.2.3 Misunderstanding programme contexts

Reintegration Program Evaluation (Liberia 2007)

While every service discontinuity is unintended, giving selective assistance for whatever reason caused wide confusion:

o There are also emerging tensions around the issue of fairness of the programme. The provision of Transitional Safety Allowances, first to the children and later in the programme to the parent, gave the wrong signal that child soldiering pays, hence fostering feelings of unfairness among other war affected children and their parents.

o In addition, the targeting of the skills training programme on ex-CAFF (with a maximum of 10% of non-CAFF) has exacerbated these feelings of unfairness, especially because there are no such programmes available for other children and youth … Since there are hardly any other vocational training programmes ongoing in the country it can be concluded that the DDR programme has created an unfair balance of trained ex-CAFF in a situation where alternative training is not available to other children.50

Child Justice Program Evaluation (Zambia 2005)

Service delivery that inadvertently levels the playing field between victim and aggressor:

o Victims are also referred to the diversion programme by the magistrate. Two cases were reported (one gang rape and one defilement) where the victim and offenders were all sent to the life skills programme.


Investments do not free the community from dependency on UNICEF:

o An additional concern arises from the fact that the programme was seen to be a “UNICEF” programme, in spite of efforts to have support for most vulnerable children to be community-based. With many demands on communities for contributions for development programmes, especially those related to education and water, those programmes with the strongest political and governmental pressures take priority, and those with obvious external support may then be marginalised in local resourcing.

49 An earlier evaluation report addressing the needs of war-affected children in Sierra Leone, Evaluation of Alternative Care

programmes (2003), has a poor evidentiary rating with mostly piecemeal impact analyses. 50 These findings should be considered alongside findings addressing another aspect of the program: “The apprenticeship

programme however, seems to book good results. A relatively high percentages [sic] of apprentices remain with their masters, to do some self-employed work within their workshops, combined with school. This is a new approach that must be supported and potentially become one of UNICEF’s approaches for the reintegration of youth.”

"There are also emerging

tensions around the issue of fairness of the programme. The provision of Transitional Safety Allowances, first to the children and later in the programme to

the parent, gave the wrong signal that child soldiering

pays, hence fostering feelings of unfairness among other war

affected children and their parents."

Liberia 2007

EXCERPT


8.2.4 Inattention to girls’ needs, opportunities


Unclear why the programme had no plan to address an obvious concern for this target demographic:

o No explicit attention to gender-based violence … The majority of Accra-based beneficiaries were victims of gender-based violence. At the some time, RPP [Rights Promotion and Protection Programme] had no explicit intervention to address this issue. There were no explicit interventions to make the legal, community, or service environment more responsive to the needs of victims. In addition, partners of RPP were rather weak with regard to their response to and support of victims.

Community Based Reintegration (Sierra Leone 2006)

The de facto subjects of this investment would seem to be school-going boys:

o … While Children’s Clubs are functioning well, they are not inclusive of girls and children that do not attend school. Girls face a number of serious protection concerns that need immediate attention and action. …Program is very inattentive to “serious protection concerns (that) face girls.”


Another instance of a de facto gender bias:

o The situation of girls confined in facilities for adult women has received scant attention in these projects.

o On the whole, the evaluation team believes that the attention paid by UNICEF to gender issues and gender sensitivity has been limited both during planning and implementation.

8.2.5 Overestimating volunteer capacities


There is no information about project managers’ reactions to this situation, but the project budget was reported to be a serious constraint:

o This form of programme delivery puts much of the onus on local committee structures which themselves frequently need strong oversight.

o Members of the committees reported problems of the size of their villages and the workload—work for which they were considered volunteers and therefore without compensation.

“…Program is very inattentive to “serious protection concerns

(that) face girls.”

Sierra Leone 2006

EXCERPT



There was no information on project managers’ responses to the volunteer fatigue or ownership problem:

o A problem identified by partners in the initiative is the lack of ownership in some areas where members are still reliant of the donor for further assistance.

o Volunteer fatigue happens for a number of reasons. …the moment an employment opportunity occurs they must take it. …they have to supplement to food or clothing of neglected children at the expense of their families … in one area the LL states that the bulk of them are only available during certain seasons.

Child Protection Networks External Evaluation (Cambodia 2004)

Again, no information here on project managers’ responses or remedies:

o Based on the assessment, services have been provided to some children. However, the support was limited and mostly on a one-off basis.

o It is also problematic to base community based services to a large extent on volunteer contributions, in the form of CSH as well as the village social funds.


Strengthened Social Welfare Systems

The impact of the project activities extend beyond the initial objectives and has contributed to developments in mental health policies and services in general.


Synopsis

Signs of strengthened social safety nets were verified in five reports of investments supporting orphans, children in residential care, psychosocial, and juvenile justice reform. The main indicators were:

o Shift from institutional to family-based care for children without primary caregivers (Iran).

o Shift in mental health policies and services (Algeria). o Program lessons being used to scale up protection/support for orphan

and vulnerable children country-wide (Tanzania). o Reunification of orphan children and improving economic status of host

families (Eritrea). o Community based alternatives to detention (CEE/CIS).

Another four reports listed signs of strengthened social safety nets (three addressing juvenile justice investments, one for gender) but without a supporting evidence base. What were the primary constraints to building stronger social safety nets? Of 15 reports where information was found to be relevant (that is, one-quarter of the entire meta-evaluation corpus of materials),51 10 possessed a credible evidence base. Three constraints were identified:

o Poor strategic information led to inaccuracies in program concept/design. o Implementation was pushed ahead of institutional capacity building. o Short-term outlooks compromised sustainability.

51 Readers will recall, further, that the meta-evaluation addresses a fraction (40%) of the total Child Protection evaluation

reports that were produced between 2002 and 2007. For the sake of argument, if only 15 reports out of 147 comment on systemic change, the lynchpin of the Protective Environment Framework, this would suggest that only 10% of Child Protection evaluation reporting is turning up information on this foundational matter.

FINDINGS — Shift from institutional to family-based care. — Shift in mental health policies and services. — Pilot lessons used to scale up protection for vulnerable children country-wide. — Reunification of children and improving economic status of host families. — Community based alternatives to detention.


8.3 Investment characteristics that contribute to stronger social safety nets

Five reports possess credible evidence of shifts in the direction of strengthened social safety nets. These shifts show UNICEF investment in a highly contextually relevant manner and they are expressive, in the opinion of the evaluator, of a patient long-term outlook: 8.3.1 Context-driven, with long-term outlooks

Mahzel Project for Social Reintegration and Protection of Disadvantaged Children as Future Development (Eritrea 2004)

Creating an enabling economic environment:

o … 3703 families (103% of the set target) and 8589 orphans (107% of the set target) have been assisted to date.

o Up to 70% of the assisted families increased the capital value of their assistance.

BAM Emergency Programme Evaluation 2003–2006 (Iran [Wiles et al] 2007)

Contributing a substantive alternative during a time of crisis:

o The impact of the joint SWO (State Welfare Organization) and UNICEF programme has been significant. The majority of children without primary care givers are living with extended family — “family based care” … This is a major departure from the prevailing practice, if not policy, in Iran.

o The experience has also influenced the way that the State Welfare Organisation views childcare in the aftermath of disasters.


Helping to forge a common body of knowledge and practice:

o The impact of the project activities extend beyond the initial objectives and has contributed to developments in mental health policies and services in general.


Effective stewardship of a pilot programme:

o Many of the outputs from the MVC programme are currently used in the process of scaling-up protection and support of the MVC in the country. These include the various guidelines and training manuals.


Taking the long view with regard to a highly technical area of Child Protection:

o Despite the complexity and long term nature of the process precludes [sic] a simplistic comparison of the investments made with the results obtained, especially while the process is still ongoing, it can be affirmed that, on balance, the results are positive and there is reason to believe that these processes will continue moving forward.

FINDINGS What works to facilitate stronger social safety nets: creating an enabling economic environment, forging a common body of knowledge and practice (five reports).

"Up to 70% of the assisted families increased the capital

value of their assistance." Eritrea 2004

EXCERPT


8.4 Constraints to strengthening social safety nets

Ten reports had credible evidence to back up findings that certain practices were interfering with progress towards strengthening social safety nets. Three constraints were identified:

Skills training in a vacuum

Implementation pushed ahead of capacity building

Short term outlooks 8.4.1 Skills training in a vacuum The Liberia evaluation (2007) uses the phrase “enabling environment” to describe the context that, properly accounted for, might have increased the long-term value of vocational skills training investments:


o The skills training programme has managed to engage large numbers of children in learning vocational and business skills, but it has not always increased their employability (underline added).

o No efforts have been made to create an enabling environment for economic reintegration and the linkages required with other actors have been too limited to make a significant difference. … Follow-up of new businesses is weak and therefore there are failures that are already being seen.

o … the main areas of training are ‘traditional’ choices of vocation such as carpentry, cosmetology, and other typical skills that are already on the market and in high competition.


The evaluator calls for a greater emphasis on making the “programmatic environments” safe and supportive for the target demographic:

o Many programmes focus on providing services and skills and neglect creating a safe and supportive environment.

8.4.2 Implementation pushed ahead of institutional capacity building In five reports the evaluators found problems relating to sustainability: one-off investments, missing exit strategy, investments where the community infrastructure was not yet in place:


“One-off” spurts of assistance:

o Material assistance to [program] was provided on a one-off basis only.

o There is a lack of institutionalising support for children that are in need of long term support and assistance.


Concern that management commitments are not in place:

o Sustainability will be determined by how successful the

FINDINGS Constraints to developing stronger social safety nets: one-off investments, missing exit strategy, investments ahead of infrastructure or capacities (10 reports).

“Capacity for program innovation limited by activity- based funding; little scope to

respond to emerging needs and deviate from planned activities

… Activity-based funding and reporting tends to fragment

global strategies.”

Cambodia Disabilities Project, Evaluation 2005

EXCERPT


partners are in guiding and supporting the transition of project activities to either community or government actors. In this respect, while funding security is an important element, high level strategic advocacy and sound technical advice on planning and managing effective exits is probably more relevant and of greater value.


Community capacities and buy-in, and program resources, are unresolved:

o In programmes for most vulnerable children in Tanzania to date, support has been provided though structures of local committees and generally in-kind—clothing, school books, for example. The overhead costs of maintaining support and supervision for this type of delivery system are high.

o Weak commitment and motivation among the stakeholders and facilitators at all levels, especially at the lower levels (ward and community level), are major challenges in efficient realization of the programme’s outputs.

MVC Community Based Care, Support, and Protection in Musoma (Tanzania 2004)

Constraints in program financing and management capacities:

o The program implementation process at all levels is constrained by poor coordination; lack of commitment by the government leaders; lack of integration of the program to other development efforts; under-resourcing the programme; poor MVC data management and lack of follow-up; and lastly, lack of integration of the non-state actors in the implementation of the program (i.e., NGOs, FBOs)

Evaluation of the Program “Improved Mechanisms of Child Protection” (Bosnia and Herzegovina 2003)

Delivery of services seems to be an end in itself, the broader enabling environment (e.g., policy development) was ignored:

o … the program … is implemented in such way that its relevance is greater in the area of service delivery than in improved systems for developing mechanisms of child protection.

o The aspect that is almost absent from this program, yet would be of great importance to its success, is a link with the policy-making institutions …

o Of course, it is questionable whether systems as a whole are improving given that in some of the municipalities the system itself is scarcely operational. It might take several years until a clear improvement could be seen.

8.4.3 Small scale investments and limited time horizons Constraints include a small target population, an over-reliance on non-core funding, a misunderstanding of activity-outcome-impact relationships and timelines, and limiting activities to the development of symbolic measures such as national plans with no effective implementation strategy:


The evaluator comments on the limited reach of this basket of investments:52

52 Another report addresses investments that were too limited in scope to generate systemic changes: Street Children, Program

Evaluation (Laos 2006). The intended target population for this project is 500 street children.


o Most of the programmes in the RPP [Rights Promotion and Programming] portfolio reached a relatively small number of children.


The results chain is poorly understood here:

o …there is a need to come up with [a] Bridging Programme that can build upon the positive dynamics initiated through the DDR programme for children. If this cannot be assured, there are serious risks that the results of the reintegration programme will not be sustainable.

o Reintegration takes much longer than the remaining 7 months of the current programme with currently approximately 70% of the target group (9,100) working, in school or in training. The community structures that have been set up are good but need more support to become sustainable. The capacities of national actors to ensure that the positive results can be maintained are not fully in place.

Evaluation: UNICEF Support to Mine Action (Global 2005)

Ad hoc financial plan:

o Mine action support is dependent on non-core funding and is struggling to prove its outputs and outcomes in an environment where competing demands, both internally and externally with donors, jeopardise sustainability of its approach.

Education as a Preventive Strategy Against Child Labour (Global 2003)

Ad hoc activities:

o Programme activities … remain relatively discrete and isolated from each other … it remained to be seen whether it (the Programme) could get beyond the formulation of a national plan/programme to support the execution of the plan/programme.


Need for a plan to move beyond pilot investments:

o Sustainability concerns arise above all with regard to the pilot diversion/alternative sentence pilot projects. There are hopeful signs that the government will rise to the challenge of funding similar projects throughout the country.

“Reintegration takes much longer than the remaining 7

months of the current programme with currently approximately 70% of the

target group (9,100) working, in school or in training. The

community structures that have been set up are good but need

more support to become sustainable. The capacities of national actors to ensure that

the positive results can be maintained are not fully in

place.” Liberia 2007

EXCERPT



Chapter 9: Monitoring and Oversight After the mother leaves the maternity together with the child, no one monitors what happens next.

Evaluation of Projects Aimed at the Prevention of Child Abandonment in Maternities (Romania 2003)

With functioning databases the program/municipalities could have had a better base for evidence based policy making, which in turn would better inform the needs of each municipality.

Evaluation of the Program “Improved Mechanisms of Child Protection" (Bosnia and Herzegovina 2003)

Introduction and Overview Monitoring and oversight is an eighth element of the Protective Environment. A protective environment for children requires an effective monitoring system that records the incidence and nature of child protection abuses and allows for informed and strategic responses. Such systems can be more effective where they are participatory and locally based. It is a responsibility of government to make sure that every country knows the situation of its children with regard to violence, abuse, and exploitation. Within the universe of this meta-evaluation, the “Monitoring and Oversight” component of the Protective Environment is the lowest rated for the quality of its programming practices and ranks as the most notoriously deficient and frequently lamented investment area across the entire Protective Environment investment portfolio. Statistically, the Protective Environment investments that align to centralized political power—as expressed, for instance, in the areas of enforcement and building national partners’ capacities—are among the least likely to be monitored for quality assurance.53 Evaluation report contents were analyzed on the basis of five “monitoring and oversight” sub-categories:

Monitoring systems including data collection on CP issues

Establishment of participatory and locally-based mechanisms

Establishment of reliable reporting systems

Establishment of community surveillance systems

Information on sensitive subjects, e.g., closed institutions

53 This finding is congruent with the current global analysis of constraints to aid effectiveness. As discussed in the Conclusions

chapter, however, it is striking that evaluation report authors put their very dramatic findings regarding monitoring into no particular context (other than resource constraints).


In the reports that were available for the meta-evaluation, there was no instance of an investment relating to “Information on sensitive subjects.” The one instance of a participatory process is—at the other extreme—a replicable methodology and a good practice (see below). Additionally:

In 16 reports—almost one-quarter of the meta-evaluation content—monitoring-related information had a good evidence support base. Of these:

o Good investments are seen in seven instances. o Constrained investments are seen in nine instances.

Monitoring-related investments were described as missing altogether in half of the evaluation reports (29).

Essentially, then, through the lens of the reports that were available for the meta-evaluation, monitoring and oversight are working properly in just 7 of 59 of the meta-evaluation programmes. Indeed, across the whole of the meta-evaluation no element of the Protective Environment saw less activity and less overall programming quality than “Monitoring and Oversight.” To see just how poorly this component of the Protective Environment Framework is faring relative to other components in the context of the meta-evaluation, the quality of its programming practices has a value of:

26% less than the programming practices for “Open Discussion,” which is the next most poorly rated for programming practice quality.

41% less than the element with the highest programme practices ratings overall, “Strengthening Government Commitments.”

In the aggregate, the “Monitoring and Oversight” component of the Protective Environment:

Significantly obscures the child protection investments that are working and worthwhile and makes them impossible to distinguish from investments that are not working and are not worthwhile.

Shows limited managerial and donor commitment without specifying either’s operating environment.

Will not see improvements without de-mystifying measurement strategies, concepts, and rationales and making these more accessible and mandatory.

Would benefit from a good practice review of participatory approaches. This chapter itemizes what is known about what works and what is not working for Tier One findings— that is, findings that are supported by good to excellent quality evidence.


Monitoring Systems, CP Data Collection

The President of the Republic of Belarus instructed the Ministry of Labour and Social Protection to set databanks on graduates of children’s homes and state schools for children and to organise their follow-up after they graduate from such institutions through the regional social protection centres.

Deinstitutionalisation Project Evaluation (Belarus 2004)

The handing over of the project responsibility from one staff to the other within UNICEF Somalia was apparently not well done, which has resulted to a meager trail of documents. This combined with a limited ability for project monitoring meant that a professional and broad mentoring and monitoring of the project did not take place as would have been expected.

Evaluation of Reintegration and Rehabilitation of Children Associated with Militia Program (Somalia 2006)

Synopsis

This area of the meta-evaluation is almost barren, which is hard to understand given the emphasis on results based tools and methods across UNICEF as a whole. The assumption is that the poor showing on monitoring systems and Child Protection data collection is an artifact either of evaluation reporting priorities or the sampling of evaluation reports that were available for the meta-evaluation. For instance:

Only three reports in 59—five percent of the whole meta-evaluation content—contained well-supported evidence of workable investments in data collection and monitoring. The distinctive element in these instances is the commitment of managers and/or government partners. 54

Over one-third (23) of the reports reported that monitoring and data collection systems were missing altogether.

Moreover,

Child Protection can find lessons learned about constrained or failed monitoring and data collection investments in only seven reports. The main lessons are:

54 Four additional evaluation reports described workable investments but the evidence quality limits their usefulness for the

meta-evaluation.

FINDINGS : Only three reports in 59—five percent of the whole meta-evaluation content—contained well-supported evidence of workable investments in data collection and monitoring. The distinctive element in these instances is the commitment of managers and/or government partners.


o An “off the shelf” data collection system is far preferable to “build as you go” approaches.

o Management commitments –not management capacities—need strengthening. o Across the board, the “why” of monitoring, not just the “how” of monitoring, needs

reinforcing and demystifying.55

9.1 Investment characteristics that support systematic monitoring, data collection

9.1.1 Management or (high level) government commitment In the three reports where the evidence base is good, the driving factor behind the presence and functionality of monitoring and data collection systems is the commitment of the management or government partner. The epigram set at the beginning of this chapter (showing that monitoring commitments flowed directly from the President of Belarus) illustrates this point. This commitment, illustrated further in the following short excerpts, sets these programs apart from any others across the entire meta-evaluation:

Street Children, Program Evaluation (Laos 2006) — Commitment, ambition, and vision:

o Peuan Mit staff have tracked the progress of all children supported to return to family care and to school through regular follow-up visits to meet children, their teachers, and parents. … During a 12 month period … the reintegration team made a total of 1,796 follow-up visits—an average of 150 each month—to children in 28 different schools and as many different communities.

Deinstitutionalisation Project Evaluation (Belarus 2004)— Commitment at the highest level possible:

o The President of the Republic of Belarus instructed the Ministry of Labour and Social Protection to set databanks on graduates of children’s homes and state schools for children and to organise their follow-up after they graduate from such institutions through the regional social protection centres. (Attribution: UNICEF budgeted 213222.76 USD between 1999–2003 inclusive of support for the development of regulatory and legal frameworks)

MVC Impact/Scaling Up Assessment (Tanzania 2007) — Collective commitment:

o … the national guidelines and monitoring and evaluation system which resulted from the pilot phase of the programme are now used by collaborating partners who support programmes for most vulnerable children with resources from the Global Fund and from PEPFAR [President's Emergency Plan for AIDS Relief].

55 Another four reports also contain descriptions of constrained or failed investments, but the evidence quality is too uncertain

for these to be utilized for the purpose of lesson learned.

“During a 12 month period… the reintegration team made a

total of 1,796 follow‑up visits—an average of 150 each month

—to children in 28 different schools and as many different

communities." Laos 2006

EXCERPT


9.2 Constraints to systematic monitoring, data collection

9.2.1 Call for an “off the shelf” system Two reports that have a reliable evidence base suggest that UNICEF Child Protection might invest in an “off the shelf” data tracking system where issues such as search functionality, user-friendliness, and data-entry parameters can essentially be largely pre-set.

Mahzel Social Reintegration Project Evaluation (Eritrea 2004)

o The central office of the MLHW (Ministry of Labour and Human Welfare) has developed a simple database from which information can be retrieved; however this data base is totally dependent on reports sent from the region, which in turn, is dependent of information forwarded by the sub zobas.

o The ET (Evaluation Team) also notes that the Children Data Bank (Child Info data base) is a “complex” software and has not been applicable either at the central or regional level. Efforts were made by partners including UNICEF in setting up a child info data base by procuring supplies and equipment in the first year of project; staffs from the central and regional level were trained both locally and at UNICEF Regional Office Nairobi. However, several factors hindered the adoption of the database, one of them being the lack of retention of trained staff by the MLHW.

Bam Emergency Programme Evaluation 2003–2006 (Iran [Wiles et al] 2007)

o An area in which UNICEF has not been as effective as it might have been is in the development of a database to support the SWO (State Welfare Organization) programme. The database is currently being revised, as the first version did not really allow users to search for data. The main reason for this is that data entry was done on the basis of the case-note forms that contained as much qualitative data as quantitative.

o IT projects developed quickly under duress rarely work—a key learning point for UNICEF from this experience will be to devise an ‘off the shelf’ system similar to the one operated by the ICRC.

9.2.2 Management commitments (not capacities) are not evident Due either to evaluation reporting gaps or to gaps that are intrinsic to the programmes being evaluated, a strong managerial hand on the matter of monitoring and data collection is hard to detect. Further, the multiple ways that managers’ hands can be tied is not an explicit point for discussion in the evaluation reports. Three of seven reports with a good evidence base reflect on the matter of management commitments:

FINDINGS Constraints to CP data collection: — An “off the shelf” data collection system is far preferable to “build as you go” approaches. — Management commitments—not management capacities—need strengthening. —The “why” of monitoring, not just the “how” of monitoring, needs reinforcing and demystifying.

" IT projects developed quickly under duress rarely work—

a key learning point for UNICEF from this experience will be

to devise an ‘off the shelf’ system…"

Iran 2007

EXCERPT



o The handing over of the project responsibility from one staff to the other within UNICEF Somalia was apparently not well done, which has resulted to a meager trail of documents. This combined with a limited ability for project monitoring meant that a professional and broad mentoring and monitoring of the project did not take place as would have been expected.


o The national level of MOSALVY was insufficiently involved in the development and implementation of the CBCP [Community Based Child Protection Networks] … Monitoring and evaluation conducted by the national level are superficial and lack analysis.

MVC Community Based Care, Support, and Protection in Musoma (Tanzania 2004)

o …break-up of program implementation facilitation and monitoring. District facilitators are not following-up or monitoring the implementation of the program. For instance in Isaba, village community dialogue and MVC identification was done in the year 2002, and the MVC committee is still waiting for instructions from the district facilitators on how to proceed from that point…

9.2.3 The “why” of monitoring—not just the “how”—needs reinforcing Of the seven reports where the evidence quality allows for a reasonable degree of certainty, three describe difficulties that stem from a fundamental lack of understanding of the rationale of monitoring and data collection. As the following excerpts suggest, the justification for watchfulness and follow through is not instinctive. In place of a coherent framework where expected outcomes and impacts are attached to inputs, investments seem inconsistent, free-floating, and unscientific:


A need for technical training in results based tools and methods is evident here, but an appreciation of its rationale must also have been lacking:

o As the outcomes and outputs required by the UNICEF evaluation framework are not separately identified … their indicators are necessarily mixed in with the indicators for the outcomes.

o The indicators are not well distinguished from the means of verification. Sometimes suggested means of verification are actually indicators not covered elsewhere…

o The indicators and means of verification are almost never expressed in a quantifiable way: for example “Number of participants included in the network” does not actually express a minimum acceptable number, which makes it impossible in principle to actually use the logframe to measure program success.

“The handing over of the project responsibility from one

staff to the other within UNICEF Somalia was apparently not

well done, which has resulted to a meager trail of

documents.” Somalia 2006

EXCERPT

“The indicators and means of verification are almost never

expressed in a quantifiable way: for example “Number of

participants included in the network” does not actually

express a minimum acceptable number, which makes it

impossible in principle to actually use the logframe to measure program success.”

Bosnia and Herzegovina 2003

EXCERPT


Evaluation of Projects Aimed at the Prevention of Child Abandonment in Maternities (Romania 2003)

The rationale for follow up was missing:

o After the mother leaves the maternity together with the child, no one monitors what happens next. It is not known whether the child is well taken care of, or whether he or she is neglected. The social workers think that the Bucharest district Departments for the Protection of the Rights of the Child are not sufficiently involved and do not have a coherent strategy for the field monitoring of mothers at social and child abandonment risk.56

o In order to better evaluate the activity of the “counseling cabinets,” we need to define / delimit and monitor certain result indicators, and also to conduct a long term study evaluation of the results and of the impact of counseling on the child’s welfare.

Appreciative Assessment of Psychosocial Interventions in Gujarat (India 2002)

An “off the shelf” system would be useful when the need to respond is sudden, such as the aftermath of a natural disaster:

o Most importantly the complete process requires predefined impact assessment systems. The monitoring using the existing educational infrastructure would be helpful but some effort is needed to monitor indicators like child labor, trafficking, and second-generation commercial work.


Only three reports in 59—five percent of the whole meta-evaluation content—contained well supported evidence of workable investments in data collection and monitoring.

Four evaluation reports described workable investments but the evidence quality limits their usefulness for the meta-evaluation.

Over one-third (23) of the reports reported that monitoring and data collection systems were missing altogether.

56 Although this is another instance of evaluation findings resting on what people say, the author communicates the contexts

usefully.


Participatory and Locally Based Mechanisms

Synopsis

This aspect of monitoring and oversight is not well represented in the reports that were available for the meta-evaluation. In fact:

Only one report in 59 shows a good practice in combination with a good evidence base.

There were no reports of investments in participatory mechanisms that went awry; instead, five reports in 59 cite the absence of participatory mechanisms as a missed opportunity.

9.3 Investment characteristics of functional participatory, locally based mechanisms

9.3.1 Strong intergenerational alliance at the community level The single report of a viable investment in participatory and locally based mechanisms has been presented in previous chapters: the Early Marriage Intervention Program, a sub-programme of the Girls’ Livelihoods Programme (Bangladesh 2005). The intervention encouraged adolescent girls to work with community leaders to advocate on behalf of a bride-to-be to convince her guardians not to commit her to a marriage that was inappropriate. Young people act as a community’s “eyes and ears” and elders with some degree of influence do the negotiating with other decision makers in the community.

9.4. Constraints to participatory, locally based mechanisms

9.4.1 Failing to recognize and make good use of obvious opportunities Five reports indicate that a failure to utilize participatory and locally based mechanisms resulted in missed opportunities. These pertain to investments in the following areas:

Street Children (Laos)

Juvenile Justice Reform (CEE/CIS)

Psycho-social/Emergencies (Iran, Indonesia)

Institutionalization


Reliable Reporting Systems

Synopsis

This is not a well represented area in the reports that were available for the meta-evaluation. In fact,

Child Protection stands to gain lessons learned about reliable reporting from just three out of 59 program investments.

In two of these instances, the investments were not successful largely because programs were established before reporting capacities/protocols were put into place.

9.5 Investment characteristics of reliable reporting systems

9.5.1 High level government involvement In the case of supports to most vulnerable children, primarily children orphaned by HIV and AIDS, the Tanzania evaluation implies that a reliable reporting system was in place, and that it had become a resource in broadening the scale and definition of the programme:


o Many of the outputs from the MVC programme are currently used in the process of scaling-up protection and support of the MVC in the country. These include the various guidelines and training manuals…

However, this same evaluation also reports near epidemic-levels of coordination problems and a lack of managerial follow through.

9.6 Constraints to reliable reporting systems

9.6.1 Programmes implemented without reporting plans Two reports find that programs had been established before reporting capacities/protocols were put into place:


o With functioning databases the program/municipalities could have had a better base for evidence based policy making, which in turn would better inform the needs of each municipality.


FINDINGS Three reports in 59 contain validated findings about reporting systems. Of these, two were not successful because programs were established before reporting capacities/protocols were put into place.


o UNICEF’s culture of accountability is one of its strengths, but juvenile justice reform is a relatively new area for the organization and limited in-house experience and expertise have had a negative impact on strategic planning, the development of indicators, the monitoring and evaluation of implementation.

o In general terms, results-based management needs to be strengthened … the objectives of juvenile justice reform are often defined in rather vague terms, while indicators, when identified, are mainly non-quantifiable and of limited use in measuring progress towards the pertinent objective; data collection, monitoring and supervision have not been given the necessary attention…


Five additional reports cite investments in reporting systems (two successful, three unsuccessful). However, in these cases the evidence quality is poor to unacceptable.

Community Surveillance Systems

Synopsis

Two evaluation reports carry reliably supported evidence of community surveillance systems. One additional report, also with a credible evidence base, described community surveillance to have been a missing link. The two reports of effective investments are:

Adolescent Girls’ Livelihood Program Evaluation, Early Marriage Prevention Sub-Project (Bangladesh 2005)


9.7 Investment characteristics of working community surveillance systems

9.7.1 Strong intergenerational alliance at the community level According to the Bangladesh report authors, it is the “presence and knowledge” of peers about pending early marriage arrangements that allows the community to mobilize and advocate for a different outcome. The “eyes and ears” of adolescents in combination with the solidarity and involvement of respected community leaders gives this surveillance method a practical orientation toward action and activism as the following excerpts illustrate:

Adolescent Girls’ Livelihood Program Evaluation (Bangladesh 2005)


o Marriages are arranged within a matter of days of the proposal. The interlude is characterized by a brief period of information gathering by the groom and the bride’s family regarding each other and marriage negotiations. The sudden nature of the marriage event along with careful management of information and the overall will of the community for early marriage make a solution to the problem of early marriage elusive.

o This intervention uses the presence and knowledge regarding marriages held by peer adolescents and the members’ networks to address the issue of early marriage while in progress. It uses dialogues and discussions, without external intrusion, to de-motivate marriages at least until the legal age of 18 is reached.

In the case of the “Child Protectors” program in Swaziland, the evaluation implies that an effective level of surveillance had taken place on the difficult topic of sexual abuse, and that the “Child Protectors” program could carry on the effort:


o The DPM’s office coordinated the survey involving community youth assessments. These assessments highlighted the urgent need to protect children from all forms of abuse, to help communities understand that sexual abuse of children is a major contributing factor to the continuing HIV and AIDS epidemic…

o Qualitative results show that the area where all abuse cases are believed to have been reported is one in which the LL was introduced by Umphaksatsi57 and welcomed by the community. This group sees their role as children’s welfare in general not exclusively child abuse issues. Many of their activities have been in conjunction with the local schools as well as conducting home visits. They work closely with the Rural Health Motivators in their area.

9.8 Constraints to mobilizing community surveillance systems


Two additional reports carry information on purportedly successful investments in this area, but their evidence quality is poorly rated.

57 The evaluation report does not explain what entity this term refers to.


Chapter 10: Child Protection in Emergencies The dividends of good relations were illustrated at Bam. Pre-disaster partnerships between UNICEF and MoH [Ministry of Health] helped in facilitating program take off.

Evaluation of Emergency Child Protection Programs in Bam (Iran [Lazo and Balanon] 2006)

Gender considerations should move beyond programmes for women and girl children and analyse how programmes will differentially impact according to gender . . .

BAM Emergency Programme Evaluation 2003–2006 (Iran [Wiles et al] 2007)

Introduction and Overview Child Protection investments in the context of emergencies is represented in the meta-evaluation in eight reports. 58 Thematically these cover natural disasters (four) and war-affected children (four).59 These reports are unusually—in the context of this meta-evaluation—systematic in analyzing impacts against international and UNICEF corporate standards. Emergencies-centered evaluation reports were analyzed on the basis of five sub-categories:

Ability to execute rapid assessments that yield necessary information in a timely manner.

Overall management to deliver a holistic response on the Child Protection-related Core Commitments for Children in emergencies.

Protection and care of separated/unaccompanied children and children associated with armed groups.

Psychosocial support interventions are targeted and quickly established.

Capacities for prevention and response to gender-based violence are quickly in place.

In the aggregate, these investments:

Require more diligence in the area of gender analysis starting with needs assessments and running through program concept, design, and implementation.

58 Three recent evaluation reports addressing UNICEF’s tsunami responses in Sri Lanka, Maldives, and Indonesia contain

information on Child Protection in emergencies. Since they are multi-sectoral reports (Child Protection is one of four areas of analysis) they were not included in the meta-evaluation proper. However, where feasible given their very late acquisition in the meta-evaluation process, findings and lessons learned can be commented upon. Titles and dates for these reports may be found in Appendix Five. 59 An additional report (Evaluation of Alternative Care Programs, Sierra Leone 2006) addressed investments that had been

designed during an earlier emergency demobilization phase. The evidence base is too unclear for conclusions to be drawn, although one of the objectives was to assess whether children are “still stuck” in interim care settings years after the emergency phase had ended.


Are much more systematic compared to the meta-evaluation’s non-emergency investments, and are more likely to follow UNICEF and international expected standards of practice.

Highlight the significance of good pre-existing relationships with governments in shaping an effective response to crises.

Face difficulties in forging sectoral cross-cooperation.

Highlight the importance of preparedness measures (e.g., trainings).

Ability to Execute Rapid Assessments

… among the cultural issues that came to light in the course of the Bam emergency program were gender biases that excluded children who lost their mothers in the category of “orphans”...


Synopsis

Mostly positively rated in five instances (Algeria, India, Indonesia, Iran, Thailand)60

Rated as negligent (Thai-Burma) and unproductive (Angola)

Ambivalently rated (Thailand post-tsunami)

The investments that were most positively rated for programming quality in this category suggest that UNICEF can execute rapid assessments but that UNICEF may not always be asking the right questions. This has fostered some unintended consequences including: neglecting the needs of disabled children, making erroneous assumptions about cultural notions of gender and orphanhood, and making erroneous assumptions about what types of services people would find to be most relevant.61

10.1 Investment characteristics of good rapid assessment capacities

10.1.1 Situation analysis has to ask the right questions Four reports describe the difficulties summarized above and point again to the value of a good situation analysis:

60 There were two reports on the Bam emergency, Iran. 61 The Sri Lanka Tsunami evaluation reports a relative de-emphasis in community participation in defining services and the

absence of a uniform assessment instrument to track the needs of internally displaced people (p.51). In the Maldives context it was found that no rapid assessments of the needs of internally displaced peoples had been undertaken, but there was no explanation as to why that was so (p.61).

FINDINGS UNICEF can execute rapid assessments but may not always be asking the right questions (four reports).


BAM Emergency Programme Evaluation 2003–2006 (Iran [Wiles et al] 2007) — A comprehensive needs assessment was missing:

o However, the evaluation team did not find evidence of a written vulnerability analysis, economic, social or otherwise. This may explain why, for example, the needs of disabled children appear to have been neglected.

Psychosocial Care for Children Traumatized by Terrorist Violence (Algeria 2003) — Failure to harmonize approaches and operating assumptions:

o … while psychologists have created a functional network around focal points in the several Wilayas, representatives of the public sector services were calling far [sic] ‘installation of a network through creating a committee of coordination.’

o From the side of UNICEF, decentralization is understood in network terms, through supporting initiatives and activities in the different Wilayas, and supporting the function of focal points, while the central management of the project perceives decentralization in structural, policy and financial terms.

Evaluation of Emergency Child Protection Programs in Bam (Iran [Lazo and Balanon] 2006) — Investment lacked fundamental cultural insights:

o … among the cultural issues that came to light in the course of the Bam emergency program were gender biases that excluded children who lost their mothers in the category of “orphans”… children who lost mothers were not registered and counted as orphaned children. Discussions and training courses clarified the definition of orphans and separated children and thus improved the registration.

Psychosocial Interventions Evaluation of UNICEF Supported Projects (Indonesia 2003) — A lack of relevance followed an apparently missing or deficient needs assessment:

o The Eye Movement Desensitization and Reprocessing (EMDR)62 project was designed to focus exclusively on problems resulting from exposure to trauma, but many beneficiaries stated that psychosocial problems resulting from other issues—such as dislocation, family problems, or financial problems—were a higher priority for them.

10.2 Constraints to rapid assessment capacities

10.2.1 Situation analysis ignored or missing In two instances (both poorly rated for evidence quality), operational information that was on hand seems to have been overlooked, but the evaluators do not provide details about the management explanation for this:

Evaluation of Data Collection Project on the Impact of Armed Conflict on Children (SSRC/UNICEF, Angola 2005)

62 “a therapy protocol to treat the effects of trauma”

“However, the evaluation team

did not find evidence of a written vulnerability analysis,

economic, social, or otherwise. This may explain why, for

example, the needs of disabled children appear to have been

neglected.”

Iran [Wiles et al] 2007

EXCERPT


o A global inventory of existing information on children and armed conflict was conducted in order to indicate knowledge gaps … However, since it was only delivered after the end of the data collection phase, the insights from this work could not have been applied at the project design stage. The evaluators have thus been unable to locate a “survey of existing knowledge” that would have served to guide the creation of research questions.

Evaluation: Psychosocial Rehabilitation Project (Thai-Burma 2006)

o Psychosocial programming, including the Training Manual for (Program) Social Workers, was made even before the Baseline Needs Assessment was completed. There is no indication that adjustments in the programming were made later.

In one instance the evaluator gives a conflicting picture of the rapid assessment capabilities:

Evaluation of UNICEF Thailand’s Post-Tsunami Psychosocial Initiatives Report (Thailand 2006)

o One commendable feature of the tsunami response was the prompt rapid appraisal of the situation on the ground. Notably, local experts (one from a private hospital and a team from the academe) were tapped and mobilized to provide basic information and to assess the needs for assistance.

o In brief, the assessment during the emergency was not mindful of appraising the social ecology and the cultural context for psychosocial responses. Efforts to check possible involvement of peer groups, temple and church groups, cooperatives and community associations as psychosocial counselors and helping agents were nil.

Overall Management to Deliver a Holistic Response

In terms of planning, there was a lack of a single overarching document or plan of action that described the rationale for UNICEF’s programming decisions.

BAM Emergency Programme Evaluation 2003–2006 (Iran [Wiles et al], 2007)

Synopsis

The meta-analysis shows that inasmuch as a holistic response starts with starting the conversation with government and civil society partners, UNICEF is well placed to deliver. At the same time, there is a relative de-emphasis on quality assurance monitoring and follow through.

The four natural-disaster-related reports (earthquakes, Iran and India, tsunami, Thailand) describe how good pre-existing relationships with government partners paved the way for an effective immediate response.63

63 Evaluators of UNICEF’s Sri Lanka tsunami response found that UNICEF had not been proactive enough in addressing known

gaps in the government’s emergency child protection preparedness. Evaluation of UNICEF’s Response (Emergency and Initial


Two conflict-related reports (Algeria, Indonesia) describe building highly sustainable ships while sailing them through rough local capacities-building waters—efforts that Child Protection will want to study in greater detail.

Of the seven emergencies-related investments, two (Angola, Thai-Burma) could be described as having been weakened through management negligence,64 although improvements in results based mechanisms are called for across the board.

10.3 Investment characteristics of well managed, holistic responses

10.3.1 Essential: partnership-building in non-emergency times


The authors emphasize the absolutely essential value of building good working relations with the right partners in non-emergency times, a finding that is repeated by the evaluator of the India Gujarat earthquake report (see below):

o UNICEF’s pre-existing relationships with government partners in its country programme provided a crucially important asset in carrying out the Bam programme work.

Evaluation of Emergency Child protection Programs in Bam (Iran [Lazo and Balanon] 2006)

Pre-existing good relationships with the Ministry of Health and UNICEF paid “dividends”:

o The dividends of good relations were illustrated at Bam. Pre-disaster partnerships between UNICEF and MoH [Ministry of Health] helped in facilitating program take off.


UNICEF’s on-the-ground networks were mobilized to leverage government “buy-in” in the early stages of the crisis:

o At the initial stage a strong ‘buy in’ from the Government of Gujarat was created. Their initial skepticism about the psychological impact of the earthquake and its prime importance was worked upon by providing information and consultations with all sectors and civil society. The concept was discussed with officials from the education department and modalities worked out. Suggestions given by the NGOs through discussions were also incorporated into the program.

Evaluation of UNICEF Thailand’s Post-Tsunami Psychosocial Initiatives Report (Thailand, 2006)

Strong pre-existing relationships facilitated a prompt management response:

o UNICEF promptly mobilized its partners and resources during the emergency. UNICEF and the Royal Thai government have been well credited for their prompt and timely responses to the tsunami survivors such as the provision of drinking water. The RTG was

Recovery Phase), Sri Lanka (UNICEF Evaluation Office, May 2006). Evaluators found weak monitoring and follow up in Maldives, Sri Lanka, and Indonesia. 64 The Thai-Burma evaluation is based on a Project of the Catholic Office for Emergency Relief and Refugees (COERR).

FINDINGS UNICEF's on-the-ground networks were mobilized to leverage government “buy-in” in the early stages of the crises (four reports, natural disasters).


seen by the world as having taken a “take charge” attitude relative to the tsunami. (emphasis in original)

Evaluation: Psychosocial Care for Children Traumatized by Terrorist Violence (Algeria 2003)

On-the-ground networking pays long-term dividends:

o The project has achieved an important objective of developing the capacity of human resources. A large number of psychologists, psychiatrists, teachers, school counselors, and general practitioners have been reached by the training component. The outcome is the emergence of an informal network of professionals active in the area of psychosocial care for children in general and with those affected by violence in particular.

10.4 Constraints to management response capacities

10.4.1 Limited quality assurance mechanisms


The Bam evaluators saw inadequacies on quality assurance monitoring and overall coherence:

o UNICEF Iran had an Emergency Preparedness and Response plan (EPRP) in place … The EPRP was not used during the Bam response … it apparently lacked usefulness as a practical document and was not used as a live document in the heat of the moment.

o In terms of planning, there was a lack of a single overarching document or plan of action that described the rationale for UNICEF’s programming decisions, analysis, the assessment of need, etc. … Monitoring activity tended to focus on outputs and on the liquidation of cash assistance to government, rather than looking at issues of programme quality.

Evaluation of Emergency Child Protection Programs in Bam (Iran [Lazo and Balanon] 2006) Authors raise the matter of coherence, noting that results-based approaches had not been fully consolidated:

o … use of the Results based Management (RBM) approach to programming was uneven. 10.4.2 Unresolved administrative, fiscal, coordination Issues

Evaluation: Psychosocial Care for Children Traumatized by Terrorist Violence (Algeria 2003)

Management did not get ahead of inter-sectoral coordination difficulties:

o Communication and coordination of activities among the different sectors and partners has been a major problem since the start of the project.

Psychosocial Interventions Evaluation of UNICEF Supported Projects (Indonesia 2003)

UNICEF had no proactive management response plan in place, and fiscal planning was weak:

o The major constraints in implementation were linked to limited local capacities and lack of experience with psychosocial programmes.

FINDINGS For the West Timor emergency work, programme analysis and fiscal planning were disjointed, causing the cessation of a well designed and run program.


o Apart from some follow-up workshops, the Psychosocial Training Project was discontinued due to a lack of funds after the introduction of the EMDR project. In retrospect, this was regrettable, since the Psychosocial Training Project was an appropriately orientated, cost-effective, and practical project with the potential to be continued and expanded.

Evaluation of Data Collection Project on the Impact of Armed Conflict on Children (SSRC/UNICEF, Angola 2005)

The administrative and managerial magnitude of the project seems to have been misperceived:

o Given the size and importance of the project, its pilot nature, the expectations generated by it, the centrality of UNICEF’s role and the institutional and other difficulties encountered along the way, the lack of a fully dedicated focal point in the early stages was a serious managerial oversight. … A separate yet related problem has to do with the genesis of the project at the international level and the related low level of buy-in among UNICEF country offices and local stakeholders such as NGOs, governments, and other potential user groups.


Protection and Care of Separated/Unaccompanied Children and Children Associated with Armed Groups

Synopsis

Reports available for the meta-evaluation show investments addressing protection and care of separated/unaccompanied children and children associated with armed groups in ten instances (five/armed conflict, five/institutionalization and reform). However, these investments were for post-emergency, recovery, work. With respect to emergencies, the Bam earthquake emergency response addresses protections to separated/unaccompanied children.

10.5 Supports to separated/unaccompanied children

10.5.1 Pre-planning for family based care As reported elsewhere in the meta-evaluation:


o The impact of the joint SWO (State Welfare Organization) and UNICEF programme has been significant. The majority of children without primary care givers are living with extended family — “family based care.” …This is a major departure from the prevailing practice, if not policy, in Iran.

o The experience has also influenced the way that the State Welfare Organisation views childcare in the aftermath of disasters…


Psychosocial Support Interventions Are Targeted and Quickly Established

What is emerging as a result of interventions in repeated crises in the Algerian context is an informal network of professional mental health workers


The psychosocial program was prompt and wide in scope … due to the disaster-preparedness efforts of MoH [Ministry of Health] and UNICEF.


Synopsis

Seven of the eight emergencies-related reports revolve directly around psychosocial support interventions.65

A recurrent theme is that children and young people have not been involved in design, implementation, or evaluation.

There seems to be some knowledge management-related confusion regarding UNICEF’s investments in psychosocial supports to adolescents.

Government and civil society partners’ uptake is substantial.

Teachers are over-burdened; out-of-school children do not get the support that they need due to a first emphasis on school-going children.

10.6 Investment characteristics of targeted psychosocial interventions

10.6.1 Ability to mobilize quickly


65 For the three additional reports on the tsunami response, the psychosocial support response was “strongly coordinated

among international and national agencies” in the Maldives (p. ii), “effective” in Sri Lanka (p.5). In Indonesia UNICEF “fulfilled its coordination role well in psychosocial assistance” (p. iii). Evaluators found there, however, that a greater attention to gender was needed in the context of psychosocial supports (p.iv).

FINDINGS Government and civil society partners show substantial uptake of psychosocial supports, but teachers are over-burdened and out-of-school children are inadvertently excluded due to a first emphasis on school-going children (seven reports).


The mobilization took place quickly, with sustainable partnerships emerging gradually:

o The project has developed through several stages from an initial response to the crises to a partnership in development with public sectors and non-governmental organization [sic]… What is emerging as a result of interventions in repeated crises in the Algerian context is an informal network of professional mental health workers.

Evaluation of emergency child protection programs in Bam (Iran [Lazo and Balanon] 2006)

Uneven application of the psychosocial framework; generic use of the UN CRC:

o The psychosocial program was prompt and wide in scope … due to the disaster-preparedness efforts of MoH [Ministry of Health] and UNICEF.

10.6.2 Providing a safe haven

BAM Emergency Programme Evaluation 2003-2006 (Iran [Wiles et al], 2007)

o The establishment of community counselling centres/drop-in centres became a significant component of UNICEF’s psychosocial work in Bam.

10.6.3 Training develops intergenerational cadre

Psychosocial Interventions Evaluation of UNICEF Supported Projects (Indonesia 2003)

Good, efficient design, but the model apparently was not disseminated as a good practice:

o This project effectively implemented the main UNICEF principles on psychosocial programming in emergencies.

o The strengths of this design in West Timor were that it was comprehensive and multi-levelled—parents, teachers, and youth volunteers trained by the project provided social support to beneficiaries, while counsellors and social workers also provided counselling for those needing more specialised assistance.

o The Psychosocial Training Project is an easily implementable, efficient, and practical project with proven psychosocial benefits for children and their communities. It is an approach that should be continued and strengthened.

10.7 Constraints to psychosocial interventions

10.7.1 Inadvertently excluding out-of-school children


Inadvertently exclusionary investments:

o The training process could address only the child who was able to come to the school … The child who had experienced personal trauma and loss got overlooked in the training process as the process focused more on a generic response to trauma. The child who … dropped out due to migration, trauma, disability, or adversity could not be addressed by these skills. The most vulnerable child then may have been missed out of the net and most impacted.

“The training process could address only the child who was

able to come to the school…”

India 2003

EXCERPT



Inadvertently exclusionary investments:

o The psychosocial first aid activities effectiveness also suffered because there were no follow-up activities. The activities also focused only on the orphans and some of the severely affected. The needs of the other children who were also affected but did not lose any family member were not addressed. Because of the focus on the schools, the responses fell short of providing support to those who are not in school, to the indigenous groups, and the migrant children.

10.7.2 Haphazard and non-integrative service delivery


Haphazard and inadvertently exclusionary investments:

o …the effort to integrate the learnings from other organizations’ work to arrive at a more complete psychosocial process, did not happen. There was no continuity of care. Most people were met for an assessment interview and there were no therapeutic processes in place. …the greatest constraint of the process was that for psychological intervention to be therapeutic a degree of continuity and consistency was required in follow up.

10.7.3 Confusion on service provision to adolescents

BAM Emergency Programme Evaluation 2003-2006 (Iran [Wiles et al], 2007)

Authors say that UNICEF needs to build the case for service provision to adolescents from the ground up, but the Psychosocial Training Project in West Timor (2003, see next) seems already to have laid that groundwork:

o The provision of services to adolescents is important but challenging, especially where there is no obvious partner. UNICEF needs to develop guidance on best practice in this area.

10.7.4 Inconsistent application of the psychosocial framework


Uneven application of the psychosocial framework; generic use of the UN CRC:

o The psychosocial program was prompt and wide in scope … due to the disaster-preparedness efforts of MoH [Ministry of Health] and UNICEF.

o UNICEF should uphold the psychosocial framework as the primary framework in emergencies per the Emergency Field Manual. This has not been consistently applied in the programs and projects of UNICEF Bam. Further, in promoting the UN CRC in the programs and projects, there is a need to go beyond the usual concepts of child abuse and prevention, towards applying the UN CRC in case work, counseling, group therapy, and even every day life and rights-based child rearing.

10.7.5 Missing needs assessment

FINDINGS The Bam report (Iran 2007) says that UNICEF needs to build the case for service provision to adolescents from the ground up, but the Psychosocial Training Project in West Timor (2003) seems already to have laid that groundwork.


Evaluation: Psychosocial Rehabilitation Project (Thai-Burma, 2006)

A missing needs assessment:

o Significant sources of psychosocial support to the children were absent. These include awareness raising and psycho-education for the community, such as information on stress and trauma and methods of coping or discussions related to issues of violence and conflict, training parents to help children manage stress or advising persons with particular symptoms to seek out local and traditional service providers…

10.7.6 Over-reliance on teachers


An over-reliance on teachers:

o The psychosocial first aid activities were effective in a sense that they did maximize the resources of the schools and the teachers were equipped with the needed knowledge and skills in assisting in the provision of psychosocial support.

o The activities fell short of working with the significant adults in the children’s lives—their parents, families, and the communities. Some parents were not even aware that there were psychosocial activities conducted in the schools.

o The psychosocial first aid activities effectiveness also suffered because there were no follow-up activities. The activities also focused only on the orphans and some of the severely affected. The needs of the other children who were also affected but did not lose any family member were not addressed. Because of the focus on the schools, the responses fell short of providing support to those who are not in school, to the indigenous groups, and the migrant children.

Prevention/Response Capacities for Gender-Based Violence Are Quickly in Place

Synopsis

No meta-evaluation reports address the topic of gender-based violence in emergencies.


Gender-Differentiated Needs Are Addressed in Emergencies

Synopsis

Reports addressing emergency responses to natural disasters—earthquakes in Iran and India—have cited gender analysis as insufficient.66 This is in line with the general meta-evaluation finding of a de facto gender bias in programming, which may be reprised here:

Although 42 of 59 reports mention gender, the majority describe analyses that are superficial or missing.

This neglect extends into the evaluation reports themselves, where it is not uncommon to see a gender component listed as a programme or project objective with no follow up in the course of the evaluation.

10.8 Investment characteristics of poorly gender-differentiated needs in emergencies

10.8.1 Ignorance of basic socio-cultural definitions


Describes how the category “orphan” excludes children who lost their mother in the Iran cultural context:

o As it came to light in the course of the FR [Family Reunification] program, SWO [State Welfare Organization] had to revisit and update its definition to become more inclusive.

10.8.2 Programmatic gender apartheid, no impact analyses


o …gender analysis was largely absent from programme documentation and UNICEF’s response to the disaster lacks a well defined gender policy approach.

o Gender considerations should move beyond programmes for women and girl children and analyse how programmes will differentially impact according to gender and be designed accordingly, consideration being given to the needs of boys as well as girls.

66 This finding is consonant with findings in the three evaluations of UNICEF’s tsunami emergency and initial recovery phase

response.

FINDINGS The Iran and India reports cite gender analysis as insufficient. This is in line with the general meta-evaluation finding that Child Protection fails to acknowledge significant differences in the needs and opportunities and outlooks of girls and boys.


10.8.3 Failure to advocate for girls’ dignity and equity


Inexplicably, the program managers allowed the popularization of a song that clearly diminished women and anyone with disabilities

o One particular song that was very popular amongst participants was about a queen who needed some things from the king. The refrain was to make light of her needs, as these were so “contradictory” to her attributes. For example, one stanza of the lyrics talked of a lame queen asking the king for a cycle. The king asks her what she would do with a cycle, as she was lame. This song was repeated over and over again through the various training programs and even performed in front of the assessors as a snap shot from the training.

"One particular song that was very popular amongst

participants was about a queen who needed some things from

the king. The refrain was to make light of her needs. …This

song was repeated over and over again through the various

training programs."

India 2003

EXCERPT



Chapter 11: Cross-Cutting Issues …the managers do not have any financial incentives to implement a social policy aimed at the development of foster families.


Too many limitations in the way budget is allocated rather than the cost itself might affect the sustainability of the programme. Targeted allocation of the funds is crucial and it should more often reflect specificity of the local needs rather than specific programme budget lines.


Introduction and Overview The meta-evaluation Terms of Reference specified a review and analysis of five themes that cut across all areas of the Protective Environment Framework. According to the authors of the terms of reference, “These themes are not direct elements of the PEF but are still critical for successful Child Protection strategies.” The five themes are:

Costs

Partnership arrangements (who, what, where, when)

CP themes and sectoral good practices

Scale (micro, mezzo, macro)

Prevention or response

Information on these themes is sparse. For ease of reference, the state of play is tabularized here. Details may be found in the respective sub-sections of this chapter.

Cross-cutting theme Findings/comments

Costs Consonant with findings reported for monitoring and oversight, cost analyses are almost non-existent.

Partnerships Evaluators do not reflect on the seniority level of UNICEF’s key partners

CP themes and sectoral good practices

Juvenile justice and family based care have good correlations, but nearly one-quarter of the evaluations report on a different type of sectorality: CP investments need companion investments in areas such as water and sanitation, and poverty alleviation

Scale More than scale, the axis of programme success is management accountability combined with government buy in.

Prevention or response The majority (40 reports) were response-oriented.


Costs

Synopsis

As discussed already with respect to monitoring and oversight, management accountabilities for investments to strengthen the Protective Environment are largely absent from the reports that are available for the meta-evaluation. Cost oversight is also missing from most of the reports.” Cost-related data was tracked in three categories:

Ability to calculate any cost-benefit or cost-effectiveness ratios.

Ability to assess CP program cost “drivers” (e.g., what propels or restrains costs).

How important is the known level of resources invested to program success?

The findings are marginal in all three categories.

11.1 Effective cost analyses

11.1.1 Cost-benefit analysis is extremely limited

Only eight reports provided cost-benefit information. The discussion is comprehensive in two instances.


o . . . the managers do not have any financial incentives to implement a social policy aimed at the development of foster families.


o . . . no attempts to calculate or estimate the cost to society of laws and practices that violate the rights of children in contact with juvenile justice have been done in any of the four countries visited, and consequently it would not be possible to weigh the investments made against the benefits realized as a result of them. Indeed, whether it is appropriate to use such arguments in an area such as this is open to question.

11.1.2 Two evaluation discuss connections between costs and sustainability

Only two reports reflected explicitly on issues that propel or restrain costs (Laos Street Children, Bosnia and Herzegovina Improved Child Protection Mechanisms), one positive and one negative.

Review of the Street Children Project (Laos, 2006)

o The project has achieved results with a relatively modest budget. Cost-effectiveness has been aided by significant government contributions: the Ministry of Labour and Social Welfare has provided the premises for the Drop-in Centre free of charge and assigned two government staff to

FINDINGS Two reports have comprehensive cost-benefit findings. Two reports carry explicit findings on issues that propel or restrain costs. A common resource-related constraint is "do-more-for-less-itus."



o Too many limitations in the way budget is allocated rather than the cost itself might affect the sustainability of the programme. Targeted allocation of the funds is crucial and it should more often reflect specificity of the local needs rather than specific programme budget lines.

11.2 Constraints to costs analyses

11.2.1 Rigid budget requirements constrain the program

o Bosnia and Herzegovina (see above, 11.1.2)—finds that “rigidity of the financing might prove to be fatal” and that program needs are constrained by rigid adherence to preset budget lines

11.2.2 “Do-more-for-less-itus”

o The resource/program success relationship was found to be sufficient, explicitly or implicitly, in three reports and constrained in 14 others. Typical constraints: “do-more-for-less-itus,” volunteer fatigue, expectations of community in- kind supports are too high.


Only four in 59 reports carry a serious costs analysis.

"Organizations are working

under considerable resource constraints, and yet they are

expected to perform a multiplicity of functions."

Ghana 2003

EXCERPT


Partnership Arrangements

Synopsis

The meta-evaluation Terms of Reference included tracking partnership arrangements in four respects:

Who (family, community, youth, international, ministry, professionals)

What (facilitate, implement, enforce, advocate)

Where (decision-making seniority, influence)

When (at what point in the project are partners coming on)

Data addressing these themes was spare, imprecise, or scattered. The askSam database will confirm, for instance, that only a handful of reports were found to address specifics of What partners were doing (e.g. “facilitate,” “implement,” “enforce,” and “advocate,” and none at all for the temporal issues (“When”). This finding accords with findings from a recent review of UNICEF evaluation report quality, where stakeholder analysis is one of six areas of lagging evaluation practice:

Areas remaining for improvement are the description of stakeholder participation and analysis of relative contributions of stakeholders to the results…67

Indeed, it can be observed with respect to the meta-evaluation reports that this area of evaluation practice, while not critically sub-standard, requires more than just a description of the information that is required. It would appear that the rationale is lacking for evaluation teams and evaluation managers to drive the case for strong analyses of partnerships, the relative areas of contributions, and the level of seniority of the government partner. This must be so, otherwise the information on this topic would not be so low a priority in the reports. As a result of the issues flagged just now, the level of specificity that was hoped for on the matter of partnerships cannot be delivered. However, the general picture is not entirely elusive:

67 UNICEF Evaluation Report Quality Review 2006

http://www.unicef.org/evaldatabase/files/UNICEF_Evaluation_Report_Quality_Review_2006.pdf The other areas noted for improvement are “attention to eligible persons not reached by the project/programme, cost analysis, and a consideration of HRBAP, gender analysis, and RBM.”

FINDINGS Partnership profiles are un- or under-reported. Average seniority level is low to middle.



Partnerships Comments

Who Majority:

- Civil society organizations, government ministries

Minority:

- Children and young people

- International agencies

- Chapter 6, “Children’s Life Skills, Knowledge, and Participation,” describes three good practice investments in children’s and young people’s participation. These are notably valuable partnerships.

- The Sudan Peace building evaluation describes unusual partnerships with tribes.

- Partnerships profiles are un- or under-reported; evaluations are typically quite self-referencing, particularly with respect to international peer organizations.

What Majority:

- Implement, advocate

- Enforcement is a significant investment at the level of training (e.g., judiciary, police), but not at the level of data collection or monitoring.

Where - Low to middle68 - Seniority is higher with regard to juvenile justice investments, but evaluators or evaluation managers did not apparently ask the question—consistently—about the level of seniority that UNICEF is working with .

When n/d

11.3 Investment characteristics of partnerships arrangements

Referring to the table, above: 11.3.1 Who: Civil Society Organizations and Government Ministries 11.3.2 What: Implementation and advocacy, not enforcement 11.3.3 Where: Low to middle-level seniority

68 Of course, in varied and multiple contexts UNICEF works with relatively high level officials: director-generals or even deputy

ministers. As such, the finding reported here may be an artifact of evaluation reporting which, as discussed, is not specific on such matters per se.


Child Protection Themes, Sectoral Good Practices

Synopsis

In the original terms of reference for the meta-evaluation, the analysis was set to monitor for correlations between good practices and child protection themes. While the sample size of any area of good practice is too limited to call out correspondences with much certainty, some trends can be observed. For example, psychosocial programming emergency contexts is aligned to good practices making family based care a reality (as an alternative to institutionalization for children without caregivers). Psychosocial programming in emergency contexts is also well aligned with good practices in working with professionals and relevant government ministries. A good practice has also been identified linking girls’ livelihoods and child participation.

However, a more pronounced issue is that Child Protection investments are more likely to succeed when they are done in collaboration with investments in other basic needs (e.g., water and environmental sanitation, povety alleviation). This theme is sounded in 16 reports, with all but two possessing a good evidence base.69 Child Protection investments in these instances included supports such as vocational training (six reports) and basic needs like food or water (nine reports).

In the majority of these reports (11), the evaluator is making the observation. In 5 of the 16 reports, the evaluator is conveying stakeholders’ point of view.

However, Child Protection does not stand to gain a substantial insight into how this matter affects outcomes and impacts because in the majority of the reports (12), the topic is mentioned more or less in passing, as if the matter is widely taken for granted. The four reports where the evaluators provide more than passing reference are: Tanzania 200470 and 2007; Ghana 2003, and Ethiopia 2003.

11.4 Inter-sectorality blind spots

11.4.1 Child Protection needs to consider socio-economic contexts

69 Searching askSam on 12a* brings up 23 reports, with 13 highly rated and 10 poorly rated. Searching on 3d* brings up 13

reports, with 9 highly rated and 4 poorly rated. Some of the same reports came up in both searches 70 Impact Assessment of the Most Vulnerable Children (MVC) Community Based Care, Support and Protection in Musoma Rural

(2004/021)

FINDINGS One-quarter (27%) of the reports suggest that CP investments are more likely to succeed in collaboration with investments in other basic needs (e.g., Water and Environmental Sanitation).


Report on Child Justice (Zambia, 2005)

o On a general level it can be concluded that an increased awareness and sensitivity

towards children’s rights in the criminal justice system were observed in nearly all the

sites… This has, however, not translated into a general improvement in service delivery

… The problems experienced by the Zambian society make it increasingly difficult to

advance the rights of a relatively small group (child offenders) when poverty and

suffering is so vast and pervasive.

Street Children and Street Mother, Partnership Programme (Ethiopia 2003) — Supports include

dwelling, skills training, and a micro-credit scheme:

o The sustainability of the outcomes of the component seems ambivalent in that it is only

a third of the street mothers that confirmed having the confidence of continuing with

their business after the withdrawal of external assistance while fifty-eight per cent

categorically stated they could not carry on their small business after the cessation of

the support. (Reporting stakeholder point of view)

Evaluation of Lihlombe Lekukhalela Child Protectors (Swaziland 2005) — Supports include basic

needs:

o It must also be understood that these issues are taking place in an environment of

poverty and marginalisation. People who have scarce access to basic resources at times

find it difficult to see how they can make a difference to the whole range of hardships

they witness in their community.

Bam Emergency Programme (Iran [Wiles et al] 2007) — Post-disaster Family Reunification,

offered food assistance:

o Economic empowerment is not part of UNICEF’s mandate, but it may be able to

advocate with the SWO [State Welfare Organization] to identify agencies (governmental

or NGO) that could work on underlying livelihood issues.

Evaluation of the program “'Improved Mechanisms of Child Protection” (Bosnia Herzegovina

2003) — Supports include mixed social welfare strategies:

o Weak economic activity in almost all core municipalities aggravates already feeble social

systems.

11.4.2 Protection must address economic vulnerabilities

Protecting Street Girls and Ending Child Migration (Ghana 2003) — Supports include vocational

training; life skills:

o Basic shelter needs to be met first. (evaluator reporting on focus groups with target

stakeholders)

Evaluation of the Prevention of Child Abandonment Project (Romania 2003) — Supports include

community services:

o Child abandonment or neglect, (which) is determined mostly by the lack of certain

resources at the level of the family.


Psychosocial Interventions in Gujarat (India 2002) — Post-disaster psychosocial support, teacher

training:

o Reduced economic options could also increase the risk of child labor and trafficking.

11.4.3 Child Protection should increase synergies with other areas

Thematic Evaluation of UNICEF’s Contribution to Juvenile Justice System Reform (Serbia,

Romania, Tajikistan 2006) — Supports include health (substance abuse programs); vocational

and life skills training; training social workers:

o It is perceived as necessary that UNICEF makes a greater effort to increase the synergy

of its work in juvenile justice with its work in other areas.

Grassroots Peace-building (Sudan 2004) — Support includes water:

o Provision of many water sources reduces conflict.

Education as a Preventive Strategy Against Child Labour (UNICEF Global 2003) — Supports

include vocational and life-skills education:

o Education was viewed as the key strategy within a broader multisectoral approach

needed to break the cycle of poverty and disadvantage that maintains child labour.

11.4.4 Poverty negatively affects community participation

Mahzel Project for Social Reintegration and Protection of Disadvantaged Children (Eritrea 2004)

— Supports include basic needs of food, shelter, and clothing:

o Impossible to earn community support and cooperation sans economic support.

(stakeholder point of view)

An Assessment of the Impact of Implementing the MVC Programme and the Potential for Scaling

Up (Tanzania 2007) — Supports include basic needs, education:

o Surprising is the fact that in those districts with highest community based support also

show highest support levels through other organisations.

Impact Assessment of MVC Support Programme Implementation (Tanzania 2004) — Supports

include secondary education, vocational training:

o Drought and poverty have negative impact on community participation.

Community-based Child Protection Programme Evaluation and Review (Somalia 2006) — This

report is poorly rated for evidence quality:

o The evaluation found that in locations where the CPCM programme had been

implemented in conjunction with other UNICEF (also see p 21) interventions such as

water, education or health, or where communities had access to other services, there

was a higher level of participation and commitment to the programme.

Report on the Evaluation of the UNICEF-Supported Chikankata CBOSP & OVC Training Projects

(Zambia 2002) — This report is poorly rated for evidence quality:


o Community participation has, however, been affected by the high poverty and hunger

levels caused by close to eight years of poor rain in the area. People have no basic

income, which they used to derive from various farming activities.

Scale

Synopsis

To the degree that credible evidence is based on action taken at certain scales, the meta-evaluation attempted to track the scale of the programmes and projects, to avoid false implications about where an investment is known to work. The more the lesson is visible across different scales, the more confidence can be placed on it. Meta-evaluation scale data was tracked in three categories:

Micro: any scale from district or less. Includes projects at community and municipality level unless they cross a major size threshold. Covers most pilot projects.

o Meta-evaluation includes three reports at micro scale.

Meso: any scale between district and national, but most important when the scale of multiple provinces is reached, which brings qualitatively different demands on project management, cultural adaptation, etc. Can also apply to major urban projects.

o Meta-evaluation includes 35 reports at meso scale.

Macro: when the entirety of the target population is covered, including projects covering entire nations but also those that cover entire groups even if geographically limited.

o Meta-evaluation includes 21 reports at macro scale. The “meso” investments carry significant management accountabilities and strategic planning burdens. With just a few exceptions, there are no stand outs-for good practices with well supported evidence quality in this category.71 In the “macro” category, the meta-evaluation finds the following distinctions:

Direct government buy-in and involvement—the eight emergencies-related reports fall into this category, as do well rated investments such as deinstitutionalization (Belarus), juvenile justice reform (CEE/CIS, Yemen, Zambia) support for reintegration (Eritrea, Liberia) and mine education (Global).

Greater management buy-in and involvement—in the majority of the well rated investments, the technical and/or political demands are driven by a noticeable degree of political/international immediacy in combination with direct government buy-in.

71 Exceptions are: Bangladesh Girls Livelihoods (2005), Cambodia Disabilities Project (2006), Laos Street Children (2006).



Prevention or Response

Synopsis

At least on a conceptual level, investments to strengthen the Protective Environment can distinguish between prevention and response. According to the meta-evaluation data:

This distinction was found to co-exist in both concept and practice in evaluation reports of 12 Protective Environment investments. Thematically, the co-existence is noticeable with respect to:

o Juvenile justice reform (CEE/CIS, Thailand, Yemen, Zambia) o Gender and child protection (Bangladesh, Ghana, Guyana, Egypt, Ethiopia) o Street working children (Ghana, Ethiopia)

In another 40 evaluation reports, the programs were presented as response-oriented, although some contained such preventive elements as livelihoods trainings. In diverse instances these may pave the way for preventive investments, such as in the case of alternatives to institutionalized care and care for unaccompanied children or street children.

Prevention was an explicit objective in seven programme evaluation reports.



Chapter 12: Conclusions

Overview In shaping the terms of reference for the meta-evaluation, the authors noted that while almost all UNICEF-supported contexts are addressing child protection issue programmatically,

Not all country offices work confidently in child protection, and the evidence base remains small, including in verifying to what extent and how the elements of the protective environment are understood and addressed in the field by UNICEF and others.

The meta-evaluation affirms this outlook. With respect to both evaluation reporting emphases and programming concept/design, the Protective Environment Framework is in the early stages of becoming Child Protection’s second skin. This is predictable, given that the evaluations were conceptualized and implemented at roughly the same time as the initial dissemination of the Protective Environment Framework. The retrospective application of the Protective Environment Framework to evaluations of its own birth era, 2002-2007, shows how several areas of the framework’s programming theory are ripe for further development. For example:

The framework’s foundation, systematization of national and societal protections, is un/under-reported, as indicated by: (a) the minimal reflection on this topic in the evaluation reports, (b) in programming concept/design, an insufficient analysis of pervasive inequalities such as gender, and no evidence of contingency planning to address the difficulties of obtaining baseline studies; (c) a consistent finding that poverty constrains community level buy in to protection issues. Theorizing the causes and enablers of inequality, essentially, and having programming strategies emanate from that, would appear to be a missing link.

With systematization at its core, Child Protection is a direct stakeholder in the global networks that are forming to press the case, largely to donors, to deliver the means to measure impact over the long term. Both the Center for Global Development Evaluation Gap Working Group and the Network of Networks Impact Evaluation Initiative (NONIE) have come into existence to address the “disincentives to financing impact evaluations such as the lack of long-term funding and the difficulty in getting just-in-time funds for design.”72 These disincentives hit home for Child Protection because 64% of the reports had no longitudinal

72 DESIGNING A NEW ENTITY FOR IMPACT EVALUATION: MEETING REPORT, Center for Global Development, 2007 http://www.cgdev.org/doc/eval%20gap/Bellagio_07_Meeting_Report.pdf The core NONIE network is comprised of the DAC Evaluation Network, The United Nations Evaluation Group (UNEG), the Evaluation Cooperation Group (ECG), and regional evaluation networks.

http://www.cgdev.org/doc/eval%20gap/Bellagio_07_Meeting_Report.pdf


data and the majority of the evaluators complained that there were no baselines in place for the programmes that they were evaluating.

While rights based approaches to programming are implicit in the Protective Environment Framework, in practice these are hardly robust: there are only four instances of verifiably working partnerships with younger stakeholders, for example, and, using the Convention on the Rights of the Child to guide programming rationale (for example, as a tool for communities to improve their lobbying capacities with state actors) is almost non-existent. The place of rights based programming in the Protective Environment –at the level of concept/design and strategy/implementation -- is a rich topic for further development.

Important achievements that represent the basic rationale of the Protective Environment have yet to be marketed as such, including (a) the significance to systematization of professionals’ capacity building, (b) contributions to strengthened social safety (e.g. shifts towards community-based care for children), and (c) a strong inter-generational alliance combined with community education opportunities to facilitate an end to harmful traditional practices. A strategic communications plan for making these valuable investments very visibly identified with the Protective Framework, that is, as central to its raison d'être, will be helpful in building general literacy with the framework’s programming theory.

Picking up from the introductory chapter’s discussion of limitations and biases, in what sense is the meta-evaluation a baseline for Child Protection? Inasmuch as the fundamental requirement of a “baseline” is to show a specific performance measure, findings for the programming practices and evidence quality meet this requirement. The meta-evaluation also provides lessons about the Protective Environment Framework in practice, for instance: quality technical inputs such as situation analyses cluster around government more than community-based stakeholders. There is also enough evidence of a repetitive nature to suggest that the findings concerning monitoring can be generalized with confidence. For individual thematic areas of Child Protection, for instance, child labor or early marriage, findings must be scrutinized for their applicability. Organization of this chapter Conclusions are presented for each element of the Protective Environment, Emergencies, and Cross-Cutting issues. Only findings that serve to illustrate themes and issues that are common throughout the meta-evaluation are brought forward into the conclusions discussion. Child Protection stakeholders met with the evaluator for a one-day workshop at UNICEF global headquarters. The purpose was to add depth and context to the conclusions and to generate an initial set of recommendations. Stakeholder inputs on the conclusions are noted, below. Initial recommendations may be found in Appendix Nine of this report.


Conclusions

Programming and Evaluation Practices

12.1.1 Crippling programming and evaluation practices are taken for granted For reasons that are not novel among development professionals, organizations like UNICEF are obliged to accept that some of the time, programming planning and implementation must transpire on a contingency basis. Political instability, donor fiduciary delays, natural disasters, any of these and more can shift the frame of carefully planned investments and suddenly strategic planning turns into the ship that has to be sailed while it is being built. That said, the meta-evaluation finds a surprising absence of urgency around certain programming and evaluation practices that, although fundamentally crippling in terms of the operational intelligence they bury, are nevertheless treated on the order of “business as usual.” Three can be named: (a) the scarcity of time series information (e.g. 64% of the reports have no longitudinal data), (b) an alarming over-reliance on Key Informant Interviews, and (c) an under-emphasis on sampling and scale (e.g. 80% lack a sampling plan and most are quite small in reach). It should be emphasized that these customs are not characteristic of Child Protection or UNICEF alone, indeed, the whole of the global evaluation establishment is wrestling with similar irregularities and irrationalities. Still, if the meta-evaluation contributes to any one change, it would be to press Child Protection donors and partners to engage with fresh eyes on the causes of and fixes for these three areas of programming and evaluation practice. 12.1.2 De facto operations assumptions may be limiting the ambition of Child Protection evaluations The meta-evaluation is inconclusive about what drives the three programming and evaluation practices discussed just now. For instance, there may be a working assumption that Child Protection impacts can be tracked only within a localized frame. Or there could be a bias against statistical analysis, as if qualitative issues cannot be tracked at scale. Or, funding could be limiting the scale of the evaluations. Whatever the reason, the ambition of the evaluations seems unnecessarily limited. 12.1.3 Improving analysis of institutional contexts is a necessity The reports provide insufficient analysis of institutional contexts (e.g. donor requirements, host government sensitivities). Without information about upstream and institutional factors, such as UNICEF global managerial priorities and donor commitments/expectations, the frame of reference for interpreting programming decision making, for example, the absence of baselines and monitoring, is narrow and un-illuminating. 12.1.4 Need to re-examine quality control, leverage, at the at the sub-programme level The meta-evaluation finds that the closer to centralized political power, the more “clout” UNICEF Child Protection appears to find and utilize and the better its programming design inputs and technical outputs. UNICEF’s downstream work shows less “clout” and quality. Most probably this is a by-product of serial outsourcing, from one civil society organization or sub-regional ministerial office to another. It does raise a question as to whether sub-contracting arrangements taking place within a context of good strategic information on where/how best to apply leverage.


12.1.5 Difficult to reconcile gender findings with corporate commitments The evaluation reports indicate that programming is not differentiating between the needs and opportunities of boys and girls. The disadvantages that girls face across diverse institutional and political environments are buried, the findings suggest, under a one-size-fits-all programming template. This finding perhaps can be considered in the context of the findings regarding the relatively weaker quality of technical inputs and outputs at the sub-programme level. However, the meta-evaluation is largely inconclusive as to how such a gap can be reconciled with UNICEF’s corporate commitments to gender sensitive programming and with Child Protection’s clearly articulated commitments in that regard. 12.1.6 The askSam database has been an indispensible tracking tool The significance of askSam to the meta-evaluation has been overshadowed somewhat by the overall focus on the main report and findings. Yet, askSam has been indispensible in keeping track of any of the 100+ unique characteristics that were set for analysis (see the Protective Environment Framework Meta-evaluation Classification System, Appendix Two). And used synergistically with Excel, where ratings assigned to the classification system characteristics were stored, askSam allowed for statistical and textual evidence to be contrasted side by side. It has been a straightforward process, for instance, to map a trend discovered through the power of AskSam textual search, find the data points in Excel, and perform further statistical analysis as evidence. This allows a field operator to know how well a programme did on a specific criteria and then to read about what they did. The opportunity for learning is greatly enhanced. A qualitative assessment of askSam is planned as a separate deliverable for this project. askSam is a simple, searchable, database. It should not be difficult to deepen technical capacities at Headquarters as a bridge to introducing a wider field utilization of askSam.

Strengthening Government Commitments

12.2.1 How the convening organization convenes It would appear from the findings that for Child Protection UNICEF acts as a convening organization at the national level with (a) a good track record in creating contexts for new partners to come together, and (b) excellence in profiling emerging issues. In this latter regard, providing information and technical analysis through situation reports has been an area of effective investment. This is a strong area of leadership for UNICEF Child Protection. 12.2.2 Disparity in provision of technical information inputs Anticipating a theme that has manifested itself in several contexts in the meta-evaluation, it can be noted here that the technical inputs mentioned in 12.2.1, above, have no analogue where civil society-level and community-based stakeholders are concerned. Of course, work with government actors is work with government actors, hence, what is the rationale for raising the matter of non-alignment in information-related technical supports to civil society? The rationale is simply this: to take note of a disparity that seems to constitute a distinctive difference in the way that UNICEF does business with varying types of stakeholders.


12.2.3 Need to clarify the reasons for dis-investments in enforcement Although UNICEF finds and uses its “clout” most effectively with government partners at the national level, the meta-evaluation finds consistent gaps in enforcement and monitoring. These may reflect an undercurrent of instability in government commitments to the Child Protection agenda, but the meta-evaluation is not conclusive on this point. It raises the question of whether enforcement weaknesses are basically the price of doing business on Child Protection, or whether UNICEF could better leverage international agreements on aid effectiveness (e.g. Paris Declaration) in forging these agendas with host governments.

Legislation and Enforcement

12.3.1 Evaluators do not address institutional contexts, re constraints to enforcement Mirroring the findings for “Government Commitments:” (a) UNICEF is doing a good job in driving technical inputs for legislative change, and (b) UNICEF almost completely de-emphasizes monitoring legislative investments and there are no instances of investments in enforcement mechanisms at the state level. The evaluations do not provide information as to how institutional factors such as government sensitivities and donor preferences could be in play here. 12.3.2 A de facto “backdoor” enforcement strategy is in operation In the void between legislation and enforcement, UNICEF is investing in building the capacities of professionals and community level advocates to recognize and respond to Child Protection issues. According to the meta-evaluation findings, this allows Child Protection to work around host governments’ limitations on enforcement or around other institutional constraints that may be in play. 12.3.3 Lopsided supports to duty bearers versus rights holders There is no example of an investment to support duty bearers in developing protection legislation being combined with capacity building of rights bearers to know and to claim their rights. That is, technical inputs are clearly aligned with legislation whereas enforcement is largely at the level of moral persuasion. The meta-evaluation finds that investments in informational efforts such as situation analyses are a priority for state-level actors in shaping a legislative agenda, but that these are not perceived to be forceful, or potent, in shaping an enforcement climate. Elsewhere (see below), a similar operational apartheid appears to be in play, where the duty bearers at the state level receive specialized, precise, technical, inputs, while rights holders receive looser, generalized, and nontechnical “awareness raising.”

Attitudes, Traditions, Customs, Behaviors, Practices


12.4.1 A wholly replicable methodology is not seeing much marketing Exemplary programming addressing two difficult issues, ending child marriage and female genital cutting, does not seem to be recognized in Child Protection as just that – exemplary. Yet, this programming utilizes a methodology that is both replicable and that sits near the top of the meta-evaluation for the quality of the evidence base. There would not seem to be a mechanism in place as yet to call out the singularity of these accomplishments and to use them to further understandings of the programming theory that underlies the Protective Environment Framework. The methodology pairs a community level intergenerational alliance with educational opportunities on basic rights-related issues. It is implemented in communities over extended periods, with a core issue at stake (e.g. ending child marriage). The sample size for this exemplary programming is exactly two reports. 12.4.2 Models/theories of societal change may be missing? The remaining two reports in this category carry vague assertions about changing practices, and both have a weak evidence base rating. Moving from descriptive to interpretive models of societal change may be an area where evaluators need to be pressed for specifics and where training on approaching these matters may be useful to Terms of Reference designers.

Open discussion/engagement with Child Protection issues

12.5.1 Need to clarify practical/theoretical differences: “information” and “awareness” As discussed above, the primary finding is that Child Protection –in programme concept and evaluation reporting—is not delineating differences in terms that are meant to play distinctive roles in building a protective environment: “knowledge,” “advocacy,” and “awareness,” are used interchangeably. One hypothesis as to why these distinctions are blurred is that the basic place of information in triggering chains of knowledge and awareness has been downplayed or otherwise not specified. As discussed earlier, however, this distinction seems more apt to be blurred with respect to civil society as opposed to state actors, for whom delivery of information through situation reporting is a high priority. 12.5.2 Rational and place for programme communications (“C4D”) are unclear Strategic communications (Communication for Development) is a missing link in the programme descriptions in all 59 evaluation reports. This absence is also aligned with findings for building capacities of those closest to the child, which tracked 31% more activity in the “response” category than in the “recognition” category According to participants in the meta-evaluation workshop, the absence of investments on C4D reflect sampling biases, and not, for example, (a) a weak emphasis on strategic communications in Child Protection, and/or (b) a need for technical support for design, monitoring, and implementation. It is noteworthy that the findings for building capacities of those closest to the child. 12.5.3 Rights based approaches have an unclear status Good outcomes are seen where the Convention on the Rights of the Child (CRC) has helped to legitimate programming investments, but in an equal number of evaluations the CRC’s status as more than a


ceremonial role is weak. It is possible that, at the end of the day, rights based approaches and the rights framework are not widely applied and that these are also less well developed areas of the Protective Environment Framework programming app.

Children’s Life Skills, Knowledge, and Participation

12.6.1 Rights based approaches are not a central emphasis The findings suggest that child participation, a basic principle of rights based approaches to programming, is not an elemental component of Child Protection investments as represented in the universe of materials that was available for the meta-evaluation. Programming is found to be top-down and particularly exclusionary at the design stage; evaluation reporting on this topic is almost accidental. In just four reports of 59, child protection stands to obtain information on what works and what doesn’t regarding child participation in the service of building a protective environment. A call for information on child protection programs with participatory components for younger stakeholders would be helpful in clarifying the actual status of child participation within Child Protection. It would also be helpful to review evaluation terms of reference to find out whether participatory efforts have been dropped inadvertently as topics for impact assessment.

12.6.2 Evaluating Life Skills investments has not been a priority The findings show that Life Skills investments are unmonitored, missing, or un-ambitious; just about one-quarter of the evaluations described these investments in any case. The status of these investments in an overall evaluation plan seems to indicate that their status vis-à-vis programming is not central.

Building capacities of those closest to the child

12.7.1 Professionals get more professional inputs from UNICEF? As noted, across both “response” and “recognition” categories, professionals’ capacity building saw the highest programme practices quality, with 19% more overall quality than investments in communities’ capacity building. One hypothesis is that training requirements for professionals are more cut and dried, technical, and hence more easily deliverable (and evaluable) than, say, the requirements for community based child protection advocates. Or perhaps the delivery mechanisms are at issue, whereby UNICEF has more direct control in respect of professionals’ capacity development than communities. This is a topic for further research. 12.7.2 There is no consistent standard for working with volunteers Findings concerning community engagement and capacities development/training suggest that when these work well, they are powerful tools. A consistent standard about how best to utilize volunteers, however, is missing, and it is unclear why trainings –when they can be so powerful and additive—might ever be under resourced or poorly orchestrated. As noted, the capacities building component of the PEF points up the importance of training programme content and delivery, such that length and/or quality of training/support make the difference, not just a difference


12.7.3 The front end information needs of civil society partners/stakeholders are taken for granted As noted, data addressing “Major actors are facilitated in recognizing child protection needs” suggests that raising awareness about rights and needs of vulnerable children is often an afterthought, secondary to actionable objectives of strengthening the capacity to respond to needs. This is substantiated further by the evidence of 31% more investment activity in the “response” category than in the “recognition” category. What is not clear is how Child Protection came to this, what understanding of information needs led to the secondary emphasis on recognizing child protection issues as opposed to responding to them. 12.7.4 Protective Environment programming theory is unclear about the status of building capacities of families As noted, data addressing “Strengthened capacities for responding to child protection needs” suggests that “top-down” (vs. “bottom-up”) strategies predominate in capacity-building CP programmes. That is, families are the least frequently facilitated in the “response” category, professionals are the most frequently targeted. This could be a simple artifact of the evaluation reports that were available for the meta-evaluation, or it could be that the place of families in the Protective Environment Framework is still somewhat under-theorized.

12.7.5 Evaluators do not address the systematization of capacities building As noted, out of 59 reports, 41 addressed systematization and/or integration of the programme under evaluation. Note therefore, that nearly 20% of the evaluations failed to even address the sustainability of capacity development. And, of the 41, only 24 are rated with credible quality of evidence. At the very least it would thus appear that at the level of evaluation reporting this elemental component of the protective environment has not yet been sufficiently internalized. It is unclear whether this conclusion is applicable also to programme design/concept. 12.7.6 Best investment towards sustainability: training professionals As noted, professionals’ response capacity building is linked to integration/systematization of Child Protection. This may be a de facto area of emphasis in Child Protection or a coincidence of the reports that were available for the meta-evaluation. The flip side is notable: where communities’ response capacities are concerned, issues concerning ownership and volunteer fatigue constrain sustainability.

Delivery of necessary, child friendly, services

12.8.1 Evaluation terms of reference: out of sync with the PEF The delivery of child friendly services is the most densely represented area of investment in the meta-evaluation, with 62% of the strategic planning-related practices rated as good quality. However, evidence quality is a significant constraint: of 27 reports possessing information on service delivery efforts only 15 rated credibly in terms of their evidence base. Evaluation reporting generally focused on strategic planning irregularities such as gaps in management responsiveness to bottlenecks and an inability to gather reliable contextual information. Disentangling who was supposed to do what took more precedence than who accessed what services to what impact. This could be expressive of an evaluation culture that is essentially “glass half empty.” It could also be a reliable indicator of the sea of contingencies and improvisational realities of service delivery.


Misunderstanding programming contexts, regularly it would seem with respect to gender differences, is found to undercut service delivery on a consistent basis. 12.8.2 Child Protection vastly understates its successes Though relatively limited in number, the five instances of shifts towards strengthened social safety nets have an arguably muted presence in Child Protection communications and marketing. They are treated as business as usual when, in fact, they would appear to demonstrate programming and planning practices that work: taking the long view on capacities building of government and professional partners, good stewardship of and investments in situation analyses.

Monitoring and Oversight

12.9.1 No explanation for monitoring findings, they are totally inexplicable Monitoring and Oversight is the most deficient area of the Protective Environment investment portfolio. At least 90% of the reports that were available for the meta-evaluation contained complaints about monitoring having been insufficient or absent altogether. Investments that align to centralized political power –for instance, relating to enforcement of a national plan--are among the least likely to be monitored for quality assurance. But the meta-evaluation also finds a lack of depth in evaluation reporting on the matter of causation: evaluators offer no interpretations at all as to factors that might explain the absence of programming addressing monitoring and baselines. Monitoring capabilities as represented in these reports do not differ significantly from the global norm in that they are processes in need of support. A concern, then, is not only about knowledge management (how does the Child Protection community learn about and build on previous efforts), but also about pragmatism (how CAN evaluation X bring our practice forward?) and even marketing (what IS new and innovative about this programme?). The meta-evaluation finds this area of programming practice to be critically sub-standard.

Child Protection in Emergencies

12.10.1 Gender and Psychosocial findings merit further study The meta-evaluation had just a handful of emergency-related reports to go on – eight in total. The findings are generally consistent with those cited already for the main components of the PEF. And (a) psychosocial investments seem to be delivering a depth and uniformity of operational information, with consistent findings that teachers are overburdened and out of school children are underserved, and (b) gender analysis is insufficient throughout. The cornerstone of good response capacity seems to be UNICEF’s cultivation of government partners and other significant stakeholder in non-emergency times.


Cross-cutting issues

12.11.1 Lack of clarity on what makes a cross-cutting issue cross-cutting Two cross-cutting issues, gender and poverty alleviation, were more consistently at issue in the meta-evaluation than all five that were preset. As both address the foundational principle of the Protective Environment, moving towards systemic changes in national and societal policies, this reinforces the meta-evaluation’s general impression that systematization is not explicitly enough positioned as a driver of programming logic. On the matter of partnerships, evaluation managers are not prioritizing analysis of partners’ seniority levels as these are vastly un/under-reported.


Appendix One: Terms of Reference

1. Purpose of Assignment: Conducting a Child Protection Meta-evaluation

UNICEF’s work in the area of child protection has evolved rapidly and considerably for over two decades.

Programme design in child protection has been shifting from projects for specific groups of children in

especially difficult circumstances to a more comprehensive approach of supporting change in social and

institutional environments. The approach entitled ”protective environment for children” was introduced

during the 2002–2005 MTSP for UNICEF’s work on child protection. The use of this approach leads child

protection programmes to become more strategic and better integrated in all UNICEF work.

All but one of the 157 UNICEF-supported countries, areas, and territories address child protection issues

programmatically. At the same time, not all country offices work confidently in child protection, and the

evidence base remains small, including in verifying to what extent and how the elements of the

protective environment are understood and addressed in the field by UNICEF and others.

In order to expand and strengthen child protection programming towards achieving the targets of the

current MTSP 2006–2009, an authoritative basis for confirming effective protection interventions is

needed.

The aim of this consultancy is therefore to determine which Child Protection programming strategies

and project interventions have been proven to work. Proven to work means that the combination of

good design, monitoring, and quality evaluation allows UNICEF to state that certain approaches—if well

executed—are likely to result in measurable improvement in resisting violations of children’s rights to

protection, and to enhancing the resiliency and social networks that will foster strong child and

community development. The evidence will be examined against the goals contained in the Protective

Environment Framework (PEF).

This evaluation is intended to contribute to UNICEF in three main ways:

1. To provide an evidence base for: a) global lessons sharing with partners and b) global, regional,

and national programming and advocacy efforts.

2. To inform any potential revisions of UNICEF’s Child Protection goals, especially those contained

in the Medium Term Strategic Plan 2006–2009, which is scheduled for a Mid-Term Review in

early 2008.

3. To support the management accountability for capturing lessons learned and examining the

impact of UNICEF’s work in child protection.

Other direct and indirect expected benefits include the following:

Identification of programmatic elements for inclusion in child protection training and guidance

Refinement of programmatic indicators/proxy indicators

Identification of specific needs in knowledge management strategies, policies, and management of financial and human resources

Better internal and external collaboration on child protection


Generating lessons learned for any future reviews of child protection research and evaluation methodologies, with a long-term goal of standardizing methods where possible to ensure valid and reliable data is always generated.

2. Details of SSA Section: Child Protection/Programme division Level: Comparable to P5 level Duration: 21 May 2007 – 21 December 2007 Supervisors: Shirin Nayernouri, CPS Sam Bickel, Evaluation Office Location: Consultant is expected to work from home with not more than 2 visits to UNICEF NY 3. Major Tasks and Deliverables

1. Mobilization – A compilation of electronic copies of Child Protection evaluations conducted by UNICEF from 2002–2007. This deliverable will be provided by UNICEF.

2. Evaluation Quality Review – A completed evaluation report rating form prepared electronically for 100–120 evaluations.

3. Tools Development and Piloting – An inception report to include: An agreed and refined set of themes to be examined An agreed method for recording and/or coding information found within the evaluations An updated workplan for completing the consultancy

4. In-depth Annotations of the Evaluations – The deliverables will include: Estimated 70 evaluations completely annotated per protocols developed in Stage 3. The

annotations will be in electronic form within the documents, and will be linked in a database that allows users to browse a hyper-linked index that sends them to their topic of interest.

Draft gap and thematic coverage analysis to circulate for comment to CP. 5. Analysis and Draft Report – the deliverables will include:

Early draft summaries of findings; the findings should include feature boxes, a synthesis of evidence, and a summary of evidence gaps.

A comprehensive draft report prepared following feedback from UNICEF on the draft summaries.

6. Final Report – The deliverables will include: Presentation of draft report, day long workshop, HQ

The final evaluation report to include the thematic chapters, as described in phase 5, and

the following additional elements:

A complete record and copy set of all prior deliverables not already transmitted to

UNICEF, including relevant notes and data files.

A report on the process of the evaluation, strengths and weaknesses of the

methodology, and other tentative process recommendations to ensure better

knowledge management of CP evaluations and evidence in future years. This report

will be from 5–10 pages long, and is an elaboration of the normal chapter in a Final

Report on Methodology. The goal is to self-analyze the methods used as a Lessons

learned for UNICEF to consider in the construction of future exercises of this type.


Stand-alone Power Point presentation of no more than 20 minutes, including all speaking text as noted. This is to permit UNICEF staff to deliver the findings in subsequent venues in the words of the consultant.

4. Qualification or Specialized Knowledge/Experience Required

Advanced university degree in Social Sciences

Proven ability to conduct detailed research or meta-evaluations

Strong technical evaluation skills

Good knowledge of child protection issues

Excellent analytical, planning, and writing skills

Experience with project management

Familiarity with UNICEF programmes

Good communication and interpersonal skills

Teamwork 5. Timeframe: This work is expected to take eight months starting 29 May 2007.


Appendix Two: Protective Environment Classification System

Introduction As stated by UNICEF, “The Protective Environment is a web of interconnected elements which individually and collectively work to protect children from violence, abuse, and exploitation.”73 As stated in the main body of the report, the Protective Environment was introduced as a programming framework for Child Protection in the context of the 2002–2005 Mid-Term Strategic Plan of UNICEF. This appendix lists and annotates where necessary the eight elements that make up the Protective Environment, plus six additional categories that the UNICEF Child Protection Section and the UNICEF Evaluation Office wished to include in the meta-evaluation analysis.

Part One: Protective Environment

1. Government commitment to fulfilling protection rights of children a. National plans of action, policies, budgets b. Initiatives to strengthen capacities for partnerships, advocacy, implementation74 c. Commitments emphasize prevention, response, or remediation

prevention = pr response = rsp remediation = rmd

d. Other

2. Legislation and enforcement a. Legal development/reform reflecting child rights principles b. Stengthened enforcement c. UNICEF understands and has capability to address bottlenecks and opportunities75 d. Other

3. Addressing harmful attitudes, customs, behaviors, practices a. Insights into local power structures, decision making networks b. Tactics are contextualized and stratified

73 http://www.unicef.org/peflash/pe2.html 74 Implementation-related advocacy was a rare investment. 75 As there is no code addressing funding constraints, ”10c” alone is not sufficient, 2c is coded with 10c in that

instance (e.g., resource found to be constraint that has not been addressed)


c. Changed attitudes and practices d. Other76

4. Open discussion and engagement with child protection issues a. Increase in public knowledge about the issue b. Open discussion on sensitive issues c. Use of media for advocacy work d. Formal linkages to CRC are visible, part of the program e. Children feel able to speak and/or act more freely about protection concerns f. Other

5. Knowledge and skills of children to protect themselves

a. Insights into what children/young people know/need to know b. Relevant information/services are provided (e.g., rights trainings, free counseling,

condoms distribution, school curricula, child media, peer-to-peer activities) c. Children/young people are facilitated to get involved (e.g., establishment of youth

networks) d. Other

6. Capacity of those closest to the child a. Major actors are facilitated in recognizing child protection needs

Family Community/CSOs Professionals

b. Strengthened capacities for responding to child protection needs Family Community/CSOs Professionals

c. Capacity development is becoming systematized, integrated, not ad hoc d. Other77

76

3d is used:

To tag observations that shed useful/important light on the local context, but that don’t necessarily come from the project itself, but rather from the evaluator (although it’s not always possible to disentangle the two); or,

To tag observations that indicate how the local socio-political context might complicate project goals (as in the case of Somalia and Swaziland, where extreme poverty overwhelms children's rights in the mind of the community).

In other words, 3d refers to the general socio-political context, rather than specific power structures/decision-making networks (such as Sudanese tribes), the appropriateness of tactics (such as targeting tribal elders), or the change the project has created. 77 6d has been used to refer to capacity building “management” issues that aren’t specific to UNICEF (8f). This is used

primarily for the Cambodia Social Work Training Evaluation (2005) to tag text that discusses the problems of the MSST programme (Management Social Service Training): problems relating to curriculum, staff, pedagogical methods, funding, etc. In other words, this is a category of logistical/programmatic issues that are different from mainstreaming and sustainability (6c) and that aren’t just to do with UNICEF (8f).


7. Access to services

a. Provision of necessary and child-friendly services b. MVC & Families facilitated to access these services c. Strengthened social welfare systems d. Other

8. Monitoring and oversight of child protection issues

a. Monitoring systems including data collection on CP issues b. Establishment of participatory and locally-based mechanisms c. Reliable reporting systems are established d. Establishment of community surveillance systems e. Information on sensitive subjects, e.g., closed f. Other78

9. Child protection in emergencies a. Ability to execute rapid assessments that yield necessary info in a timely manner b. Overall management to deliver a holistic response on the CP-related Core Commitments

for Children in emergencies c. Protection and care of separated/unaccompanied children and children associated with

armed groups d. Psychosocial support interventions are targeted and quickly established e. Capacities for prevention and response to gender-based are quickly in place79 f. Other

Part Two: Cross-Cutting Issues

10. Costs a. Ability to calculate any cost-benefit or cost-effectiveness ratios b. Ability to assess CP program cost ‘drivers’ (e.g., what propels or restrains costs)

Yes = y No = n

c. How important is the known level of resources invested to program success? Yes = y No = n

11. Partnership Arrangements

a. Who Family = aa Community = ab (CSO, religious leaders, tribe)

78

There’s no code for key programme design issues: how does UNICEF identify objectives, how does it identify

specific indicators, what do these indicators measure, how does it specify attributibility in a diffuse landscape of providers and partners, how does it monitor and evaluate its own activities and organisational challenges, etc. 8f has thus been usurped as much as possible to flag these issues.

79 As the occurrence of specifically gender-related materials became more frequent, 9e was usurped to tag non-

emergency as well as emergency-related gender information.


Youth = ac International = ad Ministry = ae Professionals = af

b. What

Facilitate = ba Implement = bb Enforce = bc Advocate = bd

c. Where (decision-making seniority, influence)

Lowest = ca Middle = cb High = cc

d. When (at what point in the project are partners coming on)

Beginning = da Middle = db End = dc

12. CP themes (sectorality)

a. Correlation between PEF good practices and child protection themes 80 13. Scale81

a. Planned to work at ______ level b. Had success at _____level

Micro = mi Mezzo = me Macro = ma

14. Prevention or Response

a. Program objectives account for differences among prevention, intervention, and remediation Yes = y

No = n Can’t say = cs

80 12a has been usurped to track evaluation data addressing another kind of intersectorality: where evaluators find

that Child Protection investments are more effective when orchestrated in tandem with, for example, water and environmental sanitation, or poverty alleviation. 81

Some of the background questions on the matter of scale were set as follows:

What level of decentralization is workable for what kind of CP interventions?

Have we shown that we CAN work at scale?

If not, what’s the reason we’ve worked at lower scales?

Is CP programming most functional at smaller contextual levels by nature?


Figure 4: Average Quality, Protective Environment Elements (Disaggregated)


This figure shows the average programming quality of each sub-component, or building block, of the Protective Environment Framework. The vertical axis shows the eight components plus emergencies with all the sub-components. The average quality of the investment strategy, that is, the appropriateness and effectiveness of the programming practices that are attached to them, is shown on the horizontal axis. Programming was rated on a scale of 0 (unacceptable) to 3 (excellent). Details on the ratings framework may be found in Appendix Four. Again, these quality ratings are not measures of evidence quality: they are measures of the quality and effectiveness of the investment strategies, as facilitated or constrained by programming practices.


Appendix Three: Evidence Quality Reporting Framework

Introduction The UNICEF Evaluation Office applies evaluation standards quality ratings to each evaluation that enters into the UNICEF Evaluation Database. For this meta-evaluation, however, the UNICEF Evaluation Office designed an instrument to assess evidence quality in depth. Its purpose was to assist in distinguishing evaluation report contents not on the basis of general evaluation practice but, rather, on the basis of the integrity of the evidence base. Thus, the meta-evaluation evidence quality ratings framework (see Figure 5, below) focuses specifically on activities relating to methodological good practice—for example, the use of comparison groups and longitudinal data. It has also been a means of identifying evaluation reports that analyze results against globally accepted standards as opposed to what individual stakeholders say (“customer satisfaction”). This appendix presents the measurement framework and a short overview of the average report ratings. The numbers in Figure 5 (below) are the values assigned for the level of quality. For instance, if a report used ample longitudinal data it might receive a 2.5 or above rating; if there were an over-reliance on what Key Informants said (as opposed to an equal or greater emphasis on other less subjective variables) the report would read more as a customer satisfaction survey and receive a poor rating (0.51- 1.5).


Figure 5: Evidence Quality Reporting Matrix

Variable 0–0.5 0.51–1.5 1.51–2.5 >2.5

Results Based Management level

Can’t say Activity Output Outcome or

impact

Comparison groups

None Selected at time

of evaluation Matched during program design

Random assignment

Data time series End (eval only) End and middle End and baseline Baseline, middle,

end

Standards used to analyze results

None: presented without analysis

Client satisfaction

Against project goals

Against globally accepted standards

Consequences analyzed

No analysis Unclear82 Intended only Intended and unintended

Data/analysis disaggregation

None One variable (e.g., gender)

Multiple single variables

Compound variables (e.g., rural boys vs.

rural girls)

Statistical analysis None

Simple tabulations of

qualitative data or n/a

Simple tabulations of

survey data

Complex analysis of survey data

Overall ranking Unacceptable

0–0.5 Poor

0.51–1.5 Good

1.51–2.5 Excellent

2.51 and above

82 Consequences unrelated to project goals = unclear.


Evidence Quality Findings The average overall rating for the meta-evaluations was 1.44. This is the higher end of the “poor” category. The evaluations were fairly equally divided in terms of evidence quality with almost half in the “poor” evidence category and half in the “good” category:

- Excellent: 4 reports (6.78%) - Good: 24 reports (40.68%) - Poor: 29 reports (49.15%) - Unacceptable: 2 reports (3.39%)

The average ratings for individual variables show that evidence quality was generally good with respect to the analysis of intended and unintended consequences and attention to results-based tools and methods. The evidence quality was generally poor in the areas of statistical analysis, data time series, and the use of comparison groups. The following average ratings per variable were calculated:

Evidence quality variable

Average rating, 59 reports

Consequences analyzed

2.2

Standards used to analyze results

2.1

Results Based Management level

1.83

Data analysis/disaggregation

1.56

Statistical analysis

1

Data time series

1

Comparison groups

0.47

Figure 6: Evidence Quality, Average Per Variable

A few of these findings can be understood in relation to general UNICEF evaluation performance standards. Drawing from a 2006 annual review of evaluation quality ratings: 83

83



- Whereas in the meta-evaluation evidence ratings, the attention to expected and unexpected consequences is quite good, according to the UNICEF Evaluation Ratings system this element was missing from 30% of reports between 2004 and 2006 and is thus considered to be an area of evaluation practice in need of strengthening.

- In both systems, the Results Based Management standards are satisfactory; however,

the meta-evaluation evidence quality tool also specifies that outcomes and impacts are analyzed. This is an area of evaluation practice that, according to the UNICEF Evaluation Standards, is poor or missing in 28% of the evaluations that were rated between 2004 and 2006. Thus the Results Based ratings for the meta-evaluation are better than those measured for UNICEF evaluations as a whole. A look at which evaluations did well in that category (Appendix Five) would suggest that investments in Tier One evaluators are also investments in Tier One evaluation reporting.

Limitations and Biases The evidence quality reporting framework is useful as a means of bringing consistency to the reports in terms of measuring evidence quality. The UNICEF Quality Ratings Standards are too generalized to accomplish this important task. One size does not fit all Generally, evaluations of activities with an on-the-ground presence fared somewhat better when subjected to this rating tool. That is, evaluations of efforts addressing, say, street children, did better than evaluations in areas such as legislation. The reason for this is that evaluations of street children investments could relatively easily utilize longitudinal data and comparison groups, whereas these methods have less bearing on the evaluation of legislative efforts, at least in the short run. Once this limitation became apparent with respect to evaluations addressing legislation, however, there was no retrofit. This was an emergent finding and a retrospective system of counter weights and measures was, in the opinion of the evaluator, not called for. Does not ask “why” The framework was also largely unresponsive to evaluation findings relating to UNICEF’s own corporate strengths and weaknesses, that is, to issues revolving around programming practice that might militate for or against good evidence quality. For example:

- A relative de-emphasis on monitoring ran through report findings and constrained what evaluators could do with regard to data time series and comparison groups, yet this tool records only that these were poorly rated categories, not why that was the case.

- UNICEF’s robust capability as a broker across often widely varying and unfamiliar

partners, a corporate strength that was seen to mediate the viability of many


investments, could not be situated except by using the “consequences analyzed” in a the very widest possible sense and often inferentially, as well.

To bridge this gap, during the Inception Phase of the meta-evaluation, the evaluator analyzed and synthesized programming practices by applying the Protective Environment Framework to the first 47 reports (or five/sixths of the entire meta-evaluation content). From this content analysis, the evaluator designed a programming tracking and measurement tool (see next appendix).


Appendix Four: Programming Practices Reporting Framework

Introduction The main report has detailed how the programme practices reporting framework came about, how it was applied, and how it relates to the matter of evidence quality. This appendix presents the framework and statistical tables to support the discussion in the main text.

Figure 7: Programme Practices Reporting Framework

Variable Unacceptable

0–0.5 Poor

0.51–1.5 Good


>2.5

A. Context In practical terms, how well does the program design fit the political, cultural, and economic climate?

Generic, cookie cutter Poor or missing situation analysis

Basic understanding of the environment

Taps into relevant formal and informal decision making networks

Activates new, unusual, partnerships

B. Intersectorality None; should have had but didn’t

Limited, but properly so (wasn’t the scope to become intersectoral)

Convenes partners; some logical or desired partners

Integrated

C. Programme communications84

Missed opportunity Limited; unconnected to the CRC

Evidence based advocacy; media plan

Activates commitments to the CP program and CRC

D. Facilitative role Moral persuasion

Missed opportunity Limited Knows who/how to lobby

Excellent use of Unicef convening power

E. Achieved level of community ownership

None or symbolic Expectations are not realistic

Develops buy-in Community defined; community driven

F. Child participation

Missed opportunity Mostly rhetorical Incorporates CRC trainings

Integral; forged at design stage

84 Refers to Communication for Development, or “C4D.”


Figure 7: Programme Practices Reporting Framework

G. Strategic planning and technical assistance How the work will get done, by whom, with what resources, etc.

None Non-identification or non-mobilization of partners Non-identification of the Protective Environment Framework

Duplication of effort Weak technical assistance

Monitoring plan with capability to make adjustments Protective Environment Framework drives planning Human rights lens in place

Future strategies can be informed by this one

H. Knowledge management

None Limited Design takes good practices into account

Proactive, comprehensive

OVERALL RANKING Unacceptable

0–0.5 Poor

0.51–1.5 Good


2.51 and above

Programming Practice Findings Strategic planning is defined as how the work will get done, by whom, and with what resources. As discussed in the main report, this was the paramount programming issue across the whole of the meta-evaluation. The density of evidence around “Strategic planning,” which is four and one-half times higher than “Child participation,” carries a freight of information about the professionalization of development practice. What is the nature of this evidence? Good? Bad? Some of both?

“Strategic planning” was cited most often in relation to the very weakest link in the Protective Framework, “Monitoring and oversight,” which had ratings of poor-to unacceptable three-quarters of the time.

On the other hand, in another area of the Protective Framework, “Building capacities of those closest to the child,” three-quarters of the strategic planning ratings were good or better. With respect to “Service delivery,” almost two-thirds of the time the ratings were at least adequate.

These and other statistic can be seen in Figure 8, below. This figure shows programming practices that were rated as “Good” to “Excellent” divided by all the ratings to show the percentage of good or better quality programming practice. Readers can see that, for instance:

With respect to UNICEF’s use of its facilitative role, 64% of the time the efforts were rated as good or better with respect to legislation and almost 70% as good or better with respect to strengthening government commitments.


On the other end of the spectrum, only a quarter of the strategic planning efforts saw adequate ratings with respect to monitoring and oversight; only a third of the instances relating strategic planning to legislation were rated as adequate.

Figure 8: Program Practices/Protective Environment Elements

Programme practice Government

commitments Legislation

Harmful attitudes, practices

Public engagement

Child participation

Building capacities

Service delivery

Monitoring &

oversight

Percentage (1.5 or higher-rated divided by all values)*

Context 0.00% 0.00% 58.06% 100.00% 82.14% 87.50% 85.71% 0.00%

Intersectorality 50.00% 0.00%

Programme communications

100.00% 43.75% 40.91% 50.00% 39.39% 0.00%

Facilitative role 68.42% 64.10% 100.00% 100.00% 50.00% 91.30% 100.00% 55.56%

Community ownership 66.67% 37.50% 66.67% 50.00% 50.00% 45.45% 100.00% 55.56%

Child participation 0.00% 40.00% 44.00% 50.00%

Strategic planning 53.85% 33.33% 40.00% 66.67% 75.00% 72.00% 61.54% 25.69%

Knowledge management

0.00%

* Some cells are empty because the value in the cell was zero. Some cells show 100% when the sample size is negligible

Limitations and Biases The incorporation of Programming Practices Reporting Framework into the meta-evaluation makes the analytical framework responsive to the actual content of the reports, which is generally more reflective of programming design and execution than of evidence based good practices. However, it is a blunt instrument: each rating of zero to three represents the average for the report, and so can only be partially illuminating. On the other hand, to test the tool’s ability to deliver replicable results, two individuals applied it to the same reports during the pilot stage. The ratings and the investments that were tagged as evidence addressing the meta-evaluation objectives were largely identical.


Appendix Five: Meta-Eval Reports/Evidence Ratings

Re

gio

n

Co

un

try

CP

th

em

e

CP

se

con

d t

he

me

Dat

e

UN

ICEF

Eval

uat

ion

rat

ing

Re

po

rt s

eq

ue

nce

RB

M le

vel

Co

mp

aris

on

gro

up

s

Dat

a ti

me

se

rie

s

Stan

dar

ds

use

d t

o

anal

yze

re

sult

s

Co

nse

qu

en

ces

anal

yze

d

Dat

a/an

alys

is

dis

aggr

ega

tio

n

Stat

isti

cal a

nal

ysis

Ove

rall

ran

kin

g

CEE

/CIS

Serbia and Montenegro

d8 q7

20

06

3.49 Evaluation of the UNICEF Impact on the Juvenile Justice System Reform in Three Countries in the Regions (Serbia, Romania, Tajikistan) 2

00

6/0

06

3 0 0 2.8 2.8 2 0 1.514

CEE

/CIS

Bosnia & Herzegovina

d7 h3

20

03

2.85 A Multidisciplinary Approach to Combat Domestic Violence and Other Form of Violence Against Women and Children 2

00

3/0

08

1.2 1.5 1.5 1.5 1.5 1.5 1.5 1.457

CEE

/CIS

Azerbaijan d8 q7

20

06

1.85 Assessment of Effectiveness of Juvenile Justice Trainings for Police Workers in Azerbaijan

20

06

/00

7

0 0 0 1.5 2 2.5 2.5 1.214

CEE

/CIS

Belarus d4 q7

20

04

2.56 Evaluation of the Deinstitutionalisation Project, Including Alternative Schemes and Family Support 2

00

4/0

07

2 0 2 3 2.5 2 2.5 2

CEE

/CIS

Tajikistan d8 q7

20

06

3.49 Thematic Evaluation of UNICEF’s Contribution to Juvenile Justice System Reform in Tajikistan

20

06

/00

6

3 0 0 2.8 2.8 2 0 1.514


Re

gio

n

Co

un

try

CP

th

em

e

CP

se

con

d t

he

me

Dat

e

UN

ICEF

Eval

uat

ion

rat

ing

Re

po

rt s

eq

ue

nce

RB

M le

vel

Co

mp

aris

on

gro

up

s

Dat

a ti

me

se

rie

s

Stan

dar

ds

use

d t

o

anal

yze

re

sult

s

Co

nse

qu

en

ces

anal

yze

d

Dat

a/an

alys

is

dis

aggr

ega

tio

n

Stat

isti

cal a

nal

ysis

Ove

rall

ran

kin

g

CEE

/CIS

Romania d12

q14

20

02

1.79 Mid Term Review UNICEF Romania Report on Child Protection Programme

20

02

/00

1

1 0 2 1.5 1.5 0.5 0.5 1

CEE

/CIS


d8 q7

20

06

2.23 Mid-Term Evaluation of the “Children’s Chance for Change” Project —A Juvenile Justice Initiative in Serbia and Montenegro

20

06

/00

3 2 0 1 2 2 0 0 1

CEE

/CIS

Tajikistan d4 q7

20

06

1.64 Mid- Term Evaluation of the Efficiency of UNICEF Deinstitutionalisation Project

20

06

/00

8

1 0 0.5 1.5 1 1.5 0 0.786

CEE

/CIS


d10

q7

20

05

2.31 External Evaluation of the Day Care Center “TISA” for Children with Disabilities, Bijelo Polje, Montenegro

20

05

/00

3 1.5 0 0.5 1.5 2 1.5 0.5 1.071

CEE

/CIS

Bosnia and Herzegovina

d12

q14

20

03

2.64 Evaluation of the program “Improved Mechanisms of Child Protection”

20

03

/00

5

2.5 0 1.9 1.5 2 1.6 1.6 1.586

CEE

/CIS

Bosnia and Herzegovina

d7 h3

20

03

1.66 UNICEF Project on Preventing Trafficking of Women and Children, Gender-Based Violence, and HIV/AIDS in BiH 2

00

3/0

13

1.5 0 0 1.5 3 0 0 0.857


Re

gio

n

Co

un

try

CP

th

em

e

CP

se

con

d t

he

me

Dat

e

UN

ICEF

Eval

uat

ion

rat

ing

Re

po

rt s

eq

ue

nce

RB

M le

vel

Co

mp

aris

on

gro

up

s

Dat

a ti

me

se

rie

s

Stan

dar

ds

use

d t

o

anal

yze

re

sult

s

Co

nse

qu

en

ces

anal

yze

d

Dat

a/an

alys

is

dis

aggr

ega

tio

n

Stat

isti

cal a

nal

ysis

Ove

rall

ran

kin

g

CEE

/CIS

Romania d4 q14

20

03

2.08 Evaluation of Projects Aimed at the

Prevention of Child Abandonment in Maternities

20

03

/01

0 2.5 0 0.9 3 2.6 1.5 0.5 1.571

CEE

/CIS

Moldova d12

q14

20

04

1.66 Evaluation of the Development of Integrated Community Based Child Protection Services project (in Ungheni) 2

00

4/0

07

1.5 0 0 1 2 1 0 0.786

EAP

RO

Cambodia d12

q14

20

04

2.8 External Evaluation of the Child Protection Network

20

04

/03

0

1.5 0 2.5 3 2.5 2 1.5 1.857

EAP

RO

Cambodia d10

q14

20

05

2.84 Evaluation of Disability Project

20

05

/16

3 0 2.5 3 3 3 0 2.071

EAP

RO

Indonesia d12

q14

20

04

2.1 Institution Building & Mainstreaming Child Protection in Indonesia UNICEF Supported Child Protection Bodies (LPAs) 2

00

3/0

23

0 0 0 0 2.5 0 0 0.357

EAP

RO

Cambodia d12

q14

20

05

2.62 Evaluation of Training of Social Workers

20

05

/17

2.6 1.5 2 3 3 2 1.9 2.286


Re

gio

n

Co

un

try

CP

th

em

e

CP

se

con

d t

he

me

Dat

e

UN

ICEF

Eval

uat

ion

rat

ing

Re

po

rt s

eq

ue

nce

RB

M le

vel

Co

mp

aris

on

gro

up

s

Dat

a ti

me

se

rie

s

Stan

dar

ds

use

d t

o

anal

yze

re

sult

s

Co

nse

qu

en

ces

anal

yze

d

Dat

a/an

alys

is

dis

aggr

ega

tio

n

Stat

isti

cal a

nal

ysis

Ove

rall

ran

kin

g

EAP

RO

Laos d5 q14

20

06

3.54 Review of the Street Children Project

20

06

/00

6 2.5 0 1.5 3 3 3 2.5 2.214

EAP

RO

Cambodia d7 h3

20

06

1.72 Lessons Learned Review on the Law Enforcement Against Sexual Exploitation and Trafficking of Children Project 2

00

6/0

35

1.5 0 0.5 2 1 0.5 0 0.786

EAP

RO

Cambodia d1 q8

20

03

1.86 Child Rights in Post Conflict Areas (Access to Health, Water & Sanitation), Aniong Veaeng and Trapeang 2

00

3/0

11

0.5 0 1 0.5 0 1 0 0.429

EAP

RO

Cambodia d7 h3

20

01

2.07 Support Systems to Reintegration Programmes for Children and Women Needing Special Protection

20

01

/00

7 1 0 0 2.5 2.5 0.8 0 0.971

EAP

RO

Indonesia d1 i10

20

03

2.54 Psychosocial Interventions, Evaluation of UNICEF Supported Projects (1999–2001)

20

03

/80

0 0.2 1 0.5 3 1.5 1.5 0.2 1.131


Re

gio

n

Co

un

try

CP

th

em

e

CP

se

con

d t

he

me

Dat

e

UN

ICEF

Eval

uat

ion

rat

ing

Re

po

rt s

eq

ue

nce

RB

M le

vel

Co

mp

aris

on

gro

up

s

Dat

a ti

me

se

rie

s

Stan

dar

ds

use

d t

o

anal

yze

re

sult

s

Co

nse

qu

en

ces

anal

yze

d

Dat

a/an

alys

is

dis

aggr

ega

tio

n

Stat

isti

cal a

nal

ysis

Ove

rall

ran

kin

g

EAP

RO

Thailand d1 i10

20

06

2.07 Evaluation of Psychosocial

Rehabilitation Opportunities, Training and Enhancement for Children Along the Thai/Burma Border N

on

e gi

ven

2.5 0 0 2,5 1.5 1.5 0 1.144

EAP

RO

Indonesia d7 d2

20

06

2.39 Evaluation of District-Based Initiatives on the Prevention of CSEC and Child Labour (in Indramayu, Surakarta and Probolonggo, Tulungagung)

20

06

/01

8 1 1 0.2 1.5 0.2 0.2 0.2 0.614

EAP

RO

Vietnam d5 q14

20

02

2.57 Thao Dan Street Children Program

20

02

/81

6

2.3 0 0 2 2 1.5 1.5 1.329

EAP

RO

Thailand d8 q7

20

07

1.8 Preliminary Assessment of the Use of Family and Community Conference Group as an Alternative Approach in Juvenile Justice 2

00

7/8

01

1.5 0 0 3 1 0.5 0 0.86

EAP

RO

Thailand d17

g1

20

06

2.33 Documentation and Evaluation of UNICEF Thailand’s Post-Tsunami Psychosocial Initiatives Report 2

00

6?

1.5 1 0 3 2.5 0 0 1.143


Re

gio

n

Co

un

try

CP

th

em

e

CP

se

con

d t

he

me

Dat

e

UN

ICEF

Eval

uat

ion

rat

ing

Re

po

rt s

eq

ue

nce

RB

M le

vel

Co

mp

aris

on

gro

up

s

Dat

a ti

me

se

rie

s

Stan

dar

ds

use

d t

o

anal

yze

re

sult

s

Co

nse

qu

en

ces

anal

yze

d

Dat

a/an

alys

is

dis

aggr

ega

tio

n

Stat

isti

cal a

nal

ysis

Ove

rall

ran

kin

g

ESA

RO

Tanzania d12

q14

20

05

3 Impact Assessment of the Most

Vulnerable Children (MVC) Community Based Care, Support, and Protection in Musoma Rural

20

04

/02

1 2.5 1.5 0 2 2.5 1.5 1.5 1.644

ESA

RO

Zambia d12

j2

20

03

2.34 Link Association for the Relief of Children (LARC) Community Action Support Project Evaluation

20

02

/01

1

3 0 0 0 2 0 0 0.714

ESA

RO

Swaziland d12

j2

20

05

1.94 Evaluation of Lihlombe Lekukhalela Child Protectors

20

05

/00

5

0 3 0 3 3 1.5 3 1.929

ESA

RO

Zambia d8 q7

20

05

3.9 Report on Child Justice in Zambia

20

05

/00

2

3 3 3 3 3 3 3 3

ESA

RO

Eritria d4 d1

20

04

2.72 Mahzel Project for Social Reintegration and Protection of Disadvantaged Children as Future Development Stakeholders 2

00

4/0

08

2.4 1.2 2 2.3 3 2 2 2.129


Re

gio

n

Co

un

try

CP

th

em

e

CP

se

con

d t

he

me

Dat

e

UN

ICEF

Eval

uat

ion

rat

ing

Re

po

rt s

eq

ue

nce

RB

M le

vel

Co

mp

aris

on

gro

up

s

Dat

a ti

me

se

rie

s

Stan

dar

ds

use

d t

o

anal

yze

re

sult

s

Co

nse

qu

en

ces

anal

yze

d

Dat

a/an

alys

is

dis

aggr

ega

tio

n

Stat

isti

cal a

nal

ysis

Ove

rall

ran

kin

g

ESA

RO

Tanzania d12

q14

20

07

2.39 An Assessment of the Impact of

Implementing the MVC Programme and the Operation of the MVC Funds and the Potential for Scaling Up to Provide National Coverage of Social Protection for Children

20

07

2.5 1.5 0.5 2.5 3 1.5 1.5 1.857

ESA

RO

Ethiopia d5 q14

20

03

2.74 Street Children and Street Mother, Partnership Programme for 2004 in Addis Ababa

20

03

/04

0

3 3 2.5 3 3 2.5 3 2.857

ESA

RO

Angola d1 g9

20

05

2.26 Evaluation of Data Collection Project on the Impact of Armed Conflict on Children (SSRC/UNICEF)

No

ne

give

n

3 0 0.5 3 3 0.5 0.5 1.5

ESA

RO

Tanzania d12

q14

20

04

2.29 Impact Assessment of MVC Support Programme Implementation in Karagwe, Kisarawe, and Magu

20

04

/02

0 3 1.5 2.5 2.5 3 2.5 0.5 2.214

ESA

RO

Zambia d12

j2

20

02

2.08 Report on the Evaluation of the UNICEF-Supported Chikankata CBOSP & OVC Training Projects

20

02

/01

0

3 0 1 1.5 .2 1 0 0.957


Re

gio

n

Co

un

try

CP

th

em

e

CP

se

con

d t

he

me

Dat

e

UN

ICEF

Eval

uat

ion

rat

ing

Re

po

rt s

eq

ue

nce

RB

M le

vel

Co

mp

aris

on

gro

up

s

Dat

a ti

me

se

rie

s

Stan

dar

ds

use

d t

o

anal

yze

re

sult

s

Co

nse

qu

en

ces

anal

yze

d

Dat

a/an

alys

is

dis

aggr

ega

tio

n

Stat

isti

cal a

nal

ysis

Ove

rall

ran

kin

g

ESA

RO

Somalia d12

d8 2

00

5 2.73 Report of the Organizational

Development Assessment and Capacity Building of UNICEF Child Protection Partners in Somalia/land 2

00

5/8

11

3 0 0.5 3 3 3 3 2.214

Glo

bal

Global d2 p1

20

03

3.67 Education as a Preventive Strategy Against Child Labour: Evaluation of the Cornerstone Programme of UNICEF's Global Child Labour Programme

20

03

/80

1 2.5 0 1.5 2.5 2.5 2.5 1.5 1.857

Glo

bal

Global d3 d1

20

06

3.87 Evaluation of UNICEF’s Support to Mine Action

20

05

/00

5 3 0 2.5 3 3 3 0 2.071

MEN

A

Iran d17

g1

20

07

2.15 Evaluation of Emergency Child Protection Programs in Bam

No

ne

give

n 1.5 0 0 2 2 2 2 1.357

MEN

A Sudan d1 q8

20

04

3.24 Evaluation — Grassroots Peace-Building Project

20

04

/00

6

3 1.5 0.5 3 3 2.5 0.5 2


Re

gio

n

Co

un

try

CP

th

em

e

CP

se

con

d t

he

me

Dat

e

UN

ICEF

Eval

uat

ion

rat

ing

Re

po

rt s

eq

ue

nce

RB

M le

vel

Co

mp

aris

on

gro

up

s

Dat

a ti

me

se

rie

s

Stan

dar

ds

use

d t

o

anal

yze

re

sult

s

Co

nse

qu

en

ces

anal

yze

d

Dat

a/an

alys

is

dis

aggr

ega

tio

n

Stat

isti

cal a

nal

ysis

Ove

rall

ran

kin

g

MEN

A

Somalia d1 f5 2

00

6

3.18 Evaluation of the ELMAN-UNICEF Partnership in Support of the Reintegration and Rehabilitation of Children Associated with Militia 2

00

6/0

05

3 0 0.5 3 3 2.5 1.5 1.929

MEN

A Somalia d1

2 q14

20

06

2.05 Community-Based Child Protection Programme Evaluation and Review

20

06

/01

0 0 0 0 0 3 1.5 0 0.643

MEN

A

Iran d17

g1

20

07

2.87 Bam Emergency Programme 2003–2006

20

07

2.5 0 2 3 3 2 2 2.071

MEN

A India d1

7 g1

20

02

2.81 Report on Appreciative Assessment of Psychosocial Interventions in Gujarat

20

02

/80

0 2.5 0 1 3 2.5 1.5 0 1.501

MEN

A Egypt d1

2 q14

20

04

2.33 Urban Child Protection in Cairo and Alexandria: An Assessment of Achievements and Prospects

20

04

/00

2 0.5 0 1 2 1 1 0.5 0.857

MEN

A Algeria d1 i1

0

20

03

2.68 Psychosocial Care for Children Traumatized by Terrorist Violence

20

03

/00

2

3 0 2 3 2.5 1.5 2 2


Re

gio

n

Co

un

try

CP

th

em

e

CP

se

con

d t

he

me

Dat

e

UN

ICEF

Eval

uat

ion

rat

ing

Re

po

rt s

eq

ue

nce

RB

M le

vel

Co

mp

aris

on

gro

up

s

Dat

a ti

me

se

rie

s

Stan

dar

ds

use

d t

o

anal

yze

re

sult

s

Co

nse

qu

en

ces

anal

yze

d

Dat

a/an

alys

is

dis

aggr

ega

tio

n

Stat

isti

cal a

nal

ysis

Ove

rall

ran

kin

g

MEN

A Yemen d1

2 q14

20

06

1.7 Evaluation of the Social Work- Course Within the Faculty of Arts University of Sana’a

20

06

/00

7

1 0 0 1 2.5 1 0.5 0.857

MEN

A Ethiopia d1

2 q14

20

04

2.03 Mid term Review — Gender and Child Protection Programme

20

04

/02

9

2 0 0 0 1.5 1.5 0 0.714

MEN

A Yemen d8 q7

20

04

2.17 Evaluation of UNICEF Support to Juvenile Justice System

20

04

/00

5 0.5 0 0.5 1.4 1.5 1.5 0.5 0.843

MEN

A

Egypt d9 h3

20

02

1.61 Protection of the Girl Child and Eradication of Malpractices

20

02

/00

1 0 0 0 0.5 1.5 1.5 0 0.504

RO

SA

Bangladesh d9 f5

20

05

2.31 Endline Report on Kishori Abhijan: An Intervention Research Study on Adolescent Girls’ Livelihood

20

05

/83

4 2.6 2.6 2.6 2.6 2.7 2.6 2.5 2.6

RO

SA Bangladesh d7 f5

20

05

2.14 Final Evaluation of the Empowerment of Children in Need of Special Protection Project 2

00

5/8

3

6

1.8 0 0.5 0.5 0.5 0.5 0.5 0.614


Re

gio

n

Co

un

try

CP

th

em

e

CP

se

con

d t

he

me

Dat

e

UN

ICEF

Eval

uat

ion

rat

ing

Re

po

rt s

eq

ue

nce

RB

M le

vel

Co

mp

aris

on

gro

up

s

Dat

a ti

me

se

rie

s

Stan

dar

ds

use

d t

o

anal

yze

re

sult

s

Co

nse

qu

en

ces

anal

yze

d

Dat

a/an

alys

is

dis

aggr

ega

tio

n

Stat

isti

cal a

nal

ysis

Ove

rall

ran

kin

g

TAC

RO

Guyana d12

q14

20

02

1.92 Participatory Evaluation for the “Women and Children at Risk” Project of St. Francis Xavier

20

02

/80

0 1 0 2 1.5 2 0.5 0 1.001

WC

AR

O Liberia d1 f5

20

07

2.68 Evaluation of the Reintegration Component of the DDRR Programme for Children 2

00

7 3 1.5 2.5 3 3 2.5 2.2 2.529

WC

AR

O Sierra Leone d1 f5

20

06

2.48 Community Based Reintegration: Programme Evaluation

20

06

/00

2 1.5 1.5 0.5 2.5 2.3 2 2.5 1.829

WC

AR

O Ghana d5 h3

20

03

2.14 Evaluation of the Protection Component of the Rights Promotion and Protection Programme

20

03

/01

1 3 0 3 3 3 3 2.4 2.486

WC

AR

O Sierra Leone d1 d4

20

03

1.92 Evaluation of Alternative Care Programmes

No

ne

give

n

0.5 0 0 1 1 1.5 0.5 0.644

Totals 111.1 27.8 56.1 124 130 92.26 57 85.47

Counts 59 59 59 59 59 59 59 59

Averages 1.883 0.471 0.95 2.11 2.2 1.5637 0.97 1.449


Tsunami reports

These reports were reviewed for the “Emergenices” chapter. They were not included in the formal meta-evaluation, however, as they came too

late. The 2004 Indian Ocean Tsunami Disaster: Evaluation of UNICEF’s Response (Emergency and Initial Recovery Phase)

Indonesia (May 2006) by Alexander et al.

Maldives (May 2006) by Allaire et al.

Sri Lanka (May 2006) by Lewis Sida et al.


Appendix Six: Frequency of Evidence, Protective Environment Elements Figure 9, below, shows the number of instances that any of the eight components of the Protective Environment or any of their sub-components addressed the meta-evaluation objectives in a meaningful way. “Meaningful” was measured by the level of instructional information that any investment carried, negative or positive, addressing the objectives of the meta-evaluation. In Figure 9, the horizontal axis shows the number of occurrences, or frequency, of evidence; the vertical axis shows the Protective Environment Framework components and sub-components.


Figure 9: Frequency of Evidence, Protective Environment elements


Appendix Seven: Thematic Overview, Meta-Evaluation Reports

Child Protection Primary and Secondary Themes Count of Overall

D 12 Multi-thematic child protection 18

Juvenile justice 1

Orphans and affected children 3

Social protection; social safety nets; conditional cash transfers (CCT) 14 D 01 Armed conflict, child soldiers 10

Children without caregivers; institutionalization 1

CRC/Child rights promotion, reporting, and monitoring 2

EMOPs M&E/data systems 1

Non-formal, vocational and life skills 3

Mental health 3 D 08 Juvenile justice 7

Child rights legal and policy/administrative reform; CRC compliance; policy and law analysis 7 D 07 Sexual exploitation, trafficking 6

Child labor 1

Gender-based violence; sexual exploitation; gender and justice 4

Non-formal, vocational and life skills 1 D 17 Psychosocial support 4


Family separation and reunification; physical protection; child protection; psychosocial issues 4 D 04 Children without caregivers; institutionalization 4

Armed conflict, child soldiers 1

Child rights legal and policy/administrative reform; CRC compliance; policy and law analysis 2

Social protection; social safety nets; conditional cash transfers (CCT) 1 D 05 Street children 4


Social protection; social safety nets; conditional cash transfers (CCT) 3 D 09 Harmful traditional practices 2


Non-formal, vocational and life skills 1 D 10 Disabilities 2

Child rights legal and policy/administrative reform; CRC compliance; policy and law analysis 1

Social protection; social safety nets; conditional cash transfers (CCT) 1 D 02 Child labor 1

Multi-thematic sectoral, including supply as a program strategy, capacity building and training; HRBAP 1 D 03 Land mines/Unexploded ordinances 1

Armed conflict, child soldiers 1

Grand total 59


UNICEF Evaluation and Research Database, Child Protection Classifications D Child protection D1 Armed conflict, child soldiers D2 Child labor D3 Land mines; unexploded ordnance (UXO) D4 Children without caregivers; institutionalization D5 Street children D6 Violence and abuse D7 Sexual exploitation; trafficking D8 Juvenile justice D9 Harmful traditional practices D10 Disabilities D11 Program communication, media analysis CP D12 Multi-thematic CP D13 Child protection surveys and status; data systems; Situation Analysis D14 Child protection sectoral or partnership review/evaluation D15 Other D16 Migration D17 Psychosocial support D18 Birth registration



Appendix Eight: How Large Are the Programs and the Evaluations?

Introduction The credibility of evaluation evidence can be measured in relation to the evaluation’s plan for surveying a subset of the program’s target population. This is the evaluation’s sampling plan: of the universe targeted for an intervention, what portion of that universe can be tapped for research purposes to get a generally accurate picture of what is going on?

Target Population and Sampled Population With a few exceptions, the reports that were available for the meta-evaluation provide limited information on the relationships between target and sampled populations on which their analyses are based. To reprise from the main report:

Less than half the reports discuss the number of stakeholders consulted for the evaluation in relation to the number of stakeholders targeted by the program.85

Of these reports, only 20% describe the rationale for how the stakeholders consulted for the research were selected (sampling plan).86

These information gaps leave readers wondering how to value the evaluation findings, how much they might be applicable to the broader target demographic and in what circumstances, and whether an evaluation’s reach was suited to the scale of the programme or project being evaluated. This state of play is illustrated in the following chart:

85 Further, the number of stakeholders targeted by the program is more commonly given than the number of

stakeholders consulted for the evaluation (30% more often). 86 In most cases, those consulted are duty bearers, not rights holders. Significant exceptions to this are the Ghana,

Tanzania, and Viet Nam reports.


Stakeholders Number of Reports Percentage

Targeted by evaluation:

- Known 43 72.88%

- Unknown 16 27.12%

Targeted by program:

- Known 33 55.93%

- Unknown 26 44.07% Targeted by Evaluation AND Program

- Both unknown 10 16.95%

- Both known 27 45.76%

- One known 22 37.29%

The meta-evaluation analysis of sampling practices also raises a question about the scale of the evaluations that CP is investing in. Fig. 10 (below), is a side-by-side view of the numbers of stakeholders targeted for the respective evaluations and programmes. The stakeholder numbers are shown on the horizontal axis; the vertical axis lists programmes by country and sequence number (e.g. Somalia 2005/811).87 As shown, there are plenty of instances when this information was not supplied in the evaluation reports. But what might also stand out is the relatively small scale of the evaluations.

87 Appendix Five lists all reports by Country, Title, Year, and Sequence Number.


Figure 10: Scale of Evaluations versus Scale of Programmes/Projects


Appendix Nine: Stakeholders’ Initial Recommendations

Introduction These recommendations were generated by members of the Child Protection Section, UNICEF Global Headquarters, during a one-day workshop at which the meta-evaluation methodology and findings were presented (April 2008). Stakeholders marked them as preliminary pending a phased roll out and deeper study of the meta-evaluation findings and conclusions. There was some discussion of a management response being developed as part of this longer-term process. The evaluator contributes two recommendations relating to improving the evidence base and using askSam.

Recommendations a. Ensure that the Child Protection Strategy and Meta-evaluation are understood to be complimentary Any presentation of either the Strategy or the Meta-evaluation should reference the other as context. b. Tailor presentations of the findings to different target groups Task force is forming to oversee this process, which will reach out to internal and external Child Protection partners. This will include a seven-week e-conference, each week addressing specific elements of the Protective Environment Framework findings. A 2-3 page synopsis of the report, beyond the Executive Summary, may be developed for different target groups. Evaluation Office focal point will also share findings on the evidence base issues with Regional Monitoring and Evaluation Officers during a global meeting in early May 2008. Child Protection Division is circulating the findings immediately to Child Protection Regional Advisors for interpretation and recommendations. c. Establish a Child Protection Monitoring and Evaluation Resource Group (MERG) Use the meta-evaluation as a platform for developing a MERG (inter-agency technical groups) by sharing findings with selected partners. MERGs can also work together to standardize indicators and become a technical nucleus for Child Protection. d. Establish a Model Terms of Reference for Child Protection evaluation reporting Also the work of the task force, to identify the reporting issues of first importance. An evaluation toolkit is also part of this recommendation.


e. Develop a results framework To operationalize the Protective Environment Framework, Evaluation Office focal point made this recommendation. f. Follow up on four specific findings Certain findings were called out for further examination: gender bias, missing child participation at the program design stage, that buy-in to Child Protection issues is constrained by a lack of basic needs, and the gap in investments addressing programme communications (Communication for Development, or “C4D”). Regarding C4D, since there is a perception among staff that Child Protection invests widely in programme communications-related activities, it was discussed that there could be a need to consolidate Child Protection-related programme communications knowledge-practice. From the evaluator: g. Ask managers for the back-story on missing pieces The broad institutional contexts of investments (e.g. donor financing, host government sensibilities, unworkable levels of abstraction in evaluation reporting requirements or in results based approaches) are absent in the evaluation reports; without them, matters such as the neglect of baselines, indicators, monitoring, and enforcement, are practically inexplicable. Focus groups with Child Protection and Evaluation managers in the country offices would be useful now in shedding light on these institutional factors. How are managers’ hands tied, for example, on the matter of baselines? Where are the bottlenecks on monitoring? Why are examples of rights based approaches to programming, particularly with respect to stakeholder participation, so lean?

h. Use the askSam database for ongoing programme evaluations askSam is essentially ready for use as an ongoing program evaluation tool. New evaluations could be added annually, coded and analyzed just as the first 59 have been, so that areas that are flagged for development or improvement now can be tracked over the next five years. In other words, additions to the database moving forward would allow trendline and other types of statistical analysis. Consistent inputs over the next five years are simple and cost-effective for two reasons. First, 80-85% of the work in establishing this (and any) database is already done: the taxonomy has been established, codes have been set, and ratings scales have been designed for (i) the quality and effectiveness of programming design and execution, and (ii) evidence quality. Second, if 50% of each year’s harvest of Child Protection evaluations continued to be entered into the database per year –roughly akin to the meta-evaluation percentages-- this is an average of 10 reports per year. The process of coding, assigning ratings to 10 reports, and assessing trends, is probably on the order of three to five days of work. This use of the initial investment might proceed in collaboration with (a) UNICEF Evaluation Office, a stakeholder particularly on the matter of improvements to evidence quality, (b) Knowledge Management, a stakeholder in the systematizing and managing and leveraging of UNICEF’s knowledge stores, and (c) Human Resources, a stakeholder in maintaining and enhancing programming skills. Further reflections on using askSam are provided in a separate technical note.

Documents

UNICEF Child Protection Meta-Evaluation › french › videoaudio › PDFs › Final_CP... · 2019-11-12 · UNICEF Child Protection Meta-Evaluation Final 15 May 2008 Commissioned