11

The Cases of Chile and ColombiaPresentation to a LAC Region Summer Seminar

9 August 2006

Keith MackayIndependent Evaluation Group

(kmckay@worldbank.org)

22

What Does “Success” Look Like?What Does “Success” Look Like?

-- Why Countries Want an M&E -- Why Countries Want an M&E System System

To support budget decision-making To support budget decision-making = performance-based budgeting= performance-based budgeting

To support national and sectoral planningTo support national and sectoral planning

To design policies and programsTo design policies and programs

To assist sector ministries / agencies in their To assist sector ministries / agencies in their managementmanagement

To strengthen accountability relationshipsTo strengthen accountability relationships

33

Chile’s M&E System Chile’s M&E System -- Architecture -- Architecture

Designed, managed and used by HaciendaDesigned, managed and used by Hacienda

Developed incrementally, over past decadeDeveloped incrementally, over past decade

Performance indicators (Performance indicators (∑∑1,600) for all 1,600) for all government programs (1994)government programs (1994)

Government program evaluations (Government program evaluations (∑ ∑ 160) 160) -- these are desk reviews (1996)-- these are desk reviews (1996)

Rigorous impact evaluations (Rigorous impact evaluations (∑∑14) (2001)14) (2001)

Comprehensive Spending Reviews -- desk Comprehensive Spending Reviews -- desk reviews of all programs in a functional area reviews of all programs in a functional area (2002)(2002)

44

Chile’s M&E System Chile’s M&E System -- Strengths (1) -- Strengths (1)

‘‘Graduated’ approach to M&EGraduated’ approach to M&E

Evaluations conducted externally, in fully Evaluations conducted externally, in fully transparent process, and are highly credibletransparent process, and are highly credible

All M&E findings reported publicly and sent All M&E findings reported publicly and sent to Congressto Congress

M&E system closely linked to the M&E system closely linked to the information needs of Hacienda, especially information needs of Hacienda, especially for budget processfor budget process

Performance information used to set Performance information used to set performance targets for ministries -- these performance targets for ministries -- these are largely metare largely met

55

Chile’s M&E System Chile’s M&E System -- Strengths (2) -- Strengths (2)

High utilization of M&E findings by High utilization of M&E findings by Hacienda in the budget process and to Hacienda in the budget process and to impose management improvements on impose management improvements on ministries / agencies (see Table) ministries / agencies (see Table)

Utilization of government evaluations -- 2000 to 2004Utilization of government evaluations -- 2000 to 2004

Minor Minor adjustment, adjustment, modification, modification, relocation of relocation of programprogram

Change in Change in program program design or design or manage-manage-ment ment processprocess

Institutional Institutional relocation relocation of programof program

Substantial Substantial redesign of redesign of program or of program or of internal internal management management processesprocesses

Program Program terminationtermination

TOTALTOTAL

25%25% 39%39% 5%5% 21%21% 10%10% 100%100%

66

Chile’s M&E System Chile’s M&E System -- Challenges -- Challenges

Unevenness in quality of evaluations -- due Unevenness in quality of evaluations -- due to cost and time constraintsto cost and time constraints

Chile probably not spending enough on Chile probably not spending enough on evaluationsevaluations

Low utilization -- low ‘ownership’ -- of Low utilization -- low ‘ownership’ -- of Hacienda’s evaluations by sector ministriesHacienda’s evaluations by sector ministries

77

Colombia’s M&E System Colombia’s M&E System (SINERGIA)(SINERGIA) -- Architecture-- Architecture

SINERGIA is managed by the Department of SINERGIA is managed by the Department of National Planning (DNP), with strong support National Planning (DNP), with strong support from the Presidenciafrom the Presidencia

On-line sub-system -- SIGOB -- for On-line sub-system -- SIGOB -- for monitoring and reporting government monitoring and reporting government progress vis-à-vis Presidential Goalsprogress vis-à-vis Presidential Goals

Ambitious agenda of impact evaluations Ambitious agenda of impact evaluations ((∑∑15 underway) 15 underway)

DNP provides technical assistance to a few DNP provides technical assistance to a few ministries/agencies to develop M&E, and to ministries/agencies to develop M&E, and to municipalities to pilot SIGOB and municipalities to pilot SIGOB and performance-based budgetingperformance-based budgeting

88

Colombia’s M&E System Colombia’s M&E System -- Strengths-- Strengths

Very high utilization of SIGOB by President Very high utilization of SIGOB by President for oversight of ministers and ministries -- for oversight of ministers and ministries -- via performance targets -- and for via performance targets -- and for accountability, i.e. ‘social control’accountability, i.e. ‘social control’

Rigorous impact evaluations conducted Rigorous impact evaluations conducted externally, and have high credibilityexternally, and have high credibility

Collaborative approach between DNP and Collaborative approach between DNP and sector ministries/ agencies, and with sector ministries/ agencies, and with municipalitiesmunicipalities

Performance budget reports; efforts to Performance budget reports; efforts to further strengthen performance budgetingfurther strengthen performance budgeting

Efforts to engage with civil societyEfforts to engage with civil society

99

Colombia’s M&E System Colombia’s M&E System -- Challenges-- Challenges

Too high reliance on donor funding for Too high reliance on donor funding for SINERGIA --SINERGIA -- low level of government funding low level of government funding supportsupport

Insufficient reliance on M&E information to Insufficient reliance on M&E information to support national planning and budget support national planning and budget decision-making -- this may now be decision-making -- this may now be changingchanging

Weak coordination of M&E roles / functions Weak coordination of M&E roles / functions within DNP and with central and sector within DNP and with central and sector ministriesministries

SIGOB data quality perceived as lowSIGOB data quality perceived as low

1010

How to Define a Government M&E How to Define a Government M&E System as Being “Good Practice”System as Being “Good Practice”

Can be dangerous concept -- each country’s Can be dangerous concept -- each country’s starting point and desired end-point are starting point and desired end-point are unique unique

What a “successful” M&E system is NOT:What a “successful” M&E system is NOT: complex set of laws, decrees, regulationscomplex set of laws, decrees, regulations # performance indicators collected# performance indicators collected # rigorous impact evaluations conducted# rigorous impact evaluations conducted # rapid evaluations conducted# rapid evaluations conducted = these simply reflect the architecture of = these simply reflect the architecture of

the system or are measures of M&E effortthe system or are measures of M&E effort

1111

How to Define a Government M&E How to Define a Government M&E System as Being “Good Practice”System as Being “Good Practice”

Chile and Colombia have good-practice Chile and Colombia have good-practice M&E systems becauseM&E systems because

1.1. the quality of their M&E work is generally the quality of their M&E work is generally reliable, andreliable, and

2.2. in particular, the monitoring information in particular, the monitoring information and evaluation findings which they and evaluation findings which they produce are used intensivelyproduce are used intensively

High utilization reflects strong demand for High utilization reflects strong demand for M&E, and can be viewed as a predictor of M&E, and can be viewed as a predictor of an M&E system’s sustainabilityan M&E system’s sustainability

1212

Lessons (1)Lessons (1)

Lessons from Chile and Colombia are Lessons from Chile and Colombia are consistent with other countries’ experienceconsistent with other countries’ experience

1.1. Key role of powerful champion of M&EKey role of powerful champion of M&E

2.2. Opportunistic development of M&E systems, Opportunistic development of M&E systems, via continuous review and modification; via continuous review and modification; non-linear development of the systemsnon-linear development of the systems

3.3. Centrally-driven, by capable ministryCentrally-driven, by capable ministry

4.4. Incentives are key -- cultural change to Incentives are key -- cultural change to strengthen demand, achieve high utilizationstrengthen demand, achieve high utilization

5.5. Avoid competing systems – Planning, Finance Avoid competing systems – Planning, Finance

1313

Lessons (2)Lessons (2)

6.6. Build reliable ministry data systemsBuild reliable ministry data systems

7.7. Role of structural arrangements to ensure Role of structural arrangements to ensure M&E objectivity and qualityM&E objectivity and quality

8.8. Long-haul effort, requiring patienceLong-haul effort, requiring patience

9.9. Limitations of relying on laws, decrees, Limitations of relying on laws, decrees, regulationsregulations

10.10. An M&E system can be built and operated An M&E system can be built and operated are relatively low costare relatively low cost

Chile’s M&E system costs $0.75m p.a.Chile’s M&E system costs $0.75m p.a. Colombia’s system costs about $2m p.a.Colombia’s system costs about $2m p.a.

1414

Useful Resources (1)Useful Resources (1)

Chile: www.dipres.cl/fr_control.html

Colombia: www.dnp.gov.co/paginas_detalle.aspx?idp=266

Ernesto May et al. (eds.), Towards the Institutionalization of Monitoring and Evaluation Systems in Latin America and the Caribbean, World Bank/IADB, 2006. http://web.worldbank.org/WBSITE/EXTERNAL/COUNTRIES/LACEXT/0,,contentMDK:20893139~pagePK:146736~piPK:146830~theSitePK:258554,00.html (Disponible en Español)

Keith Mackay, Institutionalization of Monitoring and Evaluation Systems to Improve Public Sector Management, Independent Evaluation Group, World Bank, 2006. www.worldbank.org/ieg/ecd/institutionalizing_me.html (Disponible en Español)

1515

Useful Resources (2)Useful Resources (2)

Ariel Zaltsman, Experience with Institutionalizing M&E Systems in Five Latin American Countries: Argentina, Chile, Colombia, Costa Rica and Uruguay, Independent Evaluation Group, World Bank, 2006. www.worldbank.org/ieg/ecd/experience_five_la.html

Fernando Rojas et al, Chile: Análisis del Programa de Evaluación del Gasto Publico, World Bank, 2005. http://iris37.worldbank.org/domdoc/PRD/Other/PRDDContainer.nsf/WB_ViewAttachments?ReadForm&ID=85256D2400766CC785257155005CB26B&

World Bank website on Building Government M&E Systems: www.worldbank.org/ieg/ecd/