www.coneval.gob.mx

February 2009

The Process Evaluation for Federal Social Programs

Mexico's Evaluation Strategy for Social Policy and Programs

1. Background/Preliminaries– The Mexican Evaluation Experience 2001 -2006

– CONEVAL and the 2007 evaluations

2. Process Evaluation– Objectives

– Methodology

– Testing 2008/2009

2

Agenda

• Important facts regarding evaluation in Mexico:– Progresa 1997– Federal Budget Bill for 2001 and External Evaluations for

Federal Programs– Mexican Social Development Law (2003)

• 2004-2007:– CONEVAL (2006)– Results based Budget and the Performance Evaluation

System (SHCP)– Guidelines for the Evaluation of Federal Programs (2007)– Annual Evaluation Program – In 2007, for the first time, a single type of evaluation was

applied to more than 100 federal programs

Preliminaries

The Mexican Council for the Evaluation of Social Policy (CONEVAL), has the following objectives:

1. Establish guidelines and criteria for the definition and measurement of multidimensional poverty

2. The regulation and coordination of the evaluation process for social policy and programs

CONEVAL is a decentralized public federal agency with special characteristics in the Mexican federal administration

Secretary of Social Development (SEDESOL) A board of six academic researchers democratically elected by the Social Development CommissionAn Executive Secretary (with Evaluation and Poverty Areas)

Preliminaries

• Evaluate the links and consistency between federal objectives (national and policy sector plans) and the objectives of federal programs

Federal Planning and Strategic Objectives

• To determine and disseminate the methodology for its elaboration by all federal programs (logical framework)Indicators Matrix

• Define the types of evaluation to be applied to federal programs and formulate the Annual Evaluation Plan

Evaluation Types and Coordination

PreliminariesGeneral Guidelines for the Evaluation of Federal Programs

• On the basis of the Indicators Matrix, the analysis of the design of federal programs, in terms of its logical framing Design

• Analysis of design and of the general performance of the programs in terms of its targeting, coverage, planning, operation, results and perception Consistency and Results

• Analysis of indicators design in terms of consistency with objectives, pertinence and usefulness Indicators

• The systematic analysis of the program’s performance and progress through the realization of the indicators of its main objectives and purposePerformance

• Analysis of the program’s operation and the consistency of processes with objectives and results Process

• The estimation of the change in variables of interest on beneficiaries that are due strictly to the programs implementation Impact

PreliminariesTypes of Evaluations

- Aplica para los programas que se propongan incluir en el PPEF y aquellos programas presupuestarios ya existentes que propongan ampliación o modificación sustantiva

Impact Evaluation

- Aplica para los programas nuevos que están en su primer año de operación

-Aplica para los programas con al menos dos años de operación

- Aplica para los programas con al menos dos años de operación y que tengan ECR previamente

Impact Evaluation Impact Evaluation

Performance Evaluation Performance Evaluation

Year 0 Year 3Year 2Year 1New Program

Design EvaluationConsistency and Results Evaluation

(every 3 years)Process Evaluation

Impact Evaluation

PreliminariesTypes of Evaluations

Inputs•Financial Resources

•Human Resources •Physical Resources •Infrastructure

Outputs•Patients attended•Food baskets delivered

•Scholarships •Legal Services

Outcomes•Health Conditions

•Access to services

Implementation and Management

Impact

Evaluation General Framework

Theme Findings

Compliance with norms

84% of programs are compliant with established norms on the distribution of benefits.

89% have documental evidence on established processes for federal programs

Operational Effectiveness

76% of programs have adequate procedures to process applications for beneficiaries

72% of programs have standardized and adequate procedures for the selection process of projects and beneficiaries

Physical and organizational infrastructure and resources

80% of programs have an organizational structure that allows them to reach their objectives

80% of programs have systematic information to allow for an adequate follow up of beneficiaries and projects

84% of programs have efficacy indicators of their operation

61% of programs have systematic information for their administration and operation

General Findings on the Operation section of the Consistency and Results Evaluation

The Process Evaluation analyses the implementation and management of social programs using information obtained through documental and fieldwork evidence

The purpose of the Process Evaluation is to investigate if and how the implementation of a program contributes adequately to the achievement of the objectives formulated in its design

Two important attributes for the Process Evaluation Methodology:

• Some degree of comparability between programs• Comparability over time within programs

 

Process Evaluation

Elaborating TORs

Research on concepts,

methodology and approaches

Previous experiences of Evaluation of

Processes in Mexico

Review of International Experiences

WB input

Review and Proposal of

Evaluation of Processes

• Objective• Methodolo

gy• Indicators• Types of

instruments

Internal discussion and

comments

Evaluation of Processes Workshop

Elaboration of the 1st Terms of Reference

Draft

Delivery of the ToR draft for

comments

Integration of Final

Version of ToR for triaI

Hiring of a consultant for

the first trial of the ToR

Analysis of results and

integration of the final

version of ToR for its

application by mid 2009

Elaboration of TORs Some evaluations reviewed

Microrregiones 2004 y 2006Evaluation of Results of the Microfinance Fund to Rural Women (Evaluación de Resultados del Fondo de Microfinanciamiento a Mujeres Rurales-FOMMUR) Evaluation of Processes of Seguro Popular 2005-2006 (Evaluación de Procesos del Seguro Popular 2005-2006)National Evaluation of Processes of PROGAN 2003-2004 (Evaluación Nacional de Procesos del PROGAN 2003-2004)Evaluation of the PYME Fund 2006 (Evaluación del Fondo PYME 2006)Evaluation of the Fulfillment of Goals, Unit costs and addition of the Program of Human Development OPORTUNIDADES to the Rules of Operation 2005 (Evaluación del Cumplimiento de Metas, Costos Unitarios y Apego del Programa de Desarrollo Humano Oportunidades a las Reglas de Operación 2005)External evaluation of the Program of Support to Farming and Fishing Social Organizations 2005 (Evaluación Externa del Programa de Apoyo a las Organizaciones Sociales Agropecuarias y Pesqueras 2005 - PROSAP)

Process Evaluation Objectives

Overall:

To analyze, with fieldwork and documental evidence, the implementation of the federal social programs to identify the characteristics of its operational processes, specifying problems areas and best practices.

Specific Objectives:

Describe program’s implementation by means of explaining its operative processes in the different (government) levels in which these are carried out;

Investigate program’s operative processes to qualify them in terms effectiveness, opportunity, sufficiency and pertinence for the achievement of program’s objectives;

Identify and describe the problems that limit implementation, as well as the good practices that fortify their capacity;

Assess whether the processes established in the Program’s legal framework (Reglas de Operación) are adequate, applicable, deficient or insufficient, be it in their internal consistency and on their application.

Methodology

Detailed description of the program’s implementation scheme through the description of its composing operational processes and their characteristics in terms of their attributes Description and analysis of the legal framework for its operation Description and analysis of the observational (true) processes and comparison to the norm Formulating indicators to measure the attributes of processes Detailed description of problem areas and best practices

Information Documental sources Field work: in depth interviews, semi structured interviews, case studies, surveys (flexible)

PROCES EVALUATIONMethodology and information sources

Dissemination

Request of affiliation

Selection of beneficiaries

Production of goods or services

Distribution of goods or services

EVALUATION OF PROCESSES Evaluation General scheme

Delivery of goods or services

Processes/Attributes Effectiveness Opportunity Sufficiency Relevance

Monitoring/ Follow up on beneficiaries

Instruments for collecting information

Descriptive cards of processes

Informative Cards

Semi-structured interviews or in-depth interviews

Structured interviews

Focus Groups

Types of analysis

The information processing will be detailed based on the type and amount of information obtained regarding the attributes that are in the evaluation trials. Some of the possible analyses will be:

Comparisons between indicators

Descriptive analysis with qualitative information

EVALUATION OF PROCESSES Instruments for collecting information and types of analysis

Expected characteristics of the results

Comparability of attributes of the processes within different time periods

Comparability of attributes of the proccesses between different programs

Differentiation of results between government orders

Comparability of results between programs within the same order of government

EVALUATION OF PROCESSES Expected Results

The Terms of Reference will be tested to with the following social programs.– Programa Nacional Para La Actualización Permanente De Los Maestros  de Educación Básica en

Servicio (PRONAP)– Programa de Infraestructura básica para la atención de los Pueblos Indígenas (PIBAI)– Programa de Coinversión Social– Programa de Desarrollo Local– IMSS- Oportunidades

This test will allow to determine in detail the type, amount and quality of information available

to carry out the evaluation and comply with the characteristics that allow comparison of

indicators

Based on the results of the analysis of the data obtained in the Terms of Reference test, the

instruments for the collection of data and measurement of attributes will be improved and the

Terms of Reference adjusted, so there is a final version that can be implemented to evaluate

Social Programs processes in 2009 .

TERMS OF REFERENCE TESTING