Upload
dokhuong
View
219
Download
1
Embed Size (px)
Citation preview
1
Monitoring and Evaluation: A Logical Frame
Evaluating the Impact of Projects and Programs
Beijing, China, April 10-14, 2006Shahid Khandker, WBI
2
What is M&E?
Monitoring and evaluation are tools that make it possible to identify and measure the results of projects, programs or policies
3
Why the Concern?To evaluate and adjust strategies and activitiesTo report on progress to interested parties, clients, taxpayers and the general publicTo identify and share with others best practices and lessons learnedTo improve the programming of new interventions and strategies
4
MonitoringProvides regular information on how things are workingDefn: A continuing function that uses: A) systematic data collection and analysis of specific indicators of progress.B) Provides management with indication of extent of progress towards goals
- achievement of deliverables-use of resources.
C) Contributes to performance improvement.D) Conducted by business unit.
5
Evaluation
Evaluation can only be done after a certain time and requires more thorough investigationsConducted by independent evaluators.Defn: A systematic and objective measurement of the results achieved by a project/program/policy in order to assess its relevance, the efficiency of implementation, its effectiveness, impact and sustainability.
6
Monitoring vs Evaluation
Monitoring assesses progress in implementation of ongoing programsEvaluation provides a snapshot against some benchmarks at a point in time of programs that may or may not be continuingMonitoring looks at progress relative to targets and assumes there is causalityEvaluation seeks to prove causality
7
Monitoring and Evaluation -RationaleMonitoring
Holds implementers accountable for delivery of inputsProvides basis for corrective actionProvides assessment of continued relevance
EvaluationAccountability - was money well spent?Learning - what could we do better next time?
8
… and an impact on development
Monitoring
Evaluation
Resources are mobilized…
… to undertake activities...
… the direct results of which...
… must have effects...
This chain is based on a series of logical relationships
(if… then) called the logframe
Monitoring and Evaluation, and the Chain of Results
9
Resources are mobilised
… to undertake activities...
… the direct results of which...
… must have effects…
… and an impact on development
building, training, delivery
budget funds
amount of services
provided, processes completed
Health, literacyconsumption,
life expectancy,
poverty
Allocation Inputs Outputs Outcomes Impact
Logframe of a Project or Program
10
What Type of Indicators Are Needed?-MDGs-
1. Eradicate extreme poverty and hunger2. Achieve universal primary education3. Promote gender equality and empower
women4. Reduce child mortality5. Improve maternal health6. Combat HIV/AIDS, malaria and other
diseases7. Ensure environmental sustainability8. Develop a Global Partnership for
Development
11
What Type of Indicators Are Needed? GOAL:Improve the health status of the population
through improved water supply
Impact Impact on living standards –Lower poverty; prevalence of malnutrition5-10 yrs
OutcomesAccess, usage, and satisfaction of beneficiaries - % hh with access to safe water supply; % hh with continuous water supply during the day1-3 yrs
Outputs Goods and services generated -Newly installed or rehabilitated water points;% wastewater treated
Annual
InputsResources (financial, material and human) provided - Real investment in water supply and sanitationAnnual
12
Monitoring the outcome and impact of development across sectors
Inputs
Impact
Outcomes
OutputsFocus on demand-side issues (data for decisions): logframe provides the foundation; largely project/program level: operationally-focused; tend to emphasize M&E at lower end of logframe [inputs/outputs); link to project design/implementation
Reflects a greater emphasis on: country-ownership, long-term capacity-building -also provides greater opportunities for civil society involvement and participatory M&E approaches. Link to PRSP
From project to program –The new development strategy
and its implications for M&E
13
Why Monitor?To ensure best use of scarce resourcesTo make user feedback integral to poverty reduction effortsTo mitigate the standard pitfalls in service provision to the poorTo enforce a concrete link between policy goals and policy inputsTo incorporate the many dimensions of poverty reduction into our decisions
14
What to Monitor
Identify few indicators, measure them well, and use the results for policy makersA prioritized list of input, output, outcome and impact indicators for monitoringDevelop a data collection system to institutionalize a monitoring system: budget and administrative data, facility and other survey data.
15
How to MonitorManage a monitoring system that integrates a variety of different types of information: MIS, surveys and censuses, and participatory exercisesDraw up a monitoring matrix which identifies:
-Data sources for each indicator-Frequency of measurements-The organization responsible for collecting
information.Collect information in a sequence that maximizes the complementaries between different types of data.Outputs:
-Annual progress report-Database for continuous monitoring-Desk studies
16
Features of Good IndicatorsRelevant; measure factors that reflect objectives
e.g. Number of eggs produced vs. Poverty incidenceDirect and unambiguous measure of progress
e.g. % Hh expenditures on health care vs. Immunization coverageVaries across areas, groups, over time; sensitive to changes in policies, programs and institutions
e.g. Life expectancy vs. Infant mortality ratesCan be tracked frequently; not too costly
e.g. Iron deficiency in pregnant women vs. Pre-natal care visits
Consistent with the decision making/budgetarycycle
17
How Many Indicators Should Be Monitored?
There is not a general rule but...It is more useful to identify a small number of indicators, measure them well and use the results for policy-making than to enumerate a long list of indicators, measure many of them badly and not use the results for policy decisions at all
18
At What Level of Disaggregation?Aggregate indicators are useful for giving an overall picture of country situation or cross-country comparisonsHowever, they tend to mask differences across areas, gender, or social groupAppropriate level of disaggregation depends on country conditions and the indicator itself
19
Types of DisaggregationGeographical
Rural/urbanAdministrative units (states, districts, municipalities, etc)Geoclimatic zones
GenderIncome or consumption levelSocially defined groups
20
Experience to DateLack of consistency between goals and indicators for monitoringMultitude of indicators without clear prioritizationUnbalance between intermediate (input/output) and final (outcome/impact) indicatorsInadequate/unclear indicators
21
Know,understand
and learn lessons
Manage, monitor
DecidePeople in charge
Manager,Operational
staff
Researcher
Know, be informed
Public, beneficiaries, contributors
4 mainpurposes
Purposes of an M&E
22
Key M&E Products
Before During At the end Afterwards
To check the design
To improve theimplementation
For accountabilitypurposes
To assessthe impact
Program
StartStart
23
Components of a M&E Strategy
Outcome-based monitoring systemComplemented with systematic impact evaluationCreating a feedback processBuilding capacity for monitoring and evaluationPromoting participation
24
Outcome-based Monitoring System
Setting goals and targets (including establishing the baseline) – where do we want to go?Identifying indicators that can be used to measure progress towards goalsCollecting data - what progress is being made? Providing feedback for decision making –what needs to be changed along the way
25
Monitoring System: Where to Start?
Stock taking:-Who are the actors? What are
their activities? What is their capacity? What are their roles and needs?Review the actions currently undertaken to build or re-enforce capacity.Review of needs in information
26
Monitoring System: What Limitations?
A series of issues are of concern:- Roles badly defined- Lack of coordination- Lack of reliability and relevance of
information- Difficulty in accessing information- Long delays in production of information- Lack of use of the data by the users
27
Example: Monitoring Poverty Reduction
28
Monitoring poverty reduction requires
Measuring povertyUnderstanding poverty reduction strategiesMonitoring outcomes and impacts—goes beyond project-based monitoringEvaluating m&e system and its capacity
29
Measuring povertyPoverty is a multi-dimensional concept—income, consumption, health, education, water, housing, sanitation, capability and empowermentPoverty is also dynamic—vulnerability and coping mechanismPoverty is also gender, caste, ethnic, and geographical based
30
Poverty reduction strategies
Market matters; state matters too – they can’t be substitutes; a right balance is required.Involve the poor in development process—poverty reduction requires both growth and targeted measuresDevelopment strategy needs (a) policies to improve the investment climate to raise income and employment growth, and (b) policies to equip and empower people to participate in growth.
31
Better investment climate and better participation
Macroeconomic stability and opennessGood governance and strong institutionsQuality infrastructureGrowth reduces poverty– but not all growth is equally pro-poorEmpowerment through social, human, and infrastructural investment to enable poor to participate in growth
32
Setting up an M&E system for Poverty Monitoring
1. Defining information needs. Deciding on indicators
2. What tools do we need? What tools do we have?
3. Defining institutional functions (including feedback processes)
4. Defining a 5-10 year work program5. Defining resource requirements
33
Step 2 – Measuring outcomes What tools do we need? What tools do we have?
34
Measuring outcomes - What tools do we need? What tools do we have?
Tools for measuring the extent and depth of poverty (poverty analysis)Tools for monitoring changes over time (poverty dynamics)Tools for comparing differences over space (poverty mapping)Participatory poverty monitoring (listening to the poor)
35
Measuring poverty and data quality
Tools: - A good household survey (there are various options - HBS, LSMS etc.)Measuring poverty is a data hungry exercise (means and distributions)Household Consumption (If you’re going to measure it - do it properly)Sampling and non-sampling errorMonitoring over time compounds the error problem
36
Monitoring changes over timeTools: Administrative data/MIS; Institution-based surveys; household surveysMonitoring leading indicators (Service Delivery)Key attributes of an ‘indicators survey
Simple to executeRapid reportingDisaggregatable to low levels
Core Welfare Indicators Questionnaire (CWIQ)
37
Monitoring Spatial/regional differences
Main source of information on distributional outcomes - household surveys - permit only limited disaggregation.Very large data sources (e.g. census) typically collect very limited information on welfare outcomes. Impute a preferred measure of welfare (e.g. comprehensive real consumption) from household survey into census, using statistical prediction methods
38
What are Poverty Maps?
39
Participatory Poverty monitoring (listening to the poor)
Limitations of statistical informationQualitative and participative approachesComplementary not alternative approaches
40
Data to monitor investment climate and empowerment
Firm and institutional surveysGood governance indicatorsEmpowerment indicatorsCommunity participation indicatorsData on quality of infrastructure Both quantitative and quality information required
41
Step 3 - Setting up a 5-year work program
National household budget survey with social indicators (LSMS), firm and institutional surveysAnnual monitoring survey (CWIQ)Participatory poverty assessments (PPA)Population and other censusConsumer and producer price collectionHousehold panel studyTimely national accountsCurrent public expenditure statistics
42
Illustrative S tructu re o f P overty and W elfare M onitoring S ystem
National statisticsoffice
Surveys, prices,natl.accounts
Line Ministries
Program & input monitoringAdmin. financial data
P.P.A Group
Qualitative studies
University
Panel studies
Welfare Monitoring Unit Welfare MonitoringAdvisory Group
National Poverty reduction committeeSr. Govt. Officials
Step 4-Defining Institutional functions
43
Step 5 - Defining Resource requirements
National budgetsCoordinated donor support through national budgetsStatistical capacity-building trust fund
44
Key messagesMonitoring poverty reduction goals goes beyond selection of indicators and measuring inputs and outputsA range of different monitoring tools are required Poverty reduction monitoring goals should include statistical capacity-building as a core elementMust be a national commitment to funding monitoring and statistical systemsMust ensure existence of appropriate incentives for good M&E?Feedback into budgeting and policy process