View
8.442
Download
1
Tags:
Embed Size (px)
Citation preview
How we can use impact evaluation to assure effective use of resources
for development
Maximo Torero, [email protected]
DirectorMarkets, Trade and Institutions Division
(IFPRI)
IFAD-IFPRI Partnership, January 31st. 2012
Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation
approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments
Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation
approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments
Need for impact evaluation Helps identify and measure the results Helps identify the causal link between
intervention and results Provides a systematic and objective assessment
of program impacts Helps determine if interventions are relevant and
cost effective Promotes accountability, evidence-based
policymaking, and learning.
Need for impact evaluation Over past decade, increased demand from
governments, donor agencies and general public, for evidence of Impact of development policies.
Political tool: Brings accountability regarding the use of development money
Fiscal tool / budgetary tool: Allocate resources across different sectors or programs
Management tools: Understand how to better reach the objectives.
Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation
approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments
Monitoring and Impact Evaluation:Monitoring A tool that provides regular information on:
How a project is being implemented How a project is operating in the field How a project is progressing relative to targets What is the quality of service delivery (where applicable)
Rationale for Monitoring:
Provides basis for corrective action Holds implementers accountable for delivery of inputs Provides assessment of continued relevance Provides critical information for decision-making
Monitoring and Impact Evaluation:Evaluation Impact Evaluation:
Measures effectiveness and impact of programs or policies on outcomes of interest
Seeks to establish causality
Not all programs need to be evaluated; not all outcomes need to be measured in all evaluations
Page 9
Indicators for Monitoring and Evaluation
IMPACT
OUTPUTS
OUTCOMES
INPUTS
Effect on living standards - better welfare impacts (e.g literacy, health) - increase in participation, happiness
Financial and physical resources - track resources used in the intervention
-e.g. budget support for local service delivery
Goods and services generated- more local government services delivered- e.g., textbooks, food delivered, roads built
Access, usage and satisfaction of users- e.g. school attendance, vaccination rates, - food consumption, number of mobile phones
Eva
luat
ion
Mon
itorin
g
Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation
approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments
Guiding Principles of our IE Approach1. Generate information to influence decisions
2. Specify which indicators and methods are most suitable for each type of projects
3. Identify impact pathways
4. Evaluation activities must be built into the project design
4. Consider direct and indirect beneficiaries of projects
5. Evaluation at different levels of aggregation: Individual, thematic, and overall program
6. Incorporate complementarities and substitution among project impacts
Stage 1: Consultation
Stage 2: Development
IFAD’s:• Objectives• Activities• Information
needs to
Identify:• Indicators• Methods
• Monitor performance• Evaluate Effectiveness• Asses Impact
Target PEOPLE and vulnerable GROUPS:• Poor and Women
by Theme:•Technology• Productivity• Market Access• Nutrition
by level of Aggregation:• Individual projects• Theme• Agricultural Development
program
Stage 3: Feedback
• Governments, • IFAD• Implementers• CSO
Description of the Project
Impact Evaluation: Impact Pathway
The expected causal chain of events leading from project activities to outputs, to changes in the target population, and to the achievement of project objectives:
From INPUTS OUTCOMES IMPACTS
Focus on the impact pathway allows to: Understand how impacts are (or are not) achieved Allows generalizability of findings Provides key information for scaling up Identification of indicators for each step along the impact pathway
Scholarships for plant breeders & grants for agronomic research
number and quality of varieties released
availability and adoption ofimproved crop varieties
Higher yields for farmers whoadopted improved varieties
income, poverty among farmer households
IMPACT PATHWAY INDICATORS METHODS
Spending on scholarships& research grants
No. new varietiesApproved & released
% male, female farmersUsing improved varieties
Average yields amongadopting farmers
Income, expenditure,Well-being indicatorsamong target groups (poor, women, etc.)
Internal programmonitoringPR
OC
ESS
IMPA
CT
Intra-HH surveys:Before/After, Beneficiary/Control (Diff in Diff)
Illustrative Impact Pathway, Indicators, Methods Example from: Science & Technology
Applying the Methodology to specific types of interventions
Technology Example: Bio-fortification
Productivity Example: Grants to crop breeding programs
Market Access Example: Participation of small holders in the dairy value chain, “chilling
plant hubs”
Nutrition interventions Example: Evaluation of specific interventions to improve nutrition of the
most vulnerable
Bio-fortification
Adoption of bio-fortified varieties
Greater yields for farmers who adopted bio-fortified varieties
IMPACT PATHWAY INDICATORS METHODS
Spending on bio-fortification R&D
Average yields among adopting farmers
Income, expenditure,Well-being indicators
Internal programmonitoring
PRO
CES
SIM
PAC
T
Production of bio-fortified varieties
Consumption of bio-fortified varieties
Change in micronutrient status
Improvements in health, work performance, cognitive ability
income, poverty among farmer households
No. of farmers and land adopting bio-fortified varieties.
Total production of bio-fortified varieties
No. of individuals and average consumption (by type of individual))…
Reducing micronutrient malnutrition
Morbidity, mortality, enrollment ratio in primary
Bio-fortification Project (Science and Technology)Assumption: No price effect…
HH surveys•Beneficiary, control• Farmers, consumers•DD estimator•Randomization
•Panel: first round effect vs. second round effects
•Qualitative information: two-way calling with the poor
consumption of animal products, fruits, and vegetables
Creation of farmer groups as dairy farmer business associations (DFBA)
Chilling plant construction
IMPACT PATHWAY INDICATORS METHODS
Number of DFBA created and number of farmers participating (by gender)
Number of plants and milk capacity
Income, expenditure,Well-being indicators
Internal programMonitoring
Qualitative Assessment: organizational capacity
PRO
CES
SIM
PAC
T
Increase milk production of member farmers
Sales to formal markets and traditional markets
income, poverty among farmer members
milk production of farmer members
Volume of loss due to spoilage
Value of sales to formal processors and to traditional markets
Chilling Plant Hubs (Market Access)
HH surveys•Beneficiary, control•DD estimator•Non-experimental design
•Qualitative information: two-way calling with the poor
Reduction in loss through spoilage
M&E at Different Levels of Aggregation
What needs to be learned at the theme level?
Theme specific indicators:
Market access Productivity Science and tech. Data Analysis
Meta analysis within theme
Database at project level
within themes
What needs to be learned at the strategy level?
Cross theme indicators:
poverty
Meta analysis at the strategy
level
Evaluation strategy Indicators Methods
Project level
Themelevel
Programlevel
What needs to be learned at the project level?
Project Indicators:
Process indicators Outcome
indicators
Quantitative
Qualitative analysis
Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation
approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments
Impact Evaluation: Concepts
Impact evaluation hinges on determining what would have happened if the program had not existed.
Good practice involves a comparison of outcome before and after intervention with those with and without intervention
Problem is identifying valid counterfactual
Impact Evaluation: Methods
Quantitative Methods
Pre and post intervention, no control group Pre and post intervention, with control group, but
no randomization Pre and post intervention, with control group
and randomization
cost
reliability
Qualitative Methods - complementary – help:
Interpret of quantitative results Identify unexpected impacts, or effects on groups that are not captured
by quantitative surveys, etc.
Counterfactual
Ideally: Observe the outcome variable for those in the
program and For those same individuals had they NOT
participated in the program (the counterfactual)
So, constructing the counterfactual is the key issue that any empirical method must effectively handle.
Before the program
Beneficiaries:
Non-Beneficiaries:
A: “Treatment” Status
B: “Non Treatment” Status
C: “Treatment” Status
D: “Non Treatment” Status
E: Status beforethe program
F: Status beforethe program
Shaded boxes are Observable situations
Unshaded boxes are Unobservable
After the program
RealCounterfactual
EstimatedCounterfactual
Impact Evaluation: Finding a Counterfactual
Concept: How is the outcome different than it would have been if the project had not been implemented? = A – B (but cannot be observed)So estimated impact is based on double difference: (A-E) – (D-F)
Page 24
Supposed we observe an increase in outcome Y for beneficiaries over time after an intervention
Y0
Y1
baseline(t0) follow-up(t1)
Intervention
(observed)
Page 25
To measure impact, we need to remove the counterfactual from the observed outcome
Y0
Y1
baseline(t0) follow-up(t1)
Intervention
(observed)
Y1*
Impact=Y1-Y1
*
(counterfactual)
Comparison
Treatment Effects: key obstacles
Experimental vs. Non-Experimental Data Experimental data rules out self-selection into the
program (according to observables or unobservables) as a source of bias in measuring the treatment effect
So, this contribution of experimental data brings into high relief the two key obstacles that non-experimental data methods must overcome in order to avoid biased estimates of the average treatment effect:
Treatment Effects: Key obstacles (cont)
1. Self-selection into the program due to observables characteristics
2. Self-selection into the program due to unobservable characteristics
Accounting for #1 is often difficult (or impossible) to accomplish.
Even if #1 is accounted for in the method but # 2 is not, then bias in the result will inevitably occur.
Similarly if the control and treatment groups are randomly selected from a population then there is no bias in the initial characteristics
The impact of the procedure X can be attributed to the differences in the variable Y between the control and treatment group.
Population Random selection
Treatment Group (receives procedure
X)
Control group
(not receives procedure X)
Y Exp – Y Control
Although normally experimental methods are not applied ¿Why we can not then apply a direct comparison between the control
and treatment group? Because differences in characteristics of subjects, or what is called selection bias.
Population NO random selection
Quintile I (more poor)
Quintile II Quintile III Quintile IV QuintileV (more richer)
Because of initial differences between
both groups, the effects of the treatment can not be identified by directly comparing the groups
Treatment Group (receives procedure
X)
Control group
(not receives procedure X)
Selection bias: “Graphically”
SB = 0
G=ATT
SB > 0
G>ATT
SB < 0
G<ATT
Observed difference (G)Impact on the treated (ATT) = true effect of the program on its recipientsSelection Bias (SB)
No selection bias Selection on “better-off” with respect to the
outcome
Selection on “worse-off” with respect to the
outcome
Observed G
Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation
approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments
Overcoming selection bias
Ex-ante
Experimental approach: the design of the program allows to introduce randomness in its allocation
Ex-post
Natural experiment approaches: there are events that allow to simulate “exogeneity in the choice of treatment”
Control approaches: try to neutralize (reduce) as much as possible the selection bias
Experimental approaches
Randomly allocate “Treatment” into a population. Eliminates selection bias:
Sometimes ethical critics If the exclusion of some beneficiaries is only due to the evaluation, while benefits
are well known In reality, resource constraints are the limiting factor. Then, random selection can
be considered a fair process (every potential beneficiary has same chance of being selected)
Must be designed before the start of the program
Remains the best approach.
C Ci iE Y | = E Y | 0,T C SB G ATT
How to randomize?
Randomize program as a whole. E.g. oversubscription: when there are limited supply and excess demand select recipients by lotteries.
Randomize phasing-in Program cannot reach all intended beneficiaries the first year. select first year recipients randomly
Randomize encouragement. Cannot randomize treatment for ethical or practical reasons. Randomly allocate encouragement (e.g. vouchers).
Only increases the probability that a treatment is received without changing it from zero to one specific analytical challenges (partial (or imperfect) compliance).
Natural experiment approaches Use the fact that the program was allocated to some potential beneficiaries
and not to others, for reasons that have nothing to do with the outcome itself. Find variable that is strongly linked to participation (fully or partially)
but not to outcome.
Pipeline comparisons when administrative delays. Compare current participants to prospective participants who also qualify.
Regression discontinuity when program selection based on clear threshold on a given variable. Compare people just before threshold to people just above.
Instrumental variables Use predicted participation as given by a variable linked to participation
but not to outcome
Page 36
Limitations of These Methods of Impact Analysis
Impact evaluation focuses on program benefits, ignoring costs. Measures one side of cost effectiveness.
This limitation provides motivation for cost studies (Caldés, Coady and Maluccio, 2004)
Methods provide estimates of average impact in a ‘black box’ form. Good for demonstrating impact, but limited for broader policy analysis (Ravallion, 2005)
Controls approaches
Matching: compare people with similar ex-ante observable characteristics Control for the effects of observable characteristics that may affect hh
outcome. Assumption: All components of selection bias are observable and
measured (no omitted variables).
Difference in difference: compare the evolution of the hh with treatment to the evolution of the hh without treatment Neutralize time-invariant individual characteristics (observable and
unobservable). Neutralize effect due to other external events that may have affected
outcome since the program started. Assumption: absent the treatment, the outcomes in the two groups
would have followed parallel trends
Mixed: difference in difference on matched households
SummaryWelfare measure
Before… After…With WithoutWith Without
2
Problem with “before / after” measure
Difference could be driven by other events
Problem with “with / without” measure
Difference could be driven by selection
Double difference 1: “differences in evolution”Impact = (1) – (2). Controls for other events and self selection if the latent heterogeneity is additive and time invariant.
1
Double difference 2: “differences in evolution”Impact = (1’) – (2). Where initial differences are controlled for. E.g. matching and difference in difference
1’
If randomization or natural experiment approach, then original differences should not exist. In such cases, with/without measures can be sufficient
Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation
approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments
Different Levels of Aggregation: A Common Evaluation Measure
ERR, PRRR
ERR, PRRR
∑ theme level + complementarities –substitution (potential GE effects)
∑ project level + complementarities -substitution
Careful evaluation at this level is the foundation of higher-level evaluations
Economic Rate of Return ERR
Poverty Reduction Rate of Return PRRR
Project Level:- Program Logic Diagram- Impact evaluation- Cash flow analysis
Theme level
Program level
Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation
approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments
The concept of (stochastic) profit frontier toassure external validity This approach is based on a
simple economic concept: theProduction Possibility Frontier(PPF).
All the possible productioncombinations are found withinthe PPF.
Outside of the boundary are combinations which are notachieveable under currentconditions
The efficient use of resources isalong the boundary.
C
Milkproduction
Cornproduction
ProductionPossibilityFrontier
Land use
Roads
Water bodies
Altitude
Accessibility
Advantages of Micro-Region Typology
44
Typology
Diagnostic from Poverty map
High potential and low average efficiency
Low potential and low average efficiency
High poverty areas High poverty areas
What are the principal differences between high and low efficiency households in the area?
Productive projects differentiated to meet local needs and problems
Conditional Cash Transfers and Nutritional Programs
The inclusion of socioeconomic characteristics and access in the analysis allows for the identification of bottlenecks in areas of high potential but low or medium efficiency
Productive and Efficiency potential based on market, socioeconomic, bio-physical and access characteristics.
Advantages of a micro-region typology: classification
Micro-Regions Poverty Potential EfficiencyCritical, lacking agricultural potential High Low High-Medium-LowMedium priority, no agricultural opportunities Medium Low High-Medium-LowLow priority Low Low High-Medium-LowHigh priority High Medium-High High-Medium-LowMedium priority, with agricultural opportunities Medium Medium-High Medium-LowLow priority, with agricultural opportunities Low Medium-High Medium-lowHigh performance Low Medium-High High
Estimation Methodology
Potencial: Precios de productos (P) y insumos (W), beneficios reportados por el hogar (π).
Eficiencia: tierra, valor de los activos, características socioeconómicas (Z), condiciones biofísicas (G), acceso a mercado (A).
INSUMOS PARA LA ESTIMACION
Econometría Modelo de fronteras
estocásticas de beneficios
ESTIMACION OUTPUT DE LA ESTIMACIÓN
Pesos asignados a los insumos de acuerdo a teoría económica y evidencia empírica
PASO 2: PREDICCIÓN (NIVEL REGIONAL)
Resultado de la estimación (pesos)
Frontera: Precios de productos (P) y insumos (W).
Eficiencia: tierra, valor de activos características socioeconomicas (Z), condiciones biofísicas (G), acceso a mercado (A).
INSUMOS PARA LA PREDICCION
Potencial productive a nivel regional; eficiencia
de acuerdo a las características
socioeconómicas, condiciones biofísicas,
acceso a mercado dentro del área
PASO 1: ESTIMACION (NIVEL DEL HOGAR)
RESULTADO DE LA PREDICCION
Potencial productive y eficiencia a
nivel regional
Pesos
RESULTADO FINAL
PotentialPrices of products (P) and inputs (W), profits reported by household (π).
Efficiency Land, value of activities, socioeconomic characteristics (Z), biophysical conditions (G), market access (A).
Econometric Model of the
stochasticProfit frontier
Weights assigned to inputs following
economic theory and empirical evidence
Estimation results (weights)
Boundary Product prices (P) and inputs (W)
Efficiency: Land, value of activities, socioeconomic characteristics (Z), biophysical conditions (G), market access (A).
Productive potential at the regional level;
Efficiency according to socioeconomic characteristics, biophysical conditions, market access within the area
Region level productive
potential and efficiency
Estimation inputs Estimation Estimation output
Final result Prediction result Prediction inputs
Step 1:Estimation(Household Level)
Step 2:Prediction(Regional Level)
Recap (1)… Data
Available datasets:Land characteristics, biophysical conditions, socioeconomic characteristics, assets, market access, etc.
0.1
.2.3
.4D
ensi
ty
4 6 8 10 12values X
Group 1Group 2
Variable X
.7.8
.91
Cum
ulat
ive
Den
sity
0 2 4 6 8 10Values Z
Group 1Group 2
Variable Z
PPF:Input, Output, Profits
Geo Layers
Targeting Criteria based on Efficiency
Estimated cost of Market Access
Agricultural Profit Frontier
Efficiency in Agricultural Profits
Recap (2)…E
ffici
ency
Allo
catio
n C
riter
iaE
quity
Allo
catio
n C
riter
ion
Typology combines all these criteria
MULTIPLE TARGETING DIMENSIONS
49
Recap (3)…
Recall the initial objective….
High potential and low average efficiency
Low potential and low average efficiency
Micro-Regions
Critical, lacking agricultural potential
Medium priority, no agricultural opportunities
Low priority
High priority
Medium priority, with agricultural opportunities
Low priority, with agricultural opportunities
High performance
Recap (4): Grouping diverse criteria into
seven microregions…
Recap (5)… How does this translate into policies?
High potential and low average efficiency
What are the principal differences between high and low efficiency households in the area?
Productive projects differentiated to meet local needs and problems
Recap (6)… How does this translate into policies?
Low potential and low average efficiency
Conditional Cash Transfers and Nutritional Programs
Can be applied to other settings? Guatemala
Poverty Map Efficiency in Agricultural Profits
Cost of Market Access
Agricultural Profit Frontier
Guatemala: Seven-Class Typology
Without agricultural potential
With agricultural potential
Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation
approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments
Some examples Extension services
Market information
Infrastructure in rural areas
Property rights – land titling
Some examples Extension services
Market information
Infrastructure in rural areas
Property rights – land titling
Diff-in-Diff, FE (Dercon et al 2008) Impact of road quality improvements and increased access to
agricultural extension services on consumption and poverty in rural Ethiopia.
Dependent variables: household is poor, consumption growth Treatment: receiving at least one extension visit, and access to all-
weather roads (=1 if road to nearest town is all-weather road) Identification: IV model using GMM and controlling for household fixed
effects Instrument for consumption in time t-p: fertile land holdings, number
of adult equivalents and number of livestock units (all in logs) at time t-p.
Receiving at least one extension visit reduces headcount poverty by 10 percentage points and increases consumption growth by 7 percent. Access to all-weather roads reduces poverty by 6.9 percentage points and increases consumption growth by 16.3 percent.
Ex post Supply driven
Some examples Extension services
Market information
Infrastructure in rural areas
Property rights – land titling
Page 60
Institutional arrangement for a simple price information system
Source: Hernanini (2007), World Bank
Flow of information and Institutional agreements for virtual markets
Source: Hernanini (2007), World Bank
Page 63
Due partly to costly information, price dispersion across markets is common in developed and developing countries
Between 2001 and 2006, cell phone service was phased in throughout Niger, providing an alternative and cheaper search technology to grain traders and other market actors
The author constructs a novel theoretical model of sequential search, in which traders engage in optimal search for the maximum sales price, net transport costs
The model predicts that cell phones will increase traders’ reservation sales prices and the number of markets over which they search, leading to a reduction in price dispersion across markets.
To test the predictions of the theoretical model, they use a unique market and trader dataset from Niger that combines data on prices, transport costs, rainfall and grain production with cell phone access and trader behavior.
Pseudo randomized IV - Mobile phones: the impact of Cell phones on grain Markets in Nigeria (Jenny C. Aker - 2008)
Page 64
The results provide evidence that cell phones reduce grain price dispersion across markets by a minimum of 6.4 percent and reduce intra-annual price variation by 10 percent
Cell phones have a greater impact on price dispersion for market pairs that are farther away, and for those with lower road quality. This effect becomes larger as a higher percentage of markets have cell phone coverage.
They provide empirical evidence in support of specific mechanisms that partially explain the impact of cell phones on market performance.
The primary mechanism by which cell phones affect market-level outcomes appears to be a reduction in search costs, as grain traders operating in markets with cell phone coverage search over a greater number of markets and sell in more markets.
The results suggest that cell phones improved consumer and trader welfare in Niger, perhaps averting an even worse outcome during the 2005 food crisis.
Main results
Beginning in October 2000, it set up 1700 internet kiosks and 45 warehouses in Madhya Pradesh that provide wholesale price information and an alternative marketing channel to soybean farmers in the state
Dependent variables: wholesale price of soybeans, sales in traditional markets, soybean cultivation
Treatment: presence of internet kiosks and price warehouses
Identification: variation in timing of the introduction of kiosks and warehouses
Equivalent to randomization at the village level Ex post Demand driven
Pseudo randomized IV: Internet: Internet kiosks in India to provide wholesale price information (Aparajita Goyal, 2008)
Page 66
The estimates suggest an immediate and significant increase in the monthly wholesale market price of soybeans by 1-5 percent after the introduction of kiosks, lending support to the predictions of the theoretical model
While the presence of warehouses appears to have no effect on price, warehouses are associated with a dramatic reduction in the volume of sales in the traditional markets
Moreover, there is a significant increase in the area under soy cultivation. The estimates are robust to disaggregated measures of treatment and comparisons with alternative crops grown in the same season as soy
The results suggest that information can enhance the functioning of rural markets by making buyers more competitive.
Main results
Some examples Extension services
Market information
Infrastructure in rural areas
Property rights – land titling
Pipeline comparisons when administrative delays (Torero 2008)
The road to be improved was split in the following segments:
A B C D ETable 1. Timeline
Section
Scheduled Start Date
Scheduled End Date
A July 2008 June 2010
B Completed Completed
C July 2008 December 2009
D October 2008 April 2010
E October 2008 February 2010
Based on the geographic location and the timelines, the following treatment-control groups are suggested:
Test Number
Control Group
Treatment Group
1 B A
2 B C
3 D C
4 E D
Table 2. Treatment and Control Groups
Pipeline comparisons when administrative delays
Pipeline comparisons when administrative delays.
Randomized- Barriers to connection in Ethiopia (Bernard and Torero 2009)
Connection fees range between USD 50 and USD 150 (drop down line and meter). Need to find ways to facilitate connection for the poorer.
Can CFL (energy-saving light bulb) positively influence energy use? How to promote the use of energy-saving light bulbs (consumes 4 times less, but costs 8 times more)?
What is best: 2 years loan or 5 years loan for connection fee?
Pilot study on 20 towns to assess optimal subsidies. Experimental approach (randomize encouragement through distribution of vouchers).
This image cannot currently be displayed.
Public distribution
Public distribution
Random selection…
Some examples Extension services
Market information
Infrastructure in rural areas
Property rights – land titling
Matching, IV on cross-sectional data - Land property rights on productivity (- Markussen 2008)
Dependent Variable: (log) value of output per hectare
Treatment: The plot is held with a paper documenting ownership (titles, application receipts)
Identification: IV mode of plot acquisition (dummies to indicate if the plot was given by the State, inherited, bought, donated, occupied for free) as instrument for the dummy “plot held with paper”
Ex-post Demand driven: households and landholders
apply for titles
Pseudo-Randomized - Land titling on rural households (Torero and Field 2007)
Dependent variables: household expenditure, change in rent/market value of dwelling, risk of expropriation, production, trade of land, collateral and credit markets, land ownership and tenancy, permanent ant transitory crops
Treatment: to receive land title Identification: quasi random program
implementation, kernel matching Ex post Demand driven, but few requirements and
virtually free
The survey covered 3204Peruvian rural households:521 from rural coast, 1622from rural highlands and1061 form rural jungle.
The next map plots thetowns covered by the surveyand the valleys reached bythe PETT program. Fromthese 3204 households 1793match with at least oneprevious national survey.
The Database
If control and treatment groups are randomly selected from a population then there is no bias in the initial characteristics
The impact over income can be attributed to the access to title
Population Random selection
Treatment Group (receives procedure
X)
Control group
(not receives procedure X)
Y Exp – Y Control
¿Why we can not then apply a direct comparison between the control and treatment group? Because differences in characteristics of subjects, or what is called selection bias.
Population Pseudo random
selection
Quintile I (more poor)
Quintile II Quintile III Quintile IV QuintileV (more richer)
Because of initial differences between
both groups, the effects of the treatment can not be identified by directly comparing the groups
Treatment Group (receives procedure
X)
Control group
(not receives procedure X)
Identify comparable pairs (with similar initial characteristics) and that differ only on the procedure
We will use Propensity score matching.
Population pseudo random
selection
Find the pair to assure comparability Treatment
Control
Impact of the
procedure
Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation
approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments
Final comments The impact evaluation must be a part of the program
design It is very important to identify how to incorporate it
now that the program already exists For new programs it is necessary to invest in the
design so that an impact evaluation is also part of it
It is essential to identify the impact pathways, i.e. the expected causal chain of events leading from project activities to outputs, to changes in the target population, and to the achievement of project objectives
Since the beginning of the program is necessary to specify the expected “outcomes” and the control group
Final comments It will be ideal to have an autonomous and external
laboratory of impact evaluation
Communications among all stakeholders is central
Not all interventions need to be evaluated, it will be ideal to do it before scaling up so there is assurance that the intervention works
Alignment of proper incentives – to contractors, evaluators, to implementers and to USAID country offices
Finally, policy requires a causal model; “without it, we cannot understand the welfare consequences of a policy” (Deaton 2009)
Page 85
Recommended readingsCaldés, Natalia, David Coady and John Maluccio. 2004. The Cost of Poverty
Alleviation Transfer Programs: A Comparative Analysis of Three Programs in Latin America. IFPRI FCND Discussion Paper No. 174., Washington, DC.
Duflo, Esther; Rachel Glennerster and Michael Kremer (2007):“Using Randomization in Development Economics Research: a Toolkit”CEPRdiscussion paper no. 6059
Feder et al. 2004. Review of Agricultural Economics.
Godtland et al. 2004. Economic Development and Cultural Change.
Heckman, J.J., H. Ichimura, and P.E. Todd. 1997. “Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Program.” Review of Economic Studies 64:605-654.
Hirano and Imbens. 2004. The Propensity Score with Continuous Treatments. In Gelman & Meng, eds.
Miguel and Kremer. 2004. Econometrica.
Ravallion, Martin. 2005. Evaluating Anti-poverty Programs. World Bank Working Paper Series 3625, Washington, DC. Martin Ravallion (2003): “The Mystery of the Vanishing Benefits: An introduction to Impact Evaluation” The World Bank Economic Review, volume 15, no 1, pp115-140