povertyactionlab.org

Poverty Action Lab: Translating Research into Action

MIT

povertyactionlab.org

The what, why and how of evaluations

Abhijit Vinayak Banerjee MIT and Poverty Action Lab

povertyactionlab.org

What is Evaluation

Start with a Real Example

Program • What was the program?

– Flipcharts donated to village primary schools • Where were the primary schools?

– Western Province of Kenya • When did this happen?

– 1997 • Who donated the flipcharts?

– International Child Support Africa (ICS Africa) (a local chapter of a Dutch NGO)

• Who, specifically, were they meant for? – For Students in 6th - 8th Grade

Why flipcharts?

Three sets of questions

Needs Assessment

Impact Evaluation

Process Evaluation

Needs Evaluation • Why are we doing this project: Laying out the theory • What’s the nature of the problem being solved?

– To compensate for a shortage of textbooks – To compensate for a lack of teaching and learning materials in

general – To help those who cannot read English very well – To help those who find it easier to learn from pictures – How will flipcharts solve it? – Why do we need to answer this question?

• Who is the targeted population? – All children? – ? – Why do we need to answer this question?

Needs Evaluation

• Why do we think it will work in this context? – Do teachers feel comfortable using them? – Does it fit with the curriculum?

• What else could we do?

– Adopt a different strategy for addressing the same need

– Do something entirely different with the money – Why do we need to answer this question?

Needs Evaluation Should Provide

• Clear sense of target population – Weaker students – Students with poor English – ?

• Clear sense of program design – What are teachers lacking in terms using these charts? – How to deliver? How much? What are potential barriers?

• Clear articulation of program benefits – Is a wrong being righted? Is a right being expanded?

• Clear sense of alternatives – Is this the most effective, efficient, cost-effective method of

meeting teacher/student needs?

Process Evaluation • Is the program actually doing what it says? • Are the services being delivered?

– Are flipcharts are reaching the classroom, are teachers are being trained to use them

• Are there ways of improving cost effectiveness? – Are the flipcharts being procured/delivered in the best

possible way? • Are the services reaching the right population?

– Are they being targeted to the classes with lower-performing students?

• Are the clients satisfied with service? – Teachers’, students’ response to new teaching method

Impact Evaluation

• What did the program achieve? • This question is only well-defined if we

have determined: • What measures do we expect impacts on? • What populations do we expect impact for? • When should we have answered

these questions?

Impact Evaluation

• Key question: Was there an impact? • Auxiliary questions:

– What was the nature of the benefit? • For example: Did flipcharts lead to uniform

improvements in test scores (for all students)? – Did some types of people benefit more

than others? • Students who were doing worse, poorer students,

etc.

Cost Benefit Analysis

Cost-Benefit Analysis

Process Evaluation

Impact Evaluation

Needs Assessment

Cost Benefit Analysis

• Why are all three elements useful for a cost benefit analysis?

• Impact evaluation gives benefits • Process evaluation allows careful

quantification of costs – May help to lower costs or increase efficiency

• Needs analysis frames opportunity cost of money

povertyactionlab.org

This Week • Focus on impact evaluation: Why? • It is what we know how to do well. • Surprisingly little is known about what works:

– Empowerment and Poverty Reduction: a Source-book – World Bank publication listing what works – Reports that the Gyandoot program that put computers in

villages in India was hard hit by lack of power. The computer kiosks lost money and many had shut.

– Then says, without any apparent irony, “Following the success of the program..”

povertyactionlab.org

Why impact evaluation?

• Central issues in policy design – Do we know that anything is working? – How do we identify what works? – Pick what really works

povertyactionlab.org

How much is the right answer worth?

povertyactionlab.org

Why Impact Evaluation?

• The devil, unfortunately, seems to be in the details.

• A great advantage of combining design with evaluation is that we can focus on the details

• This not only allows the possibility of refining them but also of redesigning them.

How to assess impact

What is beneficiary’s test score with program compared to without program?

Compare same individual with & without programs at same point in time

Formally, program impact is: α = (Y | P=1) - (Y | P=0)

e.g. How much does an education program improve test scores (learning)?

Solving the evaluation problem

Estimated impact is difference between treated observation and counterfactual.

Need to estimate counterfactual.

Never observe same individual with and without program at same point in time.

Counterfactual is key to impact evaluation.

We need a counterfactual

Counterfactual Criteria Treated & Counterfactual (1) Have identical characteristics (2) Except for benefitting from the intervention No other reason for differences in outcomes of treated and counterfactual

Only reason for the difference in outcomes is due to the intervention.

2 Counterfeit Counterfactuals

Those not enrolled

o Those who choose not to enroll

in the program o Those who were not offered

the program

Before and after

1. Before and After: Examples

School scholarship program on enrollment

Agricultural assistance program o Financial assistance to purchase inputs. o Compare rice yields before and after. o Before is normal rainfall, but after is drought. o Find fall in rice yield. o Did the program fail? o Could not separate (identify) effect of financial

assistance program from effect of rainfall.

Those not enrolled: Example of job training

Compare employment & earning of those who sign up to those who did not

Who signs up? Those who are most likely to benefit -i.e. those with more ability- would have higher earnings than non-participants without job training

Poor estimate of counterfactual

Those not enrolled: Example of those offered health insurance

With no insurance: Those that did not buy, have lower medical costs than that did

Compare health care utilization of those who bought insurance to those who did not o Who buys insurance?: those that expect large medical

expenditures o Who does not?: those who are healthy

Poor estimate of counterfactual

Program placement: example

Compare fertility in villages offered program to fertility in other villages

Government offers a family planning program to villages with high fertility

Program targeted based on fertility, so (1) Treatments have high fertility and (2) counterfactuals have low fertility.

Estimated program impact confounded with targeting criteria

What's wrong? Selection bias: People choose to participate for specific reasons

1

2

3

o Job Training: ability and earning o Health Insurance: health status and medical

expenditures

Many times reasons are related to the outcome of interest

Cannot separately identify impact of the program from these other factors/reasons

Need to know… All the reasons why someone gets the

program and others not.

All the reasons why individuals are in the treatment versus control group.

If reasons correlated w/ outcome cannot identify/separate program impact from other

explanations of differences in outcomes.

Possible Solutions Need to guarantee comparability of treatment and control groups.

ONLY remaining difference is intervention.

In this seminar we will consider: o Experimental design/randomization o Quasi-experiments (Regression Discontinuity,

Double differences) o Instrumental Variables.

These solutions all involve… Knowing how the data are generated. Randomization – Give all equal chance of being in control or

treatment groups – Guarantees that all factors/characteristics will

be on average equal btw groups – Only difference is the intervention If not, need transparent & observable criteria for who is offered program.

THANK YOU