#ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 ARIANNA LEGOVINI DEVELOPMENT IMPACT EVALUATION IMPACT EVALUATION FOR REAL TIME
#ieGovern Impact Evaluation Workshop Istanbul, Turkey January
27-30, 2015 ARIANNA LEGOVINI DEVELOPMENT IMPACT EVALUATION IMPACT
EVALUATION FOR REAL TIME POLICY MAKING
Slide 2
WHAT IS THE VALUE OF A POLICY? measuring HOW TO GET THE MOST
OUT OF A POLICY? reaching 2
Slide 3
Policy objective Invest in what gets us results 3 Invest in the
cause to get an effect
Slide 4
Evaluation objective Identify cause - effect 4 enable good
policy decisions
Slide 5
55 WHY IS UNDERSTANDING CAUSALITY SO IMPORTANT? I THINK I WILL
STOP USING UMBRELLAS WHEN I USE MY UMBRELLA MY SHOES GET WET It is
easy to confuse correlation with causation
Slide 6
6 Invest in the cause to get results Mentorship interacts with
quality of mentor to change attitudes and value for youth to act
upon their life-skill training and change their behavior (Bushway
and Reuter, 2002, 2007) Charismatic mentor + behavioral therapy
Attitudes and values Vocational training EMPLOYMENT More motivated
youth
Slide 7
Evaluation Narrow down causality by identifying a
counterfactual and compare 7 WHAT HAPPENED WHAT WOULD HAVE HAPPENED
TO
Slide 8
8 What is counterfactual analysis? Compare same individual with
& without intervention at the same point in time Missing data
Compare statistically identical groups of individuals with &
without intervention at the same point in time Comparable data
Slide 9
9 Counterfactual criteria Treated & control groups Have
identical initial average characteristics (observed and unobserved)
So that, the only difference is the treatment Therefore the only
reason for the difference in outcomes is due to the treatment
Slide 10
10 USE SCIENTIFIC EXPERIMENTS TO GUIDE POLICY DECISIONS
Slide 11
1111 Get the delivery right Subsidy for malaria treatment $350
million Sealing packages saved 16,600 under five + 2,200 adult
lives in 5 years Problem: poor public sector distribution of
medicines Solution: found system that reduced stockouts by 83%
Source: Vledder, M.; Friedman, J; Yadav, P., and Sjoblom M. RANDOM
ASSIGNMENT OF HEALTH DISTRICTS TO 3 DIFFERENT DISTRIBUTIO N
SYSTEMS
Slide 12
12 Price a CURE $6
Slide 13
1313 Get people to do it Source: Goldstein, Thirumurthy, and
Zivin HIV treatment $ 2.1 billion Reminders increase adherence to
treatment by 35% Problem IE scalable solution RANDOM ASSIGNMENT OF
DAILY AND WEEKLY REMINDERS TO HIV PATIENTS
Slide 14
14 Price Faiths LIFE $7
Slide 15
15 TRANSFORMATIONAL (not marginal) IMPACT
Slide 16
16 Public recognition DOUBLES performance [Ashraf et al.,
2011]
Slide 17
What does dime do? 17 Transform development policy 1.Run
experiments to inform decisions 2.Build agencies capacity to do
this systematically 3.Draw lessons and share them face-to-face to
global audiences
Slide 18
175 IEs in 47 Countries Distribution of dime s IEs by GP and
CCSA (number of IEs) 18
Slide 19
1919 dime in action Inform policy design Guide mid- course
corrections Inform adoption and scale-up Systematic use of evidence
Train & apply Learn by doing Apply knowledge Capacity building
IE Product IE DESIGN IE IMPLEMENTATION IE DISSEMINATION
2121 UNDERSTAND THE THEORY OF CHANGE Inputs Financial and human
resources Activities Actions to convert inputs into outputs Outputs
Tangible products Intermediate outcomes Effective access Use
Compliance Performance Service quality Final Outcomes Health
outcomes Lives saved Productivity WHAT (PROGRAM)HOW TO (RESPONSE)
& RESULTS
Slide 22
awareness disposition Take-up ADOPTION SKILLS WILLINGNESS
motivation participation Accountability quality sustainability 22
BREAK IT DOWN
Slide 23
Policy questions in dime s portfolio (% of questions) 23
Slide 24
Better capacity 24 Better capacity Research team Full time
field coordinator Training More and better data
Impact of Workshop on (External Double Blind) Technical Ratings
of IE proposals 27
Slide 28
dime results indicators 28
Slide 29
More effort into implementation 29 Motivation to perform Goal:
this is what we are trying to achieve Control: this is how we want
to do it and BTW somebody measures (and cares)
Slide 30
Better decisions 30
Slide 31
dime results indicators 31
Slide 32
IE increases disbursements IE helps projects stick to plans 32
Does IE help project implementation? 1300 World Bank projects - 100
impact evaluations (2005-2011) Legovini, Di Maro, Piza (2015) It
does!
Slide 33
IE increases disbursements & closes implementation gap
statistically significant at 1% level 33
Slide 34
AT WHAT COST? SOME SIMPLE COST-BENEFIT OF IMPACT EVALUATION
34
Slide 35
Is the benefit worth the cost? Zambia best supply chain IE
solution: 83% reduction in stock out 21-25% reduction in malaria
mortality Pilot: Benefit ($7.4M) / Cost ($3.8M) = 2 Scaled up:
Benefit ($60.7M) / Cost ($12.4M) = 4.9 Malawi commitment saving
account IE solution: 22% increase in farmer production value Pilot:
Benefit ($0.7M)/Cost ($0.5M) = 1.4 If scaled: Benefit ($15.6M) /
Cost ($1.7M) = 9.2 35 IE PAYS FOR ITSELF A PUBLIC GOOD WORTH
FINANCING
Slide 36
dime & i2i IE has real impacts on developmental
effectiveness and project delivery High return on investment
Spillover effects i2i opportunity to increase scale and scope Open
invitation to donors and governments 36 partners CRIME LAB
Slide 37
Human Development Education Health Social Protection i2i
pillars and thematic areas 37 Shared Prosperity Finance, Private
Sector and Jobs Agriculture Infrastructure Governance Public Sector
Governance Justice Local Development Governance Public Sector
Governance Justice Local Development Climate Change Energy
Environment Natural Resource Management Agriculture Transport
Fragility & Conflict Reintegration Governance Gender-based
Violence Urban crime & violence Gender Human Capital Economic
Opportunities Voice/Agency 5 123 64
Slide 38
Discussion Which development interventions are effective and
worth scaling up? Which promising mechanisms are worth testing
across different sectors? No one size fits all Importance of
building evidence-based institutions 38
Slide 39
#ieGovern Impact Evaluation Workshop Istanbul, Turkey January
27-30, 2015 Thank You! facebook.com/ieKnow #impacteval
blogs.worldbank.org/impactevaluations
microdata.worldbank.org/index.php/catalog/ impact_evaluation
http://dime.worldbank.org WEB