Improving evidence on the impact of agricultural research & extension

Reflections on CSISA’s experience

David J. Spielman, Patrick Ward, and Simrin Makhija

International Food Policy Research Institute

Africa RISING Monitoring and Evaluation Meeting

Arusha, Tanzania, 13-14 November 2014

A quick outline

• Constraints to adoption

• Uses and utility of evaluation

• CSISA’s experience

Why do farmers adopt new technologies?Expected utility maximization

Individuals maximize the expected “utility” they get from consuming food, non-food goods/services, and leisure with products and income derived from agricultural production

Individuals compare utility that would be achieved under different states of the world and the probability of that state of the world actually occurring

If individuals expect that adopting a new technology will maximize their total utility, then they tend to adopt…

Subject to several important constraints

Known, readily measured constraints

• Biophysical characteristics

• Land, soil, water, biology

• Farm characteristics

• Farm size, scale, system

• Individual characteristics

• Experience (age), education, hh labor supply

• Market factors

• Access to credit, input, and output markets

• Institutional factors

• land tenure arrangement, group membership

• Returns to adoption

• Net returns, variability in returns

What constrains adoption?

𝑍𝑖∗ =

1 𝑖𝑓 𝑈𝑖 > 𝑈𝑛𝑟

0 𝑖𝑓 𝑈𝑖 ≤ 𝑈𝑛𝑟

What else constrains adoption?

Lesser-known, harder-to-measure, and unobserved

constraints

Informational constraints

Experimentation, trialing (Linder et al. 1982; Marra et al. 2003)

Exposure, awareness (Diagne & Demont 2007)

Learning by doing (Foster & Rosenzweig 1995)

learning from others and peer effects (Besley & Case 1994; Conley & Udry

2010)

Learning by noticing (Hanna & Mullainathan 2012)

Individual preferences

Risk preferences (Just & Zilberman 1983; Smale et al. 1994)

Loss aversion (Tanaka et al. 2010; Liu 2013)

Time-inconsistent preferences and present bias (Duflo et al. 2008)

Aspirations (Bernard et al. 2014)

So… what questions should we be asking?1. How do different approaches to adult

education affect learning outcomes?

2. How do learning outcomes lead to

adoption?

3. How does adoption result in desirable

social, economic, and environmental

outcomes?

How do evaluations answer these questions? • Measurement: to gauge impact of alternative extension

approaches on• Adoption

• Productivity

• Sustainability

• Incomes and welfare

• Learning: to assemble evidence about what works and why

• Feedback: to improve project management and implementation

• Accountability

• Transparency

• Policy design: to credibly inform large-scale replications by government

Why do we need better designs, more rigor?• Sample selection bias

• Those who learn/adopt may be fundamentally different from those who don’t

• Bias limits our ability to make wider inferences, to scale up, and to design policies

• Endogeneity• Reverse Causation: A → B or B → A ?

• Simultaneity: the “Reflection Problem”

• Heterogeneity• Beyond average effects: Measuring outcomes for specific groups

within a population

With a better toolkit, we can do a lot more…

• Qualitative

• Quantitative

• Mixed Methods

• Understanding context

• Understanding impact pathways and theories of change

• Internal validity: good identification strategies

1. Experimental Methods: RCTs

2. Non-experimental methods: PSM, RDD, Ivs, D-in-D

• External validity: generalizability

…to explore complex processes

• Combine economics, education, and social psychology behavioral dimensions of learning and adoption

• Evaluate type and intensity of training

• Study the step-by-step process of learning

• Evaluate changes in aspirations and locus of control

• Evaluate learning failures

• Evaluate peer effects

But simplicity is key to understanding complex innovation processes

Take a single technology, practice, or system…

1. Evaluate which extension approach better facilitates learning/adoption

2. Compare different extension approaches• Training & Visit vs. Farmer Field Schools vs. Mother-Baby Trials vs.

Chalk-and-Talk

3. Measure the cost-effectiveness of each extension approach

4. Open the door to evaluation of learning approaches

Monitoring & evaluation in CSISA

Project Monitoring

Project outputs, deliverables

FTF indicators

External project reviews

Feedback to management

Project evaluation

Station, on-farm

agronomic trials

Costs and returns to farmers

Costs and returns to

service providers

Determinants of adoption

Impact evaluation

Productivity, sustainability

impacts

Income, welfare impacts

(Nutrition, health impacts)

Policy analysis

Ag subsidy policy

Industrial policy

National R&D priorities

(Ex ante scenario analysis)

What role for baseline surveys in CSISA? • As a diagnostic tool

• Characterizing farming systems, households, development domains• Measuring farm-level productivity and poverty levels• Characterizing variation in these indicators

• As an analytical tool• As data for modeling ex ante impacts at scale

• As an evaluation tool?• Too many simultaneous interventions to establish causality• “Saturation approach” to project rollout leads to contamination of

treatments• Extremely costly and time-consuming (3 yrs just to get a report out!)

The solution? We abandoned baseline surveys, focused on individual interventions

Our emerging approach

Diagnostics

(Consultations, FGDs, lit

reviews)

Valuations

(DCEs, BDMs)

Evaluations

(RCTs and quasi-

experimental designs)

Scale-up Scenarios

(Simulations models)

Tangents

(Deeper analysis of

second-order topics)

Our approach in action: LLLs in India

Is there demand for laser land leveling?

What will farmers pay for leveling?

What are the net gains to

leveling?

How do alternative targeting strategies

impact welfare?

How to gendered peer

effects influence

valuation and adoption?

Diagnostic Valuation Evaluation Scenario analysis

Tangents

Demand for LLL services20

040

060

080

0

Rs./h

ou

r

0 10 20 30 40 50 60 70 80

Percent of area

2012 2011

Source: Lybbert et al.

2014

Demand for LLL services20

040

060

080

0

Rs./ho

ur

0 10 20 30 40 50 60 70 80Percent of area

Deoria

Gorakhpur

Maharajganj

District

20

040

060

080

0

Rs./ho

ur

0 10 20 30 40 50 60 70 80Percent of area

BPL

Non-BPL

BPL

20

040

060

080

0

Rs./ho

ur

0 10 20 30 40 50 60 70 80Percent of area

<1 acres

1-2.5 acres

>2.5 acres

Landholdings

20

040

060

080

0

Rs./ho

ur

0 10 20 30 40 50 60 70 80Percent of area

First (<= .3 acres)

Second (.3-1 acre)

Third (1 or more acres)

Plot size

Source: Lybbert et al.

2014

Our approach in action: DT/WII in Bangladesh

Is there demand for

weather index

insurance vs. a DT cultivar?

What will farmers pay for

insurance?

Does behavior change

with insurance or a DT

rice cultivar?

How cost-effective is

a large-scale

rollout of DT vs. WII?

How do farmers learn about risk

management from infrequent

stochastic events?

Diagnostic Valuation Evaluation Scenario analysis

Tangents

Thank you

ReferencesBernard, T., S. Dercon, K. Orkin, & A.S. Taffesse. 2014. “The Future in Mind: Aspirations and Forward-Looking Behaviour in Rural Ethiopia.” Centre for the Study of African Economies Working Paper Series no. 2014-16. Oxford, UK: University of Oxford.

Davis, K., Nkonya, E., Kato, E., Mekonnen, D. A., Odendo, M., Miiro, R., & Nkuba, J. 2012. “Impact of Farmer Field Schools on Agricultural Productivity and Poverty in East Africa.” World Development 40(2): 402-413.

Feder, G., R.E. Just, & D. Zilberman. 1985. “Adoption of Agricultural Innovations in Developing Countries: A Survey.” Economic Development and Cultural Change 33(2): 255-298.

Hanna, R., S. Mullainathan, & J. Schwartzstein. 2012. “Learning through Noticing: Theory and Experimental Evidence in Farming.” Working Paper no. 18401. Cambridge, MA: National Bureau of Economic Research.

Haug, R. 1999. “Some Leading Issues in International Agricultural Extension, a Literature Review.” Journal of Agricultural Education and Extension 5(4): 263-274.

IPPRI (International Food Policy Research Institute). 2011. Policies, Institutions, and Markets to Strengthen Food Security and Incomes for the Rural Poor. A revised proposal submitted to the CGIAR Consortium Board. Washington, DC: IFPRI.

Jack, B.K. 2011. “Market Inefficiencies and the Adoption of Agricultural Technologies in Developing Countries.” White paper prepared for the Agricultural Technology Adoption Initiative: Cambridge, MA/Berkeley, CA: Abdul Latif JameelPoverty Action Lab (MIT)/Center for Effective Global Action.

Kondylis, F., & Mueller, V. 2010. “Seeing is Believing? Evidence from a Demonstration Plot Experiment in Mozambique.” Mozambique Strategy Support Program working paper no. 1. Washington, DC: IFPRI.

Lambrecht, I., Vanlauwe, B., Merckx, R., & Maertens, M. 2014. “Understanding the Process of Agricultural Technology Adoption: Mineral Fertilizer in Eastern DR Congo.” World Development 59: 132-146.

Lybbert, T.J., N. Magann, D.J. Spielman, A. Bhargava, and K. Gulati. 2014. Targeting technology to increase smallholder profits and conserve resources: Experimental provision of laser land leveling services to Indian farmers . mimeo.

Manski, C.F. 1993. “Identification of Endogenous Social Effects: The Reflection Problem.” Review of Economic Studies