M&E and evaluation for development

in peace precarious situations

Catherine Elkins, PhD, MALD Belling the Cat LLC–Duke University EES Workshop 01 October 2012

Assisting development: Is our territory cognita?

2

Our day’s agenda

•  Sessions and schedule in handouts – Strategic contingency approaches – Case studies over time – Six-dimensional analytical model – Lessons for peace precarious situations and lessons

from peace precarious situations

What is ‘normal’ development?

•  “Think–pair–share” •  Define

– elements of typical development context –  typical development goals –  typical development stakeholders

–  typical development implementers / partners

Peace Precarious Situations

photo © Joe Holmes 5

What is ‘peace precarious’ development?

•  Differences we experience in

– context – goals – stakeholders

–  implementers / partners

Peace precarious situations

7

•  May overlap circumstances labeled in other discussions as fragile, low-level violence, conflict, post-conflict, failed states

Peace precarious situations

8

•  Power disputes •  recent, ongoing, or chronic •  violent at least sometimes

•  Destabilized but not obliterated core institutions

•  No directional assumptions •  No ‘state’ assumptions

Introductions: Catherine Elkins

•  http://linkedin.com/in/seaelkins •  Professor, Duke University; Consultant •  About 20 years’ experience working in

developing countries •  Afghanistan, Cambodia, Eritrea, Guinea,

Guyana, Iraq, Kazakhstan, Lebanon, Mauritania, Pakistan, Rwanda, South Africa, Uganda, West Bank, Zimbabwe

•  Belize, Jordan, Senegal, Tanzania, Thailand

Introductions: Colleagues

•  Name, main occupation •  Evaluation (or M&E) experience •  Development experience •  Peace-precarious experience

Introductions: Cases 1, 2

11

Introductions: Case 3

Development in peace precarious situations

USAID 2006, Provincial Reconstruction Teams in Afghanistan: an Interagency Assessment (PN-ADG-252) . 13

•  Extreme cases: International development efforts ongoing despite widespread internal or regional areas characterized by a “mid-range of violence”

•  “Instability still precludes heavy NGO involvement, but […] is not so acute that combat operations predominate”

Critical dimensions

14

Key aspects for M&E/Evaluation •  Volatility •  Internal and external ‘constants’ are

potentially/probably variable •  And especially: Multiple parameters can

change at once

Peace precarious situations

Activity

•  Afghanistan – external evaluation – What do we know about the 6 dimensions? – What else do we need to know?

–  Ideas for evaluation plan •  In principle •  20 questions

Focus on stakeholders

17

Stakeholder complexity & sensitivities

Sources for volatility •  Who has a stake in project failure?

•  Who has mixed interests, hedged bets?

•  What additional political or other concerns do those supporting success need to consider?

18

Lunch

19

Theories of change

•  Hypothesized relationships among influential factors shaping conditions or behavior of interest

•  Probabilistic causal expectations •  Observable, measurable, falsifiable model •  Mechanisms expected to produce results

20

Implementation – or so we like to think

21

Theories of change and peace precarious development

•  “Normal” development interventions – Complicated – Complex

•  “Peace precarious” interventions – Complicated, complex, and volatile – Relationships between good program design &

good M&E/evaluation design in peace-precarious situations

22

Peace precarious stakeholders and planning

•  From initial exercise •  From Afghan case exercise •  From own experiences

•  Relationship dynamics and contingencies

Activity

•  Guinea governance M&E – Dynamics over time

–  http://www.reuters.com/article/2010/06/27/us-guinea-events-idUSTRE65Q0ZP20100627

Project spr’08

Army mutiny sum‘08

Coup Dec’08

Massacre Sep’09

Coup attempt Dec’09

2 intra-coup

transitions Jan-Feb’10

Pres election June’10

Run-off results final

Nov’10

Legislative elections fall ’11** (not yet!)

Activity (cont’d)

•  Guinea –  Initial design – Big disruption – Ongoing disruptions

M&E strategy

•  Guinea governance M&E – Model and measurement

– Elasticities

Afghanistan and Guinea

•  M&E and evaluation dynamics – Same or different challenges?

– Documentation

Model (theory of change) and measurement

28

‘Normal’ vs peace precarious challenges •  If–then hypotheses of predicted

relationships – Falsifiable using empirical evidence – Discriminating paths among multiple influences – Quasi-experimental counterfactual; may be very

loose “other things being equal”

•  Baseline, implementation, results

Fidelity and slippage

•  Contextual constants and variables •  Cost and implementation elasticities •  Articulation of expectations, execution, M&E,

and evaluation including changes from design through close-out

•  Learning systems

29

Egypt, girls’ education, and “revolution”?

globalgiving.org

31

Punctuated equilibria •  Full or partial disruptions, interruptions,

unpredictable timing •  Expansions and curtailments of components,

flexible scope •  Nature and timing of specific shocks or factor

changes (context, institutions, stakeholder/counterpart composition)

Recap: Exploring the cases

Exploring the cases – coping under stress

32

Cost factors •  Fluctuation in budget •  Fluctuation in security expenses •  Staffing through interruptions, evacuations •  Recruiting, screening, incentives, and turnover

Measurement risk factors

33

Data quality and precision •  Human capital and stress effects on timeliness,

completeness, accuracy •  Perceived risk and risk attitudes

–  Risks to participants, beneficiaries, enumerators

–  Transparency and accountability risks •  Existence, accessibility, reliability of secondary

data sources

Information uses

Lower confidence in data and…

•  Potential for political or other unintended uses of project results and evaluation findings?

•  Likelihood that certain actors and audiences will misunderstand or misuse findings?

•  Potential damage to any stakeholders from use or misuse of results or findings?

34

Information and stakeholders in context

•  International context and global dynamics – Goals, uses, audiences, information/security

dimensions affecting development results information

– Six dimensions across our cases

– Appropriate, useful M&E/evaluation? •  Possible? •  Likely? •  Under what conditions?

35

Cross-cutting conclusions and challenges

36

Gaps analysis and theory building

•  What are we missing?

•  What else can we bring to bear?

37

Strategy

38

Where program and evaluation concerns meet •  Design evaluation around challenges to project

and ways challenges are most likely to affect evaluation

•  Identify specific decision points – More than one theory of change? – Client understanding or learning curve? – Variance across stakeholder interests? – Data needs for methodology options? – Budget requirements for strategic options?

Design for development

39 handout

Useful M&E/evaluation (peace precarious)

•  Questions: – Adequacy of development models for (some)

peace-precarious situations – Adequacy of M&E/evaluation approaches for

(some) peace-precarious situations – Managing expectations and validating deliverables – Ethics and other issues

40

Final Questions?

41

Thanks!