EVALUATION FOR ACCOUNTABILITY: Measuring Development Effectiveness of NGOs

Keynote Address ECSP Conference ‘Challenges for Philanthropy: Scale, Governance & Impact’

Rotterdam, 22 September 2011

Prof. dr. Ruerd RubenDirector Policy & Operations Evaluation (IOB)Ministry of Foreign Affairs, The Netherlands

Policy & Operations Evaluation Department (IOB)

Policy & Operations Evaluation (1)

Policy & Operations Evaluation Department (IOB)

Mandate:

• Assessing policy effectiveness

• Promoting learning from experiences

• Improving evidence-based policy-making

Accountability to Parliament, Public and Partners

(autonomous programming & independent reporting)

Policy & Operations Evaluation (2)

Policy & Operations Evaluation Department (IOB)

Activities

• Impact studies on main development priorities

• Analysis of the coherence in foreign policy

• Thematic studies on key success factors

Creating Transparency & Trust

Key Messages

Policy & Operations Evaluation Department (IOB)

1. Development effectiveness has to show ‘value for money’

2. Showing effectiveness may enhance public trust (necessary, but not sufficient condition)

3. Transparency starts with autonomy & objectivity

4. Evaluability is key responsibility of donors and recipients

5. No learning without insights in quantitative impact

Development Effectiveness

Policy & Operations Evaluation Department (IOB)

• Delivery of meaningful results

• Net impact over time

• Value for Money

• Attribution/Contribution

• Additionality

Evaluating Civil Society

Policy & Operations Evaluation Department (IOB)

Large share of Dutch aid:

5% 20 % of ODA

(+ 10% ODA through

Delegated budgets)

Three components: Direct Poverty Reduction (MDGs)

Capacity Development (5C)Lobby & Advocacy

0

5

10

15

20

25

From Input to Impact

Outreach = # of participants

Throughput = activities within the system

Policy & Operations Evaluation Department (IOB)

Common Pitfalls

Policy & Operations Evaluation Department (IOB)

1. Strong selection bias

2. Spatial overlaps: externalities & free riding

3. Absence of baselines data

4. Scarce controls

5. Long result chains

6. Limited additionality

Credible evidence

Policy & Operations Evaluation Department (IOB)

Impact measurement: comparing:

• before/after (time)

• with/without (counterfactual)

Everything else is additional

(e.g. story harvesting, outcome mapping, SROI, etc)

(See: NONIE Guidelines ; 3ie)

Time

Goal

without

with

T=0 T=1

Difference in Difference (DiD)

Policy & Operations Evaluation Department (IOB)

Intervention group

Control group Difference across groups

Baseline I0 Co I0 – C0

Follow-up I1 C1 I1 – C1

Difference across time

I1 – I0 C1 – C0 (I1 – C1) – (I0 – C0) = (I1 – I0) – (C1 – C0)

Propensity Score Matching (PSM)

Policy & Operations Evaluation Department (IOB)0

.51

1.5

22.

5D

ensi

ty

0 .2 .4 .6 .8 1Propensity Score

ControlTreated

Before Matching

0.5

11.

52

Den

sity

0 .2 .4 .6 .8Propensity Score

ControlTreated

After Matching

Evaluation Criteria

Policy & Operations Evaluation Department (IOB)

1. Effectiveness

2. Efficiency

3. Relevance

4. Coherence

5. Sustainability

See: OECD-DAC Evaluation Guidelines

Methods & Approaches

Policy & Operations Evaluation Department (IOB)

Document review

Policy reconstruction

Robust Impact studies (DiD)

Experiments (Real-time evaluation)

Systematic Reviews (Campbell protocol)

Thematic studies (Key success factors)

Worrisome Evidence

Policy & Operations Evaluation Department (IOB)

‘NGO support is highly appreciated, but no evidence can be generated to demonstrate its impact’

‘Local capacities of NGO partners have grown substantially, but it remains unclear whether and how donor support has contributed to this’

‘Access to services and markets for the poor have improved, mainly due to better socio-economic performance of the country’

Declining trust

Policy & Operations Evaluation Department (IOB)

Reputation & Trust

Policy & Operations Evaluation Department (IOB)

“Trust me”

Tell me” “Show me”

“Show me”

“Trust me”

“Tell me”

Insight in development effectiveness is a necessary (but not sufficient) condition for maintaining public support

Policy & Operations Evaluation Department (IOB)

Evaluation & Accountability

Guaranteeing Evaluability

• Intervention theory

• SMART indicators

• Random sampling

• Valid baselines

• Comparison/control groups

Organizing Evaluations

Policy & Operations Evaluation Department (IOB)

Independency

Quality control systems

Peer review

Horizontal control

Certification (CBF, NEN)

Learning from Evaluations

Policy & Operations Evaluation Department (IOB)

Single, double & triple loop learning

Defining innovation spaces

Real-time experiments

Methodological Challenges

Policy & Operations Evaluation Department (IOB)

Overcoming poverty thresholds

Focus on Complementarities

Search for Policy coherence

Measuring Additionality

Institutional Strength (proxies)

Spillovers (catalyst effects)

Externalities

Institutional Challenges

Policy & Operations Evaluation Department (IOB)

Organizing Independent NGO Evaluation Units

Upfront Focus on Evaluability

Training of local (Southern) partners

Web-based open data systems (see: www.ngo-database.nl)

Openness on Failures & Successes

Comparing NGO performance (ranking)

Thanks for your Attention…

Policy & Operations Evaluation Department (IOB)