White paper and CRP6: Co-learning on Impact Evaluation Design in NRM Research Programmes

White paper and CRP6: Co-learning on Impact Evaluation Design in NRM

Research Programmes

Presented to NRM Impact Community of Practice The Worldfish Center, Penang, Malaysia, Sept. 4-5, 2012

Brian Belcher, CIFOR/Royal Roads University

THINKING beyond the canopy

CRP6 Aims

to enhance livelihoods through forestry, agroforestry and other uses of forest resources while sustaining environmental services and resource resilience.

THINKING beyond the canopy

CRP6 Components

THINKING beyond the canopy

• Stronger collaboration, focus and coherence within CG

• More, stronger and more diverse partnerships

• Results focus, shared responsibility for outcomes

• Sentinel landscapes

• Emphasis on learning by doing and on verification of progress

What’s new in CGIAR/CRP6?

THINKING beyond the canopy

CRP6 Proposal & MEIA Strategy

Discuss complexity in NRMR Recognize experimental design IE approaches not

necessarily appropriate or best Appreciate evaluation for learning & for accountability Emphasize clarifying causal assumptions through

participatory impact pathway development Recognize multiple impact pathways and multiple scales Propose using tools of Outcome Mapping, PIPA,

integrating monitoring, evaluation and impact assessment, and collaboration with other CRPs and experts

THINKING beyond the canopy

White Paper and process

White Paper and COP helps explain the need for alternative approaches and provides legitimacy for mixed methods approaches

Develop/promote consistent definitions and approaches Forum for sharing ideas, perspectives, expertise Overview and access to a range of relevant methods and

related literature Simultaneously developing “IDOs” Presentation will reflect on experience to date within

CRP6 evaluation planning and implications for WP

THINKING beyond the canopy

Concept of NRMR

CRP6 primarily contributes to SLO 1(reducing rural poverty) and 4 (sustainable NRM)

Supplementary contributions to 2 (improving food security) and 3 (improving nutrition and health)

Bottom-up approaches (“trajectories”): develop and support technology and institutions to benefit small-holders, communities)

Intermediate approaches: influence research agenda; support capacity strengthening; mainstream gender analysis)

Top-down approaches: influence policy at level of conservation orgs, development orgs, national and international policy)

THINKING beyond the canopy

Research for policy influence

predict (ex ante) or measure (ex post) effects of policy options and policy tools

provide knowledge for forming, implementing or contesting policy

identify and explain trends raise awareness of a problem improve understanding of underlying causes of economic

behaviour and environmental outcomes challenge conventional wisdom develop/influence research methods develop useful theory or conceptual framework

THINKING beyond the canopy

Implications for WP

Begin with definition of NRMR that encompasses and explains the range of ways that NRMR contributes to improvements in social and natural systems

Explicitly recognize that research processes and products contribute to change

Make stronger argument that we need alternative ways to establish “counterfactuals”

THINKING beyond the canopy

Terminology

Major source of confusion – need good, clear definitions and consistent use of key terms

e.g. SLOs are really “impacts” by contemporary definitions

still unclear whether IDOs should be defined as changes in behaviour or changes in state

THINKING beyond the canopy

Implications for WP

Distinguish “intervention” from “programme” (programme can include/support many interventions/kind of intervention)

Use “boundary partner” instead of “working partner” (consistent with K to A literature)

Distinguish & clarify “target groups” & “beneficiary partners” Distinguish between results from the use of new knowledge

and results from the process of doing the research Define outcomes as behavioural change Intended impacts include changes/conservation of

biophysical resources

THINKING beyond the canopy

Scale issues

Nested impact pathways at several scales (project, theme, component, “sentinel sites”, CRP6, SLOs)

Project-scale boundary partners & outcomes typically more direct, more tangible

Component-scale boundary partners and outcomes more difficult to define, identify and measure

THINKING beyond the canopy

Implications for WP

Need more attention to program-level issues and approaches

How to define, identify, and measure outcomes that are manifold and diffuse

How will adopting the framework change M&E and IE practice in CRP6?

supports and strengthens MEIA approaches in development in CRP6

Provides structure for building, testing methods

What benefits will this bring?

Encourages “impact culture”, clearer and more comprehensive project/program conceptualization, design and implementation

Support and encourage broader range of partnerships and interventions

Learning, feedback (monitoring for adaptive management)

What enabling changes are needed?

Recognition and legitimacy of TOC approaches within CGIAR and donors

Capacity building internally

Next steps for implementation?

TOC approaches to be used in forthcoming evaluations of past work

Recognition and legitimacy of TOC approaches within CGIAR and donors

Capacity building internally

THINKING beyond the canopy

CRP6 IDO Template

IDO

SLO

Quantified targets (10 Years)

BaselineGeographic

focus /scope

Comments [on ability to quantify

targets/baseline/geo...]

# How?

Enhanced contribution of FAT to income, food security, and nutritionIntermediate Development Impacts

Increased revenue from sale of tree products

1 Higher productivity of trees and forests coupled with improved markets and policy environment enables more rural households to participate in tree product markets and to earn more money from them. Higher productivity comes from improved germplasm and management.

Incomes from tree and forest products doubled for target households

ICRAF: new data from Sahel; data on fruits in Kenya and Malawi; range of tree products in Cameroon, Cote d’Ivoire and Sumatra; timber in NW India and fruits elsewhere;

ICRAF: new projects in Mali, BF, Niger, Sierra Leone, CDI, Malawi, Ethiopia, Rwanda, Burundi, Uganda, Kenya; Allanblackia project in Tanzania and Ghana, timber in Sulawezi, fruits in Vietnam

We have households who already sell and may enjoy better incomes; we will have households new to receiving income from FAT. We can estimate this once we decide on the locations. As for the projects mentioned, most have baselines but there is need to collate results in 2012/13.

THINKING beyond the canopy

www.cifor.cgiar.orgwww.cifor.cgiar.org