Skunk Works Evaluation Tools: How do we know if we are having an impact?

Skunk Works Evaluation Tools:

How do we know if we are having an impact?

What are we trying to achieve?

Strategies

Mind Shaping

Process: Activities

to demonstra

te application

Capacity

Building

Accessible ES Tools

Improved State of HWB and Biodiversity (Ecosystem Services)

Outcomes

Use of ES information in decisions

Operationalizing Outcomes

• Material living standards

• Physical health• Mental health• Social

connections• Spiritual

fulfillment• Cultural

identity• Security• Life

satisfaction• Equity

Use of ES and Policy

Decisions

Improved ES Improved HWB

• Increased policy documents with reference to ES as desired outcomes or factors in decision-making

• Increased discussion of ES in policy meetings

• Clean drinking water

• Reliable water flow in the dry season

• Reduced, in-stream sediment concentration

What other factors influence our intended outcomes?

Individual experience and personalities

Social and institutional context

Political climate

Other initiatives in place

Resources

• Land-use change (outside of decision)

• Climate change • Extreme events

(e.g., fire or flood)

• Other sources not related to natural resources

• Background factors (physical, mental, social health)

• Biophysical context

Use of ES and Policy

Decisions

Improved ES Improved HWB

So what to we need to consider?1) Stage of Evaluation

– At what stage are we in completing our strategies?

2) Design – How confident can we be that our intervention

caused the outcomes?

3) Data Collection Methods– Which methods are most valid to measure

expected outcomes?

Evaluation Stages

Developmental• What is promoting innovation in

the project?

Formative• How well are our strategies

getting us to our goals?

Summative (Impact)• Did it work?

Design

• Control and Experiment group/site receive same measurement before and after intervention

Quasi-experiment/

BACI• Control and experiment group/site

randomized and measured post-intervention

Posttest Only

• Pre and post measure the experimental group/site

One Group Pre-Posttest

• Control and Experimental group/site non-randomly assigned, measured post intervention

Natural Experiment

• Post intervention measurement onlyCase Study ✔

Some Example Methods…

To Test Mind Shaping:

e.g.• Stakeholder focus on ES• Mediation of different ideas• Understanding and Awareness of ES

Method: Mental Models

Method 1: Mental Models

Tells us: To what extent ES are being incorporated into Structure and Content of Knowledge and preferences

• Accomplished during an interview (1hr)• All or sample of participants, individual or group• Content analysis and stakeholder comparison• Improved with pre-post design, control group,

and/or randomization• Necessary: control for the hypothesized

intervention

To Test ES use in Decisions: e.g.• Policy documents that refer to ES as

desired outcomes or factors in decision-making

Method: Content Analysis

Method 2: Content Analysis

Tells us:The frequency and parameters of ES used to make a decision, and the relative influence of ES to other factors • Desktop document review• Can combine with clarifying interviews

with document preparers• Either inductive or deductive analysis (a

rubric helps)

To Test Increased HWB: e.g. • Physical Health• Life Satisfaction and Happiness• Political Voice and Governance

Method: Community-level Survey

Method 3: Community-level Survey

Tells us:Baseline and change values for actual and perceived HWB attributes

• Usually best developed based on ideas from interviews or focus groups

• In-person, paper or Web-based• Distributed among the target population• Some indicators may already exist from

local health departments or other agencies

A note on HWB measurements Qualitative and Quantitative

approaches necessary to hone in on appropriate indicators

Other Methods

• Network analysis: measure increase in information exchange, collaboration, and social capital related to ES and/or HWB

• Focus groups/interviews: Just about anything

• Participant Observation: Participant interaction with the tools and which are most effective at what

• Observation Protocols: Adaptation of ES specific terminology in meetings or trainings

Water quality

high

low

time

Conservation impact site

Control

Difference in mean values

improved water quality!

To Test Improved ES

e.g. improved water quality and dry-season flow

Methods: BACI and modeled controls

Method 4: BACI and modeled controlsTells us: Water quality relative to before the conservation impact and to control areas • Experimental backbone: Before-After-Control-

Impact (BACI) (conservation impact)• Modeled controls and exploration of “minimum

monitoring”• Post-treatment/impact only• Experimental controls to validate NatCap

models: model “no action” as a comparison