Embed Size (px)
Citation preview
Evaluation:It’s Not Just ‘at the End’
Terry Uyeki, MSEdDirector of Evaluation & Community Services
Program Evaluation Symposium, Sept. 10, 2010
CCRP “Hats”
Program evaluation
Meeting design & facilitation (including graphic facilitation)
Qualitative data analysis
Community based participatory research (CBPR)
EvaluationIt’s not just for the report for the funder …
and it doesn’t just happen at the end…
Evaluation is the “systematic investigation of the worth or merit of an object”
Why would you develop anevaluation plan when you design
a program or intervention?
(Who is evaluation for?)
Why?
• Formative/process evaluation– How can we improve as we develop?
• Summative/outcome evaluation– What happened? How effective was it?
Project / Program Evaluation
Thinking about evaluation (benchmarks) as you develop a project
Program / Project
Design & Delivery
Program / Project
Outcomes for
Participants
Program / Project
Dissemination
Process Evaluation (what did you do?)
Outcome Evaluation (what did they do?)
Impact Evaluation (how did it affect the problem?)
Some Models for Program Design• W.K. Kellogg Foundation Logic Model
– A systematic and visual way to represent the resources and activities of your program and proposed results
• Aspen Institute Theory of Change Outcome Framework– Specifying outcomes and assumptions, and backwards-
mapping to connect outcomes to your activities
• Intervention Mapping (Bartholomew et al., 2000)– Health Promotion program planning: Design,
adoption, implementation & evaluation
Kellogg Foundation Logic Model
Logic Model: Family Visitation Program
Healthy weight in children & families; reduced incidence of Type 2 Dm.
40 families participate. Enrolled families complete 10 PACT visits.
Increased levels of physical activity & increased fruit & veg. consumption. Positive change in mediators.
PACT program publicized. Family advisors plan, conduct visits. Annual clinics held. Group activities held.
Family advisors hired, trained as coaches. Portfolio of fun activities compiled.
Making Project Objectives SMART
How can evaluation help shape project objectives?• S Specific • M Measurable • A Attainable, Actionable• R Relevant, Results-focused• T Time framed
Making Project Objectives SMARTHow can evaluation help shape
project objectives?• S Specific • M Measurable • A Attainable,
Actionable• R Relevant, Results-
focused• T Time framed
• Specific – Define “healthy eating” as increased consumption of fruits & veggies
• Measurable – Grams of fruits and vegetables reported consumed
• Attainable – Target: 95 60% of participants will improve F/V consumption; Actionable – Parents learn new ways of preparing F/V meals
• Relevant – Consumption of soda
decreases. Results-focused – 75% of families complete at least 8 visits.
• Time framed– Families complete 10 visits within 9 months.
From “PACT Program participants will eat healthier” to
At least 60% of PACT Program participants will increase consumption of fruits and vegetables.
Implications for Evaluation:• Self-reported meal recall• Weight of reported fruits and vegetables consumed• Perceived mediators in preparation of meals and
consumption of fruits and vegetables• Participation rates of program families• Satisfaction with PACT family visits• Competence of family advisors (adherence to visit
protocol; assessment of adults relative to stage of change; coaching skills)
Logic Model: Family Visitation Program
Healthy weight in children & families; reduced incidence of Type 2 Dm.
40 families participate. Enrolled families complete 10 PACT visits.
Increased levels of physical activity & increased fruit & veg. consumption. Positive change in mediators.
PACT program publicized. Family advisors plan, conduct visits. Annual clinics held. Group activities held.
Family advisors hired, trained as coaches. Portfolio of fun activities compiled.
Aspen Institute Theory of Change Outcome Framework
• A graphic representation of the change process, mapping the pathways of change that will be brought about by an intervention/program
• From Outcomes (long-term goals and assumptions behind them)
• Backwards mapping to pre-conditions required to cause the desired change.
• Connected outcomes = pathways of change
Project “Superwomen” as a Logic Model
Project “Superwomen” as a Theory of Change Outcome Framework
From “Intervention Mapping”: DefiningPerformance Objectives for Health Behavior
• Negotiate the use of a condom with a partner (based on negotiation theory, Fisher & Ury, 1991)– State mutual goals such as prevention of
pregnancy or AIDS– State clearly the intention of using a condom as a
prerequisite for intercourse– Listen to partner’s concerns– Pose solutions to partner’s concerns that
reference mutual goals & personal requirements
Why Build Evaluation Plan as Part of Program/Intervention Design?
• Not just for the funder…• Keeps objectives realistic• Ensures that objectives are measurable• Helps with QA/QC• Enables one to make program adjustments• What happens if you did wait until the end?• Other reasons?
Contact us at
California Center for Rural Policy
Humboldt State University
707-826-3400 [email protected]
www.humboldt.edu/~ccrp/
