Embed Size (px)
Citation preview
How to Plan a Local Evaluation How to Plan a Local Evaluation and Lessons Learnedand Lessons Learned
Philip Rodgers, Ph.D.Philip Rodgers, Ph.D.Evaluation ScientistEvaluation Scientist
American Foundation for Suicide PreventionAmerican Foundation for Suicide Prevention
2013 Garrett Lee Smith Combined Annual Grantee Meeting2013 Garrett Lee Smith Combined Annual Grantee MeetingJune 11-13, 2013, Washington DCJune 11-13, 2013, Washington DC
AcknowledgementsAcknowledgementsU.S. Department of Health & Human Services
Substance Abuse and Mental Health Services Agency
Howard Sudak, MDAFSP Katrina Bledsoe, PhDSPRC
www.sprc.orgwww.sprc.org
Defining EvaluationDefining Evaluation
55
Why is evaluation important?Why is evaluation important?
Required by funding agencies.Required by funding agencies.
Improves performance.Improves performance.
Demonstrates effectiveness.Demonstrates effectiveness.
Advances knowledge.Advances knowledge.
As not everything can be done, there must be a basis for deciding which things are worth doing. Enter evaluation.
M. Q. Patton
Source for Patton Quote: U.S. Department of Health and Human Services, P. H. S. (2001). National Strategy for Suicide Prevention: Goals and Objectives for Action. Rockville, MD: U.S. Department of Health and Human Services, Public Health Service.
66
What are types of evaluation?What are types of evaluation?
Participatory evaluationParticipatory evaluation
Formative evaluationFormative evaluation
Summative evaluationSummative evaluation
Responsive evaluationResponsive evaluation
Goal-free evaluationGoal-free evaluation
Empowerment evaluationEmpowerment evaluation
Advisory evaluationAdvisory evaluation
Accreditation evaluationAccreditation evaluation
Adversary evaluationAdversary evaluation
Utilization evaluationUtilization evaluation
Consumer evaluationConsumer evaluation
Theory-driven evaluationTheory-driven evaluation
We will address:
Process, Outcome, and Impact Evaluations
77
What is process evaluation?What is process evaluation?
Process Process ““evaluation assesses the extent to evaluation assesses the extent to which a program is operating as it was intendewhich a program is operating as it was intended.d.””
Source: General Accounting Office. (2005). Performance Measurement and Evaluation: Definitions and Relationships (Vol. GAO-05-739SP). Washington DC: United States Government Accounting Office.
88
What is outcome evaluation?What is outcome evaluation?
Outcome Outcome ““evaluation assesses the extent to evaluation assesses the extent to which a program achieves its outcome-oriented which a program achieves its outcome-oriented objectives.objectives.””
Source: General Accounting Office. (2005). Performance Measurement and Evaluation: Definitions and Relationships (Vol. GAO-05-739SP). Washington DC: United States Government Accounting Office.
99
What is impact evaluation?What is impact evaluation?
““Impact evaluation…assesses the net effect of a Impact evaluation…assesses the net effect of a program by comparing program outcomes with program by comparing program outcomes with an estimate of what would have happened in the an estimate of what would have happened in the absence of the program.absence of the program.””
Source: General Accounting Office. (2005). Performance Measurement and Evaluation: Definitions and Relationships (Vol. GAO-05-739SP). Washington DC: United States Government Accounting Office.
I. Logic Models & EvaluationI. Logic Models & Evaluation
1111
Inputs Activities Outputs
Funds
Trainers
Materials
Trainees
Gatekeeper
Training
People trained
Satisfaction with training
Fidelity of training
Logic models can drive evaluationsLogic models can drive evaluationsGeneric Gatekeeper Training Logic Model
Outcome Evaluation
ProcessEvaluation
Short-term
a.
Knowledge
b.
Attitudes
Intermediate
c.
Identification of those at risk
d.
Referral
Long-term
f.
Treatment
g.
Suicide
deaths
a.
Knowledge
b.
Attitudes
c.
Identification of those at risk
d.
Referral
f.
Treatment
g.
Suicide
deaths
Impact Evaluation
vs. Control Group
II. How to Develop II. How to Develop Evaluation QuestionsEvaluation Questions
1313
Where do evaluation questions come from?Where do evaluation questions come from?
Generally, from the goals listed in a logic model.Generally, from the goals listed in a logic model.
More specifically, defined by stakeholders More specifically, defined by stakeholders through a collaborative process.through a collaborative process.
Depending upon circumstances, stakeholders Depending upon circumstances, stakeholders can be funders, participants, trainers, can be funders, participants, trainers, evaluators, and others, or a combination of evaluators, and others, or a combination of these.these.
1414
Divergent, convergent process…Divergent, convergent process…
Stakeholders meet with relevant materials (grant Stakeholders meet with relevant materials (grant application, logic model, etc.).application, logic model, etc.).
After a review of materials, engage in divergent After a review of materials, engage in divergent process—a free association of evaluation process—a free association of evaluation questionsquestions
After the divergent process, stakeholders After the divergent process, stakeholders collectively narrow list of questions to collectively narrow list of questions to manageable proportions through a convergent manageable proportions through a convergent process.process.
III. How to Answer III. How to Answer Evaluation QuestionsEvaluation Questions
1616
What is measurement?What is measurement?
Measurement is the means you use to collect Measurement is the means you use to collect data.data.
It includes how you collect data and what data It includes how you collect data and what data you collect (and how well you collect data).you collect (and how well you collect data).
1717
How will you collect data?How will you collect data?
Questionnaires Questionnaires (in-person, mail, email, phone)(in-person, mail, email, phone)
Psychological TestsPsychological Tests
InterviewsInterviews
Health RecordsHealth Records
Health StatisticsHealth Statistics
ObservationsObservations
LogsLogs
1818
Where do you find measures?Where do you find measures?
Create your own Create your own (pilot test!)(pilot test!)
Borrow from other evaluations Borrow from other evaluations (with permission!)(with permission!)
Search the literature Search the literature (see Additional Resources)(see Additional Resources)
Use standardized measures Use standardized measures (may cost)(may cost)
– Brown (adult) and Goldston (adolescent) reviewsBrown (adult) and Goldston (adolescent) reviews
Use existing data sources and recordsUse existing data sources and records
IV. How to Develop IV. How to Develop an Evaluation Planan Evaluation Plan
2020
What evaluation design do you need?What evaluation design do you need?
What is your purpose?What is your purpose?– Performance assessment?Performance assessment?– Evidence of concept?Evidence of concept?– Evidence of effectiveness?Evidence of effectiveness?
2121
There are four basic evaluation designsThere are four basic evaluation designs
Posttest OnlyPosttest Only XX OO
Pre- and PosttestPre- and Posttest OO XX OO
Posttest Only w/controlPosttest Only w/control XX OOOO
Pre- and Posttest w/controlPre- and Posttest w/control OO XX OOOO OO
2222
Additional data collection points can be Additional data collection points can be added to the basic desginsadded to the basic desgins
Posttest OnlyPosttest Only XX OO OO
Pre- and PosttestPre- and Posttest OO XX OO OO
Posttest Only w/controlPosttest Only w/control XX OO OOOO OO
Pre- and Posttest w/controlPre- and Posttest w/control OO XX OO OOOO OO OO
2323
Best evidence comes when subjects are Best evidence comes when subjects are randomly assigned to groupsrandomly assigned to groups
xExp. Group
Con. Group
Exp. Group
Con. Group
Pool of Subjects or
Groups
Random assignment increases the likelihood that subjects in both groups are equivalent in regards to factors that may be related to outcomes.
2424
If random assignment is not possible…If random assignment is not possible…
Compare groups that are similar.Compare groups that are similar.
Use a pretest so that group differences—to Use a pretest so that group differences—to some extent—are accounted for.some extent—are accounted for.
Evaluation Planning FormsEvaluation Planning Forms
2525
