Upload
nikolas-edney
View
220
Download
1
Tags:
Embed Size (px)
Citation preview
February 2009
Measuring Value: Using Program Evaluation to Understand What’s
Working -- Or Isn’t
Juliana M. Blome, Ph.D. , MPHOffice of Program Analysis and Evaluation
National Institute of General Medical Sciences
MORE Program Directors MeetingColorado Springs, Colorado
June 12, 2009
NIGMS
February 2009
“Program Evaluation…!?”
NIGMS
February 2009
Program Evaluation: What is it?
Program evaluations are individual, systematic studies that use objective measurement and analysis to answer specific questions about how well a program is working.
- GAO/GGD-00-204 Program Evaluation
NIGMS
February 2009
Evaluation Answers Questions Such As….
• Does it work?• How well does it work?• Does it do what we want it to? • Does it work for the reasons we think it does? • Is it cost effective? • Are the benefits worth it? • What are the unintended consequences?
NIGMS
February 2009
Research vs. Program Evaluation
Evaluation
Judges merit or worth Policy & program interests of
stakeholders paramount Provides information for
decision-making on specific program
Conducted within setting of changing actors, priorities, resources, & timelines
Research
Produces generalizable knowledge
Scientific inquiry based on intellectual curiosity
Advances broad knowledge and theory
Controlled setting
NIGMS
February 2009
Why bother?
To gain insight about a program and its operations
To improve practice - to modify or adapt practices to enhance the likelihood of success
To assess effects – to determine if we’re meeting our goals and provide evidence of effectiveness
NIGMS
February 2009
Guidelines for Conducting Successful Evaluations
Invest heavily in planning early on Integrate evaluation into ongoing program activities Use knowledgeable, experienced evaluators
NIGMS
February 2009
Evaluator Skills
Evaluation theory and methods Research methods (design, planning, statistics,
qualitative and quantitative methods) Data collection, analysis and interpretation Communication and Interpersonal skills Content area skills Project management Ethics
At universities and colleges, this type of expertise is found in the social and behavioral sciences departments!
NIGMS
February 2009
Evaluation Costs
The National Science Foundation's “rule of thumb” about evaluation budgets is 10%
of the total grant amount.
NIGMS
February 2009
Types of Evaluations
Needs Assessment What is nature & extent of the issues program should address? Planning phase
Feasibility Study Is evaluation appropriate and/or affordable? Maturity/timeliness issue? Process or outcome evaluation produced
Process Evaluation Is program is being conducted & producing output as planned? How can process can be improved?
Outcome Evaluation To what extent have a program’s goals have been met?
NIGMS
February 2009
StandardsUtility
Feasibility
Propriety
Accuracy
Engage Stakeholders
Focus Evaluation
Design
Describe the program
Gather credible evidence
Justify conclusions
Use and share lessons learned
CDC Framework for Program Evaluation
Steps
Milstein et al, Health Promotion Practice, July 2000, Vol 1(3): 221-228
For more info:
http://www.cdc.gov/EVAL/
NIGMS
February 2009
Evaluation Standards
UtilityEvaluations should serve the practical information needs of a given audience
FeasibilityEvaluations take place in the field and should be realistic, prudent, diplomatic and frugal
ProprietyThe rights of individuals affected by evaluations should be protected
Accuracy Evaluations should produce and convey accurate information about a program’s merit and/or worth
Guiding Principles for Evaluators, American Evaluation Association, www.eval.org
NIGMS
February 2009
CDC Framework: Key Steps in Evaluation
1. Engage stakeholders
2. Describe the program
3. Focus the evaluation design
4. Gather credible evidence
5. Justify conclusions
6. Ensure use and share lessons
NIGMS
February 2009
Step 1- Engage Stakeholders
Who are the stakeholders?
Those involved in program operations, those affected
by the program operations, and users
of evaluation results
NIGMS
February 2009
What are the goals and specific aims of the program?
What problem or need is it designed to address?
What are the measurable objectives? What are the strategies to achieve the objectives?
What are the expected effects?
What are the resources and activities?
How is the program supposed to work?
Step 2 - Describe the Program
NIGMS
February 2009
“I think you should be more explicit here in Step Two.”By Sidney Harris, Copyright 2007, The New Yorker
NIGMS
February 2009
Step 3 - Focus the evaluation design
• What do you want to know?
• Consider the purpose, uses, questions, methods, roles, budgets, deliverables etc.
An evaluation cannot answer all questions for all stakeholders.
NIGMS
February 2009
Step 4 - Gather credible evidence
Evidence must be believable, trustworthy, and relevant
• Information scope, sources, quality, logistics • Methodology & data collection • Who is studied and when
NIGMS
February 2009
Step 5 - “Justify” Conclusions
Consider data:
• Analysis and synthesis - determine findings
• Interpretation - what do findings mean?
• Judgments - what is the value of findings based on accepted standards?
• Recommendations – - what claims can be made?
- what are the limitations of your design?
NIGMS
February 2009
Step 6 - Use and share results
Share lessons learned with stakeholders!
Provide feedback, offer briefings. disseminate findings
NIGMS
February 2009
Are you overwhelmed?
Next Session – Moving from the abstract to the concrete