Overview of Program EvaluationProgram Evaluation Basics Webinar Series
Mary E. Arnold, Ph.D.Associate Professor and Youth Development Specialist
Oregon State University
4-H Professional Development WebinarOctober 11, 2012
Overview of Program Evaluation
Let’s start at the very beginning- a very good place to start
Goals for Today’s Webinar1. Learn a basic history of youth program
evaluation2. Learn reasons for conducting
evaluations3. Learn the different levels of evaluation4. Learn Bennett’s Hierarchy- an well
known Extension evaluation model5. Learn different evaluation purposes6. Learn the steps of the evaluation
process7. Review expectations for evaluation for
OSU Extension8. Review expectations for compliance
with OSU’s Institutional Review Board (IRB)
Arnold, M. E. & Cater, M. (2011). From then to now: Emerging directions for youth program evaluation. Journal of Youth Development, 6(3). On-line: Article NO:110603FA005.
A Brief History of Youth Program Evaluation
• Publications related to youth program evaluation have flourished in the past 10 years; 20 years ago there was very little to be found
• The evaluation field as a whole is only now approaching 40 years
• 1960s saw rise to federally mandated (and funded) evaluations
• Evaluation results were poor, and had little effect on program expansions or reductions
• This gave rise to methodological critiques• Focus on rigor- experimental design to measure
outcomes• The importance of qualitative methods to
understand the “human” side of social programs• And, add in the developmental influences at play
in youth programs
*** Carnegie Council on Youth Development (1989). Turning points: Preparing American youth for the 21st century. Washington, DC: Author.
Youth Development Program Evaluation
• Began in the late 1800s• Focused on success stories – it was the right thing
to do!• 1980s dawn of the age of accountability• Late 1980s youth development programs
emerged as different from intervention programs, with different outcomes*** for youth:
• Intellectually reflective (we might call this critical thinking today)
• Enroute to a life of meaningful work• Good citizens• Caring and ethical• Healthy
• Even so, first efforts were focused on “reach” and participant satisfaction
• Finally, emerged an emphasis on demonstrating impact
Poll # 1: What has been the primary focus of evaluations you have done?
Why Do We Evaluate?• Help others understand the program
(stakeholders)• Understand the need for a program • Improve the program• Improve teaching• Understand the program’s impact• Determine if the program is progressing as
planned• Determine if the program is worth the cost• Meet grant reporting criteria• Meet administrative requirements
Poll # 2: Why do you evaluate your programs?
Tiers of Program Evaluation
Adapted From:Weiss, H. B. & Jacobs, F. H. (1988). Evaluating Family Programs. New York: Aldine de Gruyter.
“Bennett’s Hierarchy”
Bennett, C. (1975). Up the hierarchy. Journal of Extension. PDF available at: http://www.joe.org/joe/1975march/1975-2-a1.pdf
Types of Evaluation• Planning Evaluation (Needs Assessment)
• Process Evaluation• Evaluability Evaluation• Outcome Evaluation• Efficiency Evaluation
Garber, P.A., & Grinnell, R. M. (1994). Evaluation and Quality Improvement. Needham Heights, MA: Allyn & Bacon.
Poll # 3: What types of evaluations have you done?
The Steps of a Program Evaluation
Forming an
Evaluation Team
Developing
Evaluation Questions
Gathering Data
Conducting Data Analysis
Interpreting
Evaluation Results
Sharing Results
Taking Action
The Steps of a Program Evaluation
Forming
an Evaluation Team
Developing
Evaluation
Questions
Gathering Data
Conducting Data
Analysis
Interpreting
Evaluation
Results
Sharing
Results
Taking Action
The Steps of a Program Evaluation
Forming an
Evaluation Team
Developing Evaluation Questions
Gathering Data
Conducting Data Analysis
Interpreting
Evaluation
Results
Sharing
Results
Taking Action
The Steps of a Program Evaluation
Forming an
Evaluation Team
Developing
Evaluation Questions
Gathering Data
Conducting Data Analysis
Interpreting
Evaluation
Results
Sharing
Results
Taking Action
The Steps of a Program Evaluation
Forming an
Evaluation Team
Developing
Evaluation Questions
Gathering Data
Conducting
Data Analysis
Interpreting
Evaluation
Results
Sharing
Results
Taking Action
The Steps of a Program Evaluation
Forming an
Evaluation Team
Developing
Evaluation Questions
Gathering Data
Conducting Data Analysis
Interpreting Evaluation Results
Sharing
ResultsTaking Action
The Steps of a Program Evaluation
Forming an
Evaluation Team
Developing
Evaluation Questions
Gathering Data
Conducting Data Analysis
Interpreting
Evaluation Results
Sharing Results
Taking Action
The Steps of a Program Evaluation
Forming an
Evaluation Team
Developing
Evaluation Questions
Gathering
Data
Conducting Data Analysis
Interpreting
Evaluation Results
Sharing Results
Taking Action
Program Evaluation Expectations
ALL Extension faculty members must:
• Document measurable program outcomes on at lest one priority educational effort annually through an impact statement in SOARS
• This expectation takes into account the faculty member’s length of service, rank and experience in current assignment
• For those who have scholarship in their position description, evaluation efforts often lead to scholarly production (conference presentations, journal articles, etc.)
Poll # 4: What is the primary reason you have conducted program evaluations in the past 3 years?
Institutional Review Board (IRB)AKA… Do I have to do that?
Research is determined by three qualities:1. Systematic inquiry into a phenomena2. That is designed to develop or contribute3. Generalizable knowledge
Human Subjects Are:Living individuals about whom an investigator conducting research obtains:
4. Data through an intervention or interaction with the individual, or5. Identifiable private information
Institutional Review Board (IRB)Okay, I have to do it… now what?
Go to the IRB Website at: http://oregonstate.edu/research/irb/1. Complete the “Does Your Study Require IRB Review? Form2. Complete online ethics training modules3. Complete the steps listing under “Preparing and Initial
Submission” at http://oregonstate.edu/research/irb/preparing-initial-submission