View
405
Download
0
Tags:
Embed Size (px)
Citation preview
1
Program Theory & The Theory-Driven Approach to Evaluation
Jeffrey Sheldon, M. A., Ed. M.School of Behavioral & Organizational Sciences
Claremont Graduate UniversityThe Claremont Colleges
25 October 2006
2
What does this model tell us?
3
Challenges in Program Evaluation
• Inadequate program conceptualization.
• Poor program implementation.
• Insensitive program evaluation.
• Poor stakeholder – evaluator relations.
• Scarcity of cumulative knowledge and wisdom.
4
Black Box Evaluation
• Evaluation of program outcomes without the benefit of an articulated program theory that provides insight into what is presumed to be causing those outcomes, and why.
Rossi, Lipsey & Freeman (2004)
5
Contingency View
• No single best way to conduct program evaluation.
• The choice of approaches and methods for program evaluation should be situational.
• The individual natures of programs and the uniqueness of evaluation purposes and contextual circumstances require use of a range of evaluation approaches and methods.
Chen (2005)
6
Theories Used in Evaluation
• Evaluation theory.– Guides evaluation practice, e.g., empowerment
evaluation, theory-driven evaluation, goals-free evaluation.
• Social science theory.– Theory from the extant literature, e.g., Social
Learning Theory, Theory of Reasoned Action.
• Program theory– Stakeholder theory – varies by program.
7
Program Theory
• The set of assumptions about the manner in which the program relates to the social benefits it is expected to produce and the strategy and tactics the program has adopted to achieve its goals and objectives. Within program theory we can distinguish impact theory – the nature of the change in social conditions brought about by program action and process theory – the program’s organizational plan and service utilization plan.
Rossi, Lipsey & Freeman (2004)
8
Program Theory
• Assumptions made by stakeholders about what action is required to solve a social problem and why the problem will respond to this action.
• Perception of nature of problem is from experience, conventional wisdom, discussions with peers, etc…
• Solution to the problem is practicable.
9
Assumptions
• Prescriptive – the action that is required to solve a social problem.
– Explained by the Process Model
• Descriptive – why the problem will respond to the action.
– Explained by the Impact Model
10
Process Model
• Components and activities program designers and key stakeholders see as necessary for program success.
• Example: Wired With Wisdom Parent RecruitmentLetters from PrincipalsChampion Parents: phone, email, personal contactChildren of target parents
11
WWK
Principal
Project Manager
PTO/PTAOfficers
Computer Instructor
Parents
Effects intrinsic motivation
leading to use
TeachersStudents
Champs
Positive Letter
Negative Letter
Incentive Letter
Phone
E - Mail
Personal
Follow-up
Follow-up Letter
Internet training
No effect on intrinsic
motivation leading to non-
use
Direct influence/communication:
Direct action:
Two-way influence/communication:
Process Model
Or
Recruitment Program
12
Components of Process Model
• Intervention and service delivery protocols.
• Implementing organization: assess, enhance, and ensure its capacity.
• Program implementers: recruit, train and maintain both competency and commitment.
13
Components of Process Model
• Associate organizations/community partners: establishing collaborations.
• Ecological context: seek its support at micro and macro levels.
• Target population: identify, recruit, screen, serve.
14
Impact Model
• Assumptions about causal processes through which intervention is supposed to work.
• Example:
Parent recruitment Increases or Successful recruitment,
Methods Reduces Intrinsic use of program
Motivators
15
Safer children
Wired With WisdomParent Recruitment Program
(LettersPhone calls
E - mailsPersonal contact)
Successfulrecruitment =
use of Wired With
Wisdom
Well managed family internet environment &
safety plan
ReducePerceivedBarriers
IncreasePerceivedBenefits
IncreasePerceived
Susceptibility
IncreasePerceivedSeverity
IncreaseActionCues
Impact Model
16
Components of the Impact Model
• Intervention/treatment
• Determinants– Mediators– Moderators
• Goals/outcomes: – Distal– Intermediate – Proximal
17
Logic Models & Program Theory
18
Program Theory Considerations
• Parsimony (the core of the program).
• Precision of relationships.
• Bi-directional approach.
• Program-effect decay functions.
• Dose-response functions.
• Mediators.
• Moderators.
19
Program-effect Decay Functions
20
Dose-Response Functions
21
Direct Effects Model
22
One Mediator Model
23
Indirect Effects Model
24
Multiple Mediator Effects Model
25
Moderator of Mediator Effect Model
26
Moderator of Mediator-Outcome Relationship
27
Small Group Exercise
Based on the following model, describe the program, its actions and the changes that are expected to result. Use the questions below as a guide.
What is the intervention/treatment?What are the mediators?What are the moderators?What are the proximal outcomes?What is the distal outcome?
28
Computers In Our Future
29
Theory Calls Evaluation Practitioner’s Attention To:
• Which stage or stages of the program cycle will be the focus of the evaluation?
• What do stakeholders want from the evaluation?
• What evaluation options potentially fit the given program’s context?
• What trade-offs among these options will be most profitable?
30
Using Program Theory to Design Evaluations
• Compels evaluators to be thoughtful before acting.
• Enhances understanding of program.
• Program assumptions used as scaffolding for the study.
• Informs method choices – qualitative, quantitative or mixed, that is, the contingency view!
31
Using Program Theory to Design Evaluations
• Highlights elements of program activity that deserve attention in the evaluation.
• Helps tailor evaluations to answer the most important questions (remember, parsimony).
• Heightens evaluation responsiveness and sensitivity.
• Increases validity – construct & internal.
32
Using Program Theory to Design Evaluations
• Fosters cumulative wisdom.
• Helps evaluators meet American Evaluation Association professional evaluation standards – Utility, Feasibility, Practicality, Accuracy.
• Can choose to collect data on linkage mechanisms assumed to be operative in one theory or in several theories.
• Can direct the evaluation toward investigating one link in the theory chain.
33
Is Theory-Driven Evaluation Methodologically Rigorous?
“The tie-in, or relationship, of the theory-driven approach with our best methodological work is impressive. Think about the way we establish the validity of constructs in experimental research. In essence, construct validation requires a theory, an understanding of the hypothetical network of causal associations and non-causal relationships among the variables that we might try to understand”
Crano (2003)
34
Theory-driven Evaluation The CDC Framework
• Engage stakeholders – evaluability assessment.
• Describe the program through the action and change models.
• Formulate & prioritize evaluation questions.
• Focus the evaluation design.
35
Theory-driven Evaluation The CDC Framework
• Gather credible evidence through rigorous scientific methods.
• Justify conclusions.
• Ensure utilization and lessons learned.
36
CDC Evaluation Framework
37
Effective Theory-Driven Evaluations
1. Future action directedness.
– Useful to stakeholders.
– Assessing merit is a means rather than an end.
– Provides useful information for stakeholders to improve current or future programs.
38
Effective Theory-Driven Evaluations
2. Scientific and stakeholder credibility.
– Follows scientific methods and principles to optimize validity and reliability.
– Responding to stakeholders’ values, views, concerns, and needs.
39
Effective Theory-Driven Evaluations
3. Holistic Approach
– Intrinsic value.
– Context.
40
Explicating Program Theory: Basics
Facilitated by Evaluator:
• Stakeholders reflectively examine what they are doing.
• Stakeholders identify elements that are essential for achieving program goals.
• Stakeholders articulate causal relationships.
41
Explicating Program Theory: Process
• Face-to-face meetings with stakeholders (working group or intensive interview)
• Facilitating conceptualization of program: – “Tell me how your program works.”– “What do you want your program to do?”– “What circumstance does it mitigate or need it meets?”– “Who does it impact?”
• Theorizing methods – backward reasoning (start with intended outcomes) to inputs, forward reasoning, or both.
42
Explicating Program Theory: Results
• Stake-holder buy-in & support of evaluation.
• Systematic understanding of stakeholder views, needs, and values.
• Utilization of knowledge produced by the evaluation
– Conceptual (understanding/education)– Instrumental (decision-support)– Process (making use of the logic of the evaluation)– Symbolic (justify a priori decisions)– Influence
43
Evaluation Questions Hierarchy
44
Types of Theory-Driven Evaluations
• Action Model Theory-driven process evaluation.
• Change Model Theory-driven outcome evaluation.
• Action Model + Change Model Integrated Theory-driven process/outcome evaluation.
45
Theory-Driven Process Evaluation
• Systematically assess how the following major components of an action model are being implemented in the field:
– Intervention and service delivery protocols– Target populations– Implementing organization– Implementers– Associate organizations/partners– Ecological support
46
Theory-Driven Outcome/Impact Evaluation
• Serves accountability and program improvement needs by investigating underlying causal mechanisms.
• Comments on construct ability
• Increases internal validity.
• Generates two kinds of information: – Assesses if program achieving its predetermined goals– Investigates why and how program succeeds or does not succeed
47
Integrated Process – Outcome/Impact Evaluation
Implementation of
parent recruitment
action model
Parent recruitment Increases or Successful recruitment,
Methods Reduces Intrinsic use of program
Motivators
(Action theory of success) (Conceptual theory of success)
48
References• Bickman, L. (Ed.) (1987). Using program theory in evaluation.
New Directions for Program Evaluation, No., 47. San Francisco, CA: Jossey-Bass.
• Birkmayer, J. D., & Weiss, C. H. (2000). Theory-based evaluation in practice. Evaluation Review, 21(4), 407 – 431.
• Chen, H. T. (2005). Practical Program Evaluation: Assessing, and Improving Planning, Implementation and Effectiveness. Thousand Oaks, CA: Sage.
• Chen, H. T. (1990). Theory-Driven Evaluation. Newbury Park, CA: Sage
49
References• Chen, H. T., & Rossi, P. H. (1983). Evaluating with sense: The
theory-driven approach. Evaluation Review, 7, 283 – 302.
• Chen, H. T., & Rossi, P. H. (1987). The theory-driven approach to evaluation. Evaluation and Program Planning, 10, 95 – 103.
• Crano, W. D. (2003). Theory-driven evaluation and construct validity. In S. I. Donaldson and M. Scriven (Eds.), Evaluating Social Programs and Problems: Visions for the New Millennium. Mahwah, NJ: Erlbaum.
• Donaldson, S. I. (Forthcoming). Program theory-driven evaluation science: Strategies and applications. Mahwah, NJ: Erlbaum.
50
References• Donaldson, S. I. (2002). Theory-Driven Evaluation in the New
Millennium. In S. I. Donaldson and M. Scriven (Eds.). Evaluating Social Programs and Problems: Visions for the New Millennium. Mahwah, NJ: Erlbaum.
• Donaldson, S. I., & Lipsey, M. (2006). Roles for theory in contemporary evaluation practice: Developing practical knowledge. In. I. Shaw, J. C. Greene, and M. H. Mark (Eds.). The SAGE Handbook of Evaluation, Thousand Oaks, CA: Sage.
• Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2004). Program Evaluation: Alternative Approaches and Practical Guidelines (3rd Ed.). Boston, MA: Pearson.
• Lipsey, M. (1988). Practice and malpractice in evaluation research. Evaluation Practice, 8(4), 5 – 24.
51
References• Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004).
Evaluation: A Systematic Approach (7th Ed.). Thousand Oaks, CA: Sage.
• Shadish, W. R., Cook, T., & Leviton, L. (1991). Foundations of Program Evaluation: Theories of Practice. Newbury Park, CA: Sage.
• Weiss, C. H. (1997). How can theory-based evaluation make greater headway? Evaluation Review, 21(4), 501 – 524.
• Weiss, C. H. (1998). Evaluation: Methods for Studying Programs and Policies (2nd Ed.). Upper Saddle River, NJ: Prentice Hall.