Embed Size (px)
Citation preview
Integrating Evaluation into the Design of the Minnesota
Demonstration Project
Paint Product Stewardship Initiative
St. Paul, MN
May 1, 2008
Matt Keene, Evaluation Support Division
National Center for Environmental Innovation
Office of Policy, Economics and Innovation
US Environmental Protection Agency
2
Purpose of the Presentation
Provide the PPSI with an understanding of the work of the evaluation committee and the process of integrating evaluation into the design of the Minnesota Demonstration Project.
3
Presentation Outline
1. Introduction to Program Evaluation
2. The Evaluation Committee & Goal 6
3. Integrating Evaluation into MN
4. Questions, Comments, and Next Steps
4
Program Evaluation
Definition
• A systematic study that uses measurement and analysis to answer specific questions about how well a program is working to achieve its outcomes and why.
Orientation/Approaches to Evaluation
• Accountability External Audience
• Learning & Program ImprovementInternal/External Audiences
5
Measurement and Evaluation
Program Evaluation:
A systematic study that uses measurement and analysis to answer specific questions about how well a program is working to achieve its outcomes and why.
Performance Measurement:
A basic and necessary component of program evaluation that consists of the ongoing monitoring and reporting of program progress and accomplishments, using pre-selected performance measures.
6
The Evaluation Committee & Goal 6
Evaluation Team (Committee)
Purpose and Function of the Evaluation Team
Funding and Support
The Work Plan and Goal 6
7
The Evaluation Committee & Goal 6
What will we evaluate?
• Paint, Management Systems, Education, Markets
Why are we evaluating the program?
• Leadership, Learning, Transfer
Can we evaluate this project?
• We must integrate evaluation into the project• We need a framework to follow…and we are
building it as we go• Initially, integrating evaluation into your
program is a design and planning activity
1. Team 2. Mission3. Goals & Objectives4. Logic Model
Integrating Evaluation into The Minnesota Project
Integrating Evaluation into Program Design
Program
QuestionsDocumentation
Measures
1. Context2. Audience3. Communication4. Use
1. Data Sources2. Collection Methods & Strategy3. Analysis Tools4. Data Collection5. Data Management
1. Evaluation Policy2. Evaluation Methodology
is our program
Describing the MN program
Mission
Goals and objectives
Logic model
Program
Select and Describe the Program
1. Team 2. Mission3. Goals & Objectives4. Logic Model
Integrating Evaluation into Program Design
1. Team 2. Mission3. Goals & Objectives4. Logic Model
Integrating Evaluation into Program Design
Program
Questions
What are the critical questions to understanding the success of the MN program?
Evaluation Questions
Examples of Draft Questions
Has this been a collaborative and cooperative process?
How successful is the PSO?
How effective are education and outreach materials?
How effective are the paint management systems?
What are the best options for a national system?
1. Team 2. Mission3. Goals & Objectives4. Logic Model
Integrating Evaluation into Program Design
Program
Questions
1. Team 2. Mission3. Goals & Objectives4. Logic Model
Integrating Evaluation into Program Design
Program
Questions
What contextual factors may influence the answers to each question?
Who are the audiences for each question?
•What’s the best way to communicate with each audience?
•How might each audience use the answer to each question?
Evaluation Questions
1. Context2. Audience 3. Communication4. Use
Evaluation Questions
1. Team 2. Mission3. Goals & Objectives4. Logic Model
Integrating Evaluation into Program Design
Program
Questions
1. Context2. Audience 3. Communication4. Use
What are the critical questions to understanding the success of the MN program?
What contextual factors may influence the answers to each question?
Who are the audiences for each question?
• What’s the best way to communicate with each audience?
• How might each audience use the answer to each question?
1. Team 2. Mission3. Goals & Objectives4. Logic Model
Integrating Evaluation into Program Design
Program
Questions
1. Context2. Audience 3. Communication4. Use
What can we measure to answer each question?
Where can we find the information for each measure?
How can we collect the information?
Given our questions and information to be collected, what will be an effective collection strategy?
Performance Measures
1. Data Sources2. Collection Methods & Strategy3. Analysis Tools4. Data Collection5. Data Management
Measures
What analytical tools will give us the most useful information?
How will we implement the collection strategy?
How will we manage the data?
Performance Measures
1. Team 2. Mission3. Goals & Objectives4. Logic Model
Integrating Evaluation into Program Design
Program
Questions
1. Context2. Audience 3. Communication4. Use
1. Data Sources2. Collection Methods
& Strategy3. Analysis Tools4. Data Collection5. Data Management
Measures
What can we measure to answer each question?
Where can we find the information for each measure?
What methods are best suited for each measure?
What analytical tools will give us the most useful information?
Given our questions and information to be collected, what will be our collection strategy?
•How will we implement the collection strategy?•How will we manage the data?
Performance Measures
1. Team 2. Mission3. Goals & Objectives4. Logic Model
Integrating Evaluation into Program Design
Program
Questions
1. Context2. Audience 3. Communication4. Use
1. Data Sources2. Collection Methods
& Strategy3. Analysis Tools4. Data Collection5. Data Management
Measures
1. Team 2. Mission3. Goals & Objectives4. Logic Model
Integrating Evaluation into Program Design
Program
Questions
1. Context2. Audience 3. Communication4. Use
1. Data Sources2. Collection Methods
& Strategy3. Analysis Tools4. Data Collection5. Data Management
Documentation: Methodology & Policy
Evaluation Methodology
The process of integrating evaluation generates a framework for our methodology
Evaluation Policy
Guide MN & PPSI
Guides strategy and planning for evaluation and program management
1. Evaluation Methodology2. Evaluation Policy
Measures
Documentation
Check the Logic and Flow Revisit the process and the decisions made
Look for the flow in the process and identify potential breaks
Identify potential obstacles to our approach to understanding and managing the performance of the MN demonstration program
1st cycle is integrating – next cycle begins implementation
1. Team 2. Mission3. Goals & Objectives4. Logic Model
Integrating Evaluation into Program Design
Program
Questions
1. Context2. Audience 3. Communication4. Use
1. Data Sources2. Collection Methods
& Strategy3. Analysis Tools4. Data Collection5. Data Management
1. Evaluation Methodology
2. Evaluation Policy
Measures
Documentation
19
Questions, Comments and Clarifications
1. Questions for the Demonstration Committee
2. Questions for the Evaluation Committee
1. Introduction to Program Evaluation
2. The Evaluation Committee & Goal 6
3. Integrating Evaluation into MN
20
Thank You!
Evaluation Support DivisionNational Center for Environmental
InnovationOffice of Policy, Economics and InnovationU.S. Environmental Protection Agency
Matt Keene
(202) 566-2240
www.epa.gov/evaluate
