Upload
kiera-chelton
View
218
Download
0
Embed Size (px)
Citation preview
Objectives
• Why should we evaluate?
• Which activities should we evaluate?
• When should we evaluate?
• How should we evaluate (methods)?
Why Do We Evaluate?
• Design a program. (or item; e.g., curriculum, poster)Baseline, formative
• Determine if objectives were met. (best practices)Outcome, process, impactBuild an evidence base for ACSM
• Re-design or refine a program. (lessons learned)Process, outcome
Which Activities Should We Evaluate?
Is this a new activity?
Is this an activity that may be expanded?
Have we invested a lot in this activity?
Is something unexpected happening?
Does the funder want an evaluation?
IF YES ….. EVALUATE!
Case Example
Is this a priority program for evaluation?
Why or why not?
Which objectives or activities are most
important to evaluate?
FORMATIVE EVALUATION
PROCESS EVALUATION
OUTCOME EVALUATION
IMPACT EVALUATION
ACSM ASSESSMENT
KAP Surveys
Key Informant Interviews
Media Scans
Focus Groups
Analysis of Program Data
Secondary Analysis
Simple Surveys
Client Exit Interviews
When Can We Use Quantitative Methods?
BASELINE FORMATIVE PROCESS OUTCOME IMPACT
Routine Surveillance
Analysis of NTP Data
Analysis of Project Data
Simple Surveys
Population-Based SurveysPopulation-based
Surveys
Use a QUANTITATIVE Method to:
• Count things or people.
• Measure your program against a benchmark.
• Monitor trends over time.
• Statistically correlate factors: Is there a connection between variables?
When Can We Use Qualitative Methods?
BASELINE FORMATIVE PROCESS OUTCOME IMPACT
Focus Groups
Exit Interviews
Media Scans Media Scans
Informant Interviews
Exit Interviews
Use a QUALITATIVE Method to:
• Find out overall concerns, opinions.
• Gather initial information or sense of direction.
• Get details of/reasons for any problems.
• Test communication materials.
Combine BOTH for Best Results:
• Identify key issues with a focus group or interviews.
• Analyze NTP data to look for correlates or to obtain a baseline measurement.
• Use surveys to measure before and after results.
• Conduct interviews to assess how your process is going.
• Link your final outcome data to surveillance or NTP data.
Which Method and When?
BASELINE FORMATIVE PROCESS OUTCOME IMPACT
Analysis of Routine Surveillance
Focus Groups and In-Depth Interviews
KAP KAP
Exit Interview
Exit Interview
Simple Survey
Media Scan
Media Scan
What is the purpose of the evaluation?
I am at the end of a three-year, district-level ACSM project. Did we produce the expected outputs and short-term outcomes?
• Analysis of project data, focus group, or KAP survey?
I am designing a media campaign to inform my workers that TB treatment is free. What is the best way to get this message to them? What do they already know?
•Routine surveillance, KAP survey, or focus group?
What is the most rigorous method to answer the evaluation question?
The NTP wants an impact evaluation of ACSM because these new strategies have never been used at the national level.• Experimental design, secondary data analysis, exit interviews?
I am looking for a successful ACSM model that I can replicate in my district. • Case study, KAP survey, experimental design?
We trained 50 DOTS nurses on interpersonal communication and counseling skills and we want to know if they are using them correctly.• Time location survey, exit interviews, secondary data analysis?
How quickly do you need the results?
The NTP just announced funding for ACSM to reduce default. I have to design an activity in six weeks! I need to know why defaults are high in my district.
• KAP survey, simple survey, analyze surveillance data?
We have Global Fund money to implement a whole “package” of ACSM interventions to improve case detection over five years. We really need to know if the activities are effective!
•Research study, informant interviews, KAP survey?
What financial and human resources do I have?
I am SO lucky! My donor wants a formal evaluation of ACSM and gave us a lot of money for it. AND we have partners at the School of Public Health to help us!
•KAP survey, routine surveillance, research study?
My donor thinks that radio/TV/newspapers are the best way to reach our population and is giving me a lot of money to improve coverage of TB issues. But they want all of the money to go toward implementation. How can I know if it is effective?
•KAP survey, media scan, focus group?