Upload
kasey-mcneil
View
221
Download
1
Tags:
Embed Size (px)
Citation preview
1
Developing An Evaluation Plan
For TB Control Programs
Developing An Evaluation Plan
For TB Control Programs
Division of Tuberculosis EliminationNational Center for HIV, STD, and TB Prevention
Centers for Disease Control and Prevention
2
Developing An Evaluation Plan
For TB Control Programs
Developing An Evaluation Plan
For TB Control Programs
Reference: A Guide to Developing an Evaluation Plan
3
Why Develop an Evaluation Plan?
Provides a cohesive approach to Provides a cohesive approach to conducting evaluation and using the conducting evaluation and using the resultsresults
Guides evaluation activitiesGuides evaluation activities
Explains what, when, how, why, whoExplains what, when, how, why, who
Documents the evaluation process for all Documents the evaluation process for all stakeholdersstakeholders
Ensures implementation fidelityEnsures implementation fidelity
4
Guide to Developing An Evaluation Plan
Document referenced throughout Document referenced throughout presentationpresentation
Provides a template and instructions to Provides a template and instructions to help TB program staff develop an help TB program staff develop an evaluation planevaluation plan
Steps to evaluation are explained in Steps to evaluation are explained in detail detail
Completing sections and tables will result Completing sections and tables will result in an evaluation planin an evaluation plan
5
The CDC Program Evaluation Framework
Standards
UtilityFeasibilityProprietyAccuracy
Engagestakeholders
Steps
Describethe program
Gather credibleevidence
Focus theEvaluation
design
Justifyconclusions
Ensure useand share
lessons learned
6
The CDC Program Evaluation Framework
Systematic method for evaluationSystematic method for evaluation
Based on research and experienceBased on research and experience
Flexible and adaptableFlexible and adaptable
Promotes a participatory approachPromotes a participatory approach
Focuses on using evaluation findingsFocuses on using evaluation findings
7
Sections of an Evaluation Plan
IntroductionIntroduction
Stakeholder Assessment Stakeholder Assessment Step 1: Engage StakeholdersStep 1: Engage Stakeholders
Background and Description of the TB Background and Description of the TB Program and Program Logic ModelProgram and Program Logic Model Step 2: Describe the ProgramStep 2: Describe the Program
Focus of the EvaluationFocus of the Evaluation Step 3: Focus the Evaluation DesignStep 3: Focus the Evaluation Design
8
Sections of an Evaluation Plan
Gathering Credible Evidence: Data Gathering Credible Evidence: Data Collection Collection Step 4: Gather Credible EvidenceStep 4: Gather Credible Evidence
Justifying Conclusions: Analysis and Justifying Conclusions: Analysis and InterpretationInterpretation Step 5: Justify ConclusionsStep 5: Justify Conclusions
Ensuring Use and Sharing Lessons Ensuring Use and Sharing Lessons Learned: Reporting and DisseminationLearned: Reporting and Dissemination Step 6: Ensure Use and Share Lessons LearnedStep 6: Ensure Use and Share Lessons Learned
9
Introduction
An introduction provides background An introduction provides background information, identifies the purpose of the information, identifies the purpose of the evaluation, and provides a roadmap of the evaluation, and provides a roadmap of the plan.plan.
Evaluation GoalEvaluation Goal
What is the purpose of the evaluation?What is the purpose of the evaluation?
Evaluation TeamEvaluation Team
Who is your evaluation coordinator?Who is your evaluation coordinator?
Who are the members of your evaluation team?Who are the members of your evaluation team?
Reference: Table 1 in the Evaluation Plan Guide
10
Stakeholder Assessment
Stakeholders are individuals with vested Stakeholders are individuals with vested interests in the success of the TB program. interests in the success of the TB program. Involving stakeholders increases the credibility Involving stakeholders increases the credibility of the evaluation and ensures that findings are of the evaluation and ensures that findings are used as intended.used as intended.
Who are the stakeholders in your TB program?Who are the stakeholders in your TB program?
What are their interests in the evaluation?What are their interests in the evaluation?
What role do they play in the evaluation?What role do they play in the evaluation?
How do you plan to engage the stakeholders?How do you plan to engage the stakeholders?
Reference: Table 2 in the Evaluation Plan Guide
11
Background and Description of the TB Program
The program description ensures that The program description ensures that stakeholders have a shared understanding of stakeholders have a shared understanding of the program and identifies any unfounded the program and identifies any unfounded assumptions and gaps.assumptions and gaps.
NeedNeed
What problem does your program address?What problem does your program address?
What are the causes and consequences of the What are the causes and consequences of the problem?problem?
What is the magnitude of the problem?What is the magnitude of the problem?
What changes or trends impact the problem?What changes or trends impact the problem?
12
Background and Description
ContextContext What are environmental factors that affect your program?What are environmental factors that affect your program?
Target Population Target Population Does your program target the TB concerns of one Does your program target the TB concerns of one
population?population?
Program ObjectivesProgram Objectives What objectives have been set for your program?What objectives have been set for your program?
Stage of DevelopmentStage of Development Is this a new initiative or is it well established?Is this a new initiative or is it well established?
13
Background and Description
ResourcesResources What resources are available to conduct the program What resources are available to conduct the program
activities?activities?
ActivitiesActivities What are program staff doing to accomplish program What are program staff doing to accomplish program
objectives?objectives?
Outputs Outputs What are the direct and immediate results of program What are the direct and immediate results of program
activities (materials produced, services delivered, etc.)?activities (materials produced, services delivered, etc.)?
OutcomesOutcomes What are the intended effects of the program activities?What are the intended effects of the program activities?
Reference: Table 3 in the Evaluation Plan Guide
14
Program Logic Model
A logic model is a graphic depiction of the A logic model is a graphic depiction of the program description.program description.
Arrows describe the links between Arrows describe the links between resources, activities, outputs and resources, activities, outputs and outcomes outcomes
A logic modelA logic model Provides a sense of scope of your programProvides a sense of scope of your program
Ensures that systematic decisions are made Ensures that systematic decisions are made about what is to be measuredabout what is to be measured
Helps to identify and organize indicatorsHelps to identify and organize indicators
15
Program Logic Model
16
Contact InvestigationContact Investigation
Goal: Prevent TB among contacts to cases (by finding and testing contacts for Goal: Prevent TB among contacts to cases (by finding and testing contacts for TB and LTBI, and then treating infected contacts to completion). TB and LTBI, and then treating infected contacts to completion).
A
Inputs
B
Activities
C
Short-termOutcomes
D
IntermediateOutcomes
E
Long-term Outcomes
1Adequate infrastructure
i Qualified, trained and and motivated staffii Community and congregate setting partnershipsiii Policies, procedures, and guidelines iv Ongoing data collection, monitoring, and reporting systemsv Adequate physical, diagnostic, and treatment resourcesvi Linkages between
jurisdictionsvii Adequate data collection
tools viii Partnership with private
providers
1a a
Interview/reinterview casesi Build rapportii Provide educationiii Obtain information about source case and contacts
b Locate and evaluate contact:i Follow-upii Educationiii Examination & testing*
cOffer treatment
d Treat contact – case management(DOT/DOPT/incentives
2a
Comprehensiveinterview tool
bStaff trained in interviewtechniques
3a
Legal mandate to collectcontact information fromcongregate settings
2
aReporting
3a Monitor:
i Data collectionii Data managementii Data analysisiv Data dissemination
a4Conduct periodic reviewof cases/contacts and progresstoward contact treatment goals
aCases identify contacts
1
2a
Contacts educated
3 aContacts evaluated
4a
Contacts followed up
5 aContacts start treatment
6 aEvidence-based decisionsabout continuation or termination of contact investigation
2 a
1aContacts completeappropriate treatment for active TB or LTBI
1aActive TB cured in contactsb TB (prevented) in contacts with LTBI
2a
Reduced incidence and prevalence of TB
3a
TB eliminated
Improved approaches forcontact investigation
17
Sample Logic ModelSample Logic Model
Project Description of TB Support ProgramProject Description of TB Support ProgramResourcesResources ActivitiesActivities OutputsOutputs OutcomesOutcomes
InitialInitial SubsequentSubsequent Short-termShort-term IntermediateIntermediate Long-termLong-term
LHAs of
Salvadoran community
TB Staff
Funding
Community based organizations serving Salvadoran community
Hiring LHAs LHAs training LHAs hired & trained
Provide Spanish speaking and culturally competent services for Salvadoran community
Increase utilization of TB services by Salvadoran Community
Eliminate TB in Salvadoran community
Community outreach
Education TB outreach and education conducted
Increase TB knowledge in Salvadoran community
Counseling & support provided
Trust built between health care providers and the Salvadoran community
TB screening/
testing
Testing done and referrals made
Early TB and LTBI detection and interventions
Reduce TB transmission
Patients offered treatment
Develop treatment plans
DOT visits
Treatment plans
DOT administered
Patients accept treatment for TB and LTBI;
Patients adhere to treatment
Complete treatment
Increase completion of therapy rate
( reduce hospital admissions for TB among Salvadorans)
18
Focus of the Evaluation
Since you cannot feasibly evaluate Since you cannot feasibly evaluate everything, you must focus the everything, you must focus the evaluation by prioritizing and selecting evaluation by prioritizing and selecting evaluation questions.evaluation questions.
Stakeholder NeedsStakeholder Needs Who will use the evaluation findings?Who will use the evaluation findings?
How will the findings be used?How will the findings be used?
What do stakeholders need to learn/know from What do stakeholders need to learn/know from the evaluation? the evaluation?
19
Focus of the Evaluation
Process EvaluationProcess Evaluation
What resources were required?What resources were required?
What program activities were accomplished?What program activities were accomplished?
Were they implemented as planned?Were they implemented as planned?
Outcome EvaluationOutcome Evaluation
Is the program producing the intended Is the program producing the intended outcomes?outcomes?
Is there progress toward program objectives and Is there progress toward program objectives and goals?goals?
20
Focus of the Evaluation
Evaluation QuestionsEvaluation Questions
Based on the needs of your Based on the needs of your stakeholdersstakeholders
Address process and outcome Address process and outcome
Assess Your QuestionsAssess Your Questions
Feasible to collectFeasible to collect
Provide accurate resultsProvide accurate results
21
Focus of the Evaluation
Key Issues in Evaluation DesignKey Issues in Evaluation Design
Will you have a comparison or control Will you have a comparison or control group?group?
When will you collect data?When will you collect data?
Will the data be collected retrospectively Will the data be collected retrospectively or prospectively?or prospectively?
What type of data do you need?What type of data do you need?
What data do you have already?What data do you have already?
22
Focus of the Evaluation
Other Design ConsiderationsOther Design Considerations Standards for “good” evaluationStandards for “good” evaluation
TimelinessTimeliness
Stage of developmentStage of development
Data neededData needed
Strengthen Your DesignStrengthen Your Design Mix methods whenever possibleMix methods whenever possible
Use repeated measuresUse repeated measures
TriangulateTriangulate
23
Gathering Credible Evidence: Data Collection
Identify indicators, standards, and data Identify indicators, standards, and data sources to address evaluation questions. sources to address evaluation questions.
IndicatorsIndicators Visible, measurable signs of program performanceVisible, measurable signs of program performance Reflect program objectives, logic model and Reflect program objectives, logic model and
evaluation questionsevaluation questions
Program Benchmarks and TargetsProgram Benchmarks and Targets Reasonable expectations of program performanceReasonable expectations of program performance Benchmarks against which to measure Benchmarks against which to measure
performanceperformance
Reference: Table 4 in your Evaluation Plan Guide
24
EvaluationEvaluation
QuestionQuestion
Process and Outcome Process and Outcome IndicatorsIndicators Program BenchmarksProgram Benchmarks
Have Spanish-speaking persons been treated appropriately for LTBI or TB?
Number of Spanish-speaking persons treated by clinic for TB & LTBI between Jan – June.
Increase in the number of Spanish-speaking patients
Number of times clinical treatment standards are met for Spanish-speaking patients
Clinical standards are met 100% of time
Percent of time that signs and forms are available in Spanish and written for persons with low-literacy skills
Patient education signs and forms in Spanish are available 100% of time; literacy level of materials is at a 3rd grade reading level
Linking evaluation questions, indicators and program benchmarks. Linking evaluation questions, indicators and program benchmarks. Example from the Guide – Table 4.Example from the Guide – Table 4.
Gathering Credible Evidence: Data Collection
25
Gathering Credible Evidence: Data Collection
Data CollectionData Collection Where are the data? Where are the data?
What methods will be used to collect What methods will be used to collect data?data?
How often will the data be collected?How often will the data be collected?
Who will collect the data?Who will collect the data?
Tools for Data CollectionTools for Data Collection Collect only the information you need Collect only the information you need
Easy to administer and useEasy to administer and use
Reference: Table 5 in your Evaluation Plan Guide
26
Table 5. Data Collection PlanTable 5. Data Collection Plan
IndicatorIndicator Data SourcesData Sources CollectionCollection
WhoWho WhenWhen HowHow
Gathering Credible Evidence: Data Collection
Linking indicators and data sources and specifying Linking indicators and data sources and specifying your data collection plan. Example from the Guide – your data collection plan. Example from the Guide – Table 5.Table 5.
27
Gathering Credible Evidence: Data Collection
Human Subjects ConsiderationsHuman Subjects Considerations
Evaluation TimelineEvaluation Timeline Ensures that all stakeholders are aware of what Ensures that all stakeholders are aware of what
activities are occurring at any timeactivities are occurring at any time
Helps to determine if your evaluation resources will be Helps to determine if your evaluation resources will be strained by too many activities happening at oncestrained by too many activities happening at once
Data Management and StorageData Management and Storage Ensures confidentiality and data qualityEnsures confidentiality and data quality
Reference: Table 6 in your Evaluation Plan Guide
28
Justifying Conclusions:Analysis and Interpretation
Once the data are collected, analysis and Once the data are collected, analysis and interpretation will help you understand what the interpretation will help you understand what the findings mean for your program.findings mean for your program.
AnalysisAnalysis What analysis techniques will you use for each data What analysis techniques will you use for each data
collection method?collection method? Who is responsible for analysis?Who is responsible for analysis?
InterpretationInterpretation What conclusions will you draw from your findings?What conclusions will you draw from your findings? How will you involve stakeholders?How will you involve stakeholders?
Reference: Table 7 in your Evaluation Plan Guide
29
Ensuring Use and Sharing Lessons Learned: Reporting and
Dissemination
A plan for dissemination and use of the evaluation A plan for dissemination and use of the evaluation findings will avoid having evaluation reports “sit findings will avoid having evaluation reports “sit on the shelf.”on the shelf.”
DisseminationDissemination What medium will you use to disseminate findings?What medium will you use to disseminate findings?
Who is responsible for dissemination?Who is responsible for dissemination?
UseUse How, where, and when will findings be used?How, where, and when will findings be used?
Who will act on the findings?Who will act on the findings?
Reference: Table 8 in your Evaluation Plan Guide
30
Tips for Evaluation Planning
Start small – focus on one initiative or program Start small – focus on one initiative or program component to start with and limit the number of component to start with and limit the number of evaluation questionsevaluation questions
Use what you already know about the programUse what you already know about the program
Consider existing sources of dataConsider existing sources of data
Be realistic in your timeline and assessment of Be realistic in your timeline and assessment of resources resources
Use the template and tables provided in the guide, Use the template and tables provided in the guide, adapt as neededadapt as needed
Seek help with your evaluationSeek help with your evaluation
31
Evaluation ResourcesSome Web-Based ResourcesSome Web-Based Resources
Centers for Disease Control and Prevention: Centers for Disease Control and Prevention: http://http://www.cdc.gov/evalwww.cdc.gov/eval//
W.K. Kellogg Foundation: W.K. Kellogg Foundation: http://http://www.wkkf.org/Publications/evalhdbkwww.wkkf.org/Publications/evalhdbk//
University of Wisconsin Extension: University of Wisconsin Extension: http://http://www.uwex.edu/ces/pdante/evaluat.htmwww.uwex.edu/ces/pdante/evaluat.htm//
Selected PublicationsSelected Publications
Connell JP, Kubisch AC, Schorr LB, Weiss, CH. Connell JP, Kubisch AC, Schorr LB, Weiss, CH. New Approaches to New Approaches to Evaluating Community Initiatives, Evaluating Community Initiatives, New York, NY: Aspen Institute, New York, NY: Aspen Institute, 1995.1995.
Patton MQ, Patton MQ, Utilization-focused Evaluation, Utilization-focused Evaluation, Thousand Oaks, CA: Sage Thousand Oaks, CA: Sage Publications, 1997.Publications, 1997.
Rossi PH, Freeman HE, Lipsey MW. Rossi PH, Freeman HE, Lipsey MW. Evaluation: A Systematic Evaluation: A Systematic Approach. Approach. Newbury Park, CA: Sage Publications, 1999.Newbury Park, CA: Sage Publications, 1999.
Taylor-Powell E, Steele S, Douglas M. Taylor-Powell E, Steele S, Douglas M. Planning a Program Planning a Program Evaluation. Evaluation. Madison, Wl: University of Wisconsin Cooperative Madison, Wl: University of Wisconsin Cooperative Extension, 1996.Extension, 1996.