TM

Low Cost and No Cost Tools for Program Evaluation

Victor Balaban, PhDProgram Evaluation Team (PET)

Field Services and Evaluation Branch (FSEB)

Division of Tuberculosis Elimination (DTBE)

NCHHSTP/CDC

TM

Disclaimer

• The contents and conclusions in this presentation have not been formally disseminated by CDC and should not be construed to represent any agency determination or policy.

TM

Evaluation

• Evaluation is:• An activity that assists in planning and

measuring programs

• a way of managing, improving and being accountable for:

•resources•activities•results

• Evaluation answers the question- “Is the program doing what we intend it to do?”

TM

What Can Be Evaluated?

• Direct service interventions

• Community mobilization efforts

• Research initiatives

• Surveillance systems

• Policy development activities

• Outbreak investigations

• Laboratory diagnostics

• Communication campaigns

• Infrastructure-building projects

• Training and educational services

• Administrative systems

Source: MMWR, 1999, Framework for MMWR, 1999, Framework for Program Evaluation in Public HealthProgram Evaluation in Public Health

TM

Why is Evaluation Important?• Improve knowledge of program design• Improve program implementation• Reporting• Ensure that a program reaches those who need

it most • Give visibility to work• Demonstrate accountability • Share information• Enhance understanding of what works best and

what does not work – and why

TM

Low Cost and No Cost Tools for Program Evaluation

• When designing an evaluation, there are often existing tools that can be used

• Many were not designed for evaluating TB programs and so will need to be modified

• Focus on evaluation instruments and tools for data collection and analysis, not guidelines and trainings

TM

CDC Resources

TM

Find TB Resources

• http://www.findtbresources.org/programevaluation.aspx

• Resources:• Links to tools and resources related to

program evaluation

TM

CDC - Office of the Associate Director for Program (OADPG)

• http://www.cdc.gov/eval/resources/index.htm#datacollection

• Resources:• Step by Step Manuals• Manuals/Assistance with Specific Evaluation

Steps:• Logic models• Data Collection Methods and Sources

• Websites Offering Comprehensive Evaluation Resources and Assistance

TM

Evaluation Guidance Handbook: Strategies for Implementing the

Evaluation Guidance for CDC-Funded HIV Prevention Programs

• http://www.cdc.gov/hiv/topics/evaluation/health_depts/guidance/strat-handbook/appendix.htm

• Instruments:• Intervention plan worksheets• Data collection forms

TM

CDC - Healthy Youth! Program Evaluation

Strategic Planning Tools & Resources• http://www.cdc.gov/healthyyouth/evaluat

ion/sp_toolkit.htm#3• Strategic Planning Kit for School Health

Programs:• Data Sources Matrix• Identifying Stakeholders Worksheet• SWOT Worksheet for School Health Programs• Program Strategy Checklist• Program Communication Tracking Tool• Data Collection Framework

TM

CDC - Healthy Youth! Handbook for Evaluating HIV Education

• http://www.cdc.gov/HealthyYouth/publications/hiv_handbook/index.htm

• Instruments:• HIV and AIDS knowledge related to HIV-risk behaviors • Attitudes toward people with AIDS • Attitudinal dimensions related to HIV-risk behaviors • Confidence in one's ability to resist peer pressures • Perceptions of peers' behaviors and values• Intentions regarding HIV-risk behaviors and their

precursors

• HIV-risk behaviors

TM

TB-PEN Resources

TM

TB-PEN – Tools Committee

http://tbpen.pbworks.com/w/page/4124488/FrontPage

Instruments: (organized by National TB Objectives)

• Completion of Treatment

• TB Case Rates • Contact Investigations • Laboratory Reporting • Treatment Initiation • Sputum Culture

Conversion• Sputum Culture

Reporting • Data Reporting

• Recommended Initial Therapy

• Universal Genotyping • Known HIV Status • Evaluation of Immigrants

and Refugees  • Program Evaluation • Human Resource

Development Plan • TB Training Focal Points • Cohort Review  

TM

American Evaluation Association (AEA) Resources

TM

AEA – Online Resources

• http://www.eval.org/Resources/instruments.asp• Links to collections of evaluation instruments

• http://www.eval.org/Resources/surveylinks.asp• Links to companies that provide software for the

development, distribution, scanning, and analysis of surveys/questionnaires

TM

AEA 365

• http://www.aea365.org/blog/• A blog dedicated to highlighting Tips, Tricks,

Resources, and Lessons Learned for evaluators

TM

Other Resources

TM

Children, Youth and Families Education and Research Network (CYFER)

• http://www.cyfernet.org/index.php?c=6• Instruments for:

• Process Evaluation - populations served, satisfaction, program accountability, etc.

• Program Sustainability• Organizational Capacity

TM

UCSF - Center for AIDS Prevention Studies (CAPS)

• http://caps.ucsf.edu/research/survey-instruments/

• Instruments for:• Evaluation• Treatment adherence• Focus group guides

TM

Innovation Network – Point K Learning Center

• http://www.innonet.org/index.php?section_id=4&content_id=16

• Instruments:• Organizational Assessment Tool• Logic Model Builder• Evaluation Plan Builder

TM

Penn State Extension: Program Evaluation

• http://extension.psu.edu/evaluation/examples.html

• Resources:• Evaluation tipsheets that answer questions

about program evaluation• Evaluation examples, including selections

from instruments

TM

University of Wisconsin - Extension

• http://www.uwex.edu/ces/pdande/evaluation/evalinstruments.html

• Instruments:• Community Organization Assessment Tool• Program Planning Questionnaire

TM

The Measurement Group• http://www.themeasurementgroup.com/

evaluationtools/evaluation_tools.htm• Instruments

• General Purpose Instruments and Evaluation Modules Evaluation of the HRSA SPNS Innovative Models of HIV/AIDS Care

• Evaluation Report Forms for the JAHF Institute for Geriatric Nursing• General Purpose Instruments and Specialized Evaluation Materials

for Evaluation of the HRSA SPNS Adolescent HIV/AIDS Care Models• Metro DC Collaborative for Women with HIV/AIDS Needs

Assessment Instruments• Evaluation Instruments for the HIFY Empowered Youth Evaluation

(EYE) Model

TM

Wilderdom

• http://www.wilderdom.com/tools/ToolsIndex.html

• Instruments• List of 30+ tools for assessing the psycho-social

impacts of intervention programs e.g. program quality surveys

• Summary reviews of the main features of each instrument, with overall rating and links to further information.

• Links to the major sites on the web for finding more research instruments.

TM

DoView

• http://www.doview.com/• DoView is a software program that enables

evaluators to create logic models

TM

Data Collection and Analysis

TM

Data Collection and Analysis

• CDC Resources• Epi-Info• EZ-Text

• Google Resources• Google Analytics• Google Refine

TM

Epi-Info• http://wwwn.cdc.gov/epiinfo/index.htm

• Software for the rapid creation of quantitative data collection instruments, analyzing and visualizing data, and generating reports

• Creates questionnaires and data entry forms• Automatically creates a database from the

questionnaire • Reads and analyzes data into statistics,

tables, graphs, and maps • Generates reports

•

TM

EZ-Text• http://www.cdc.gov/hiv/topics/

surveillance/resources/software/ez-text/index.htm• Software program to assist in creating,

managing, and analyzing qualitative data• Interview responses are entered into EZ-

Text as a verbatim transcript (e.g., from a tape recording), or a summary generated from the interviewer's notes

TM

EZ-Text (cont.)• http://www.cdc.gov/hiv/topics/

surveillance/resources/software/ez-text/index.htm• Users can investigators can interactively

create on-line codebooks, apply codes to specific response passages, develop case studies, conduct database searches to identify text passages, and export data for further analysis with other qualitative or statistical analysis software programs

TM

Google

• Google Analytics• http://www.google.com/analytics/index.html

• A free program that help to analyze traffic to websites and measure the success of social media programs

• Google Refine• http://code.google.com/p/google-refine/

• for data cleanup and transformation to other formats to help clean messy data

TM

Conclusion

• Using and adapting existing instruments can save time and resources when conducting evaluations

• DTBE/FSEB/PET is available to provide information to help you identify and use evaluation tools

TM

Thank you

TM

Acknowledgements• DTBE/FSEB/

Program Evaluation Team (PET)• Awal Khan• Judy Gibson• Lakshmy Menon• Brandy Peterson

• DTBE/FSEB/Field Services Teams I & II• Dan Ruggiero• Gail Burns-Grant• Vincent Fears• Alstead Forbes• Regina Gore• Vernard Green• Andy Heetderks• Mark Miner• Dawn Tuckey