19
Selecting the Right Evaluation Method

Selecting the Right Evaluation Method. Objectives Why should we evaluate? Which activities should we evaluate? When should we evaluate? How should we

Embed Size (px)

Citation preview

Selecting the Right Evaluation Method

Objectives

• Why should we evaluate?

• Which activities should we evaluate?

• When should we evaluate?

• How should we evaluate (methods)?

Why Do We Evaluate?

• Design a program. (or item; e.g., curriculum, poster)Baseline, formative

• Determine if objectives were met. (best practices)Outcome, process, impactBuild an evidence base for ACSM

• Re-design or refine a program. (lessons learned)Process, outcome

Which Activities Should We Evaluate?

Is this a new activity?

Is this an activity that may be expanded?

Have we invested a lot in this activity?

Is something unexpected happening?

Does the funder want an evaluation?

IF YES ….. EVALUATE!

Case Example

Is this a priority program for evaluation?

Why or why not?

Which objectives or activities are most

important to evaluate?

BASELINE

FORMATIVE

PROCESS

OUTCOME

IMPACT

When do we evaluate?

FORMATIVE EVALUATION

PROCESS EVALUATION

OUTCOME EVALUATION

IMPACT EVALUATION

ACSM ASSESSMENT

KAP Surveys

Key Informant Interviews

Media Scans

Focus Groups

Analysis of Program Data

Secondary Analysis

Simple Surveys

Client Exit Interviews

Teri Gilleland Scott
Replaced "Analyze" with "Analysis of".

When Can We Use Quantitative Methods?

BASELINE FORMATIVE PROCESS OUTCOME IMPACT

Routine Surveillance

Analysis of NTP Data

Analysis of Project Data

Simple Surveys

Population-Based SurveysPopulation-based

Surveys

Use a QUANTITATIVE Method to:

• Count things or people.

• Measure your program against a benchmark.

• Monitor trends over time.

• Statistically correlate factors: Is there a connection between variables?

When Can We Use Qualitative Methods?

BASELINE FORMATIVE PROCESS OUTCOME IMPACT

Focus Groups

Exit Interviews

Media Scans Media Scans

Informant Interviews

Exit Interviews

Use a QUALITATIVE Method to:

• Find out overall concerns, opinions.

• Gather initial information or sense of direction.

• Get details of/reasons for any problems.

• Test communication materials.

Combine BOTH for Best Results:

• Identify key issues with a focus group or interviews.

• Analyze NTP data to look for correlates or to obtain a baseline measurement.

• Use surveys to measure before and after results.

• Conduct interviews to assess how your process is going.

• Link your final outcome data to surveillance or NTP data.

Which Method and When?

BASELINE FORMATIVE PROCESS OUTCOME IMPACT

Analysis of Routine Surveillance

Focus Groups and In-Depth Interviews

KAP KAP

Exit Interview

Exit Interview

Simple Survey

Media Scan

Media Scan

Factors to Consider

What is the purpose of the evaluation?

I am at the end of a three-year, district-level ACSM project. Did we produce the expected outputs and short-term outcomes?

• Analysis of project data, focus group, or KAP survey?

I am designing a media campaign to inform my workers that TB treatment is free. What is the best way to get this message to them? What do they already know?

•Routine surveillance, KAP survey, or focus group?

What is the most rigorous method to answer the evaluation question?

The NTP wants an impact evaluation of ACSM because these new strategies have never been used at the national level.• Experimental design, secondary data analysis, exit interviews?

I am looking for a successful ACSM model that I can replicate in my district. • Case study, KAP survey, experimental design?

We trained 50 DOTS nurses on interpersonal communication and counseling skills and we want to know if they are using them correctly.• Time location survey, exit interviews, secondary data analysis?

How quickly do you need the results?

The NTP just announced funding for ACSM to reduce default. I have to design an activity in six weeks! I need to know why defaults are high in my district.

• KAP survey, simple survey, analyze surveillance data?

We have Global Fund money to implement a whole “package” of ACSM interventions to improve case detection over five years. We really need to know if the activities are effective!

•Research study, informant interviews, KAP survey?

What financial and human resources do I have?

I am SO lucky! My donor wants a formal evaluation of ACSM and gave us a lot of money for it. AND we have partners at the School of Public Health to help us!

•KAP survey, routine surveillance, research study?

My donor thinks that radio/TV/newspapers are the best way to reach our population and is giving me a lot of money to improve coverage of TB issues. But they want all of the money to go toward implementation. How can I know if it is effective?

•KAP survey, media scan, focus group?

Evaluation Template

ACSM Objective:

ACSM Activity:

Evaluation Question

Evaluation Category/Method

When or How Often

SamplingResources

NeededPlan for the Results