Blending the Theoretical and the Empirical in Evaluation Science:

A Case Example from Cancer Control/Prevention and a General Discussion

Ross ConnerUniversity of California Irvine USA and

Senior Adviser and Past President – International Organisation for Cooperation in Evaluation – IOCE

Workshop Outline

1. Evaluation Overview

2. The Theoretical Component

3. The Empirical Component

4. Case Example: Chinese-Korean Cancer Prevention/Control Program

5. Conclusions

6. Questions & Answers; General Discussion

1. Evaluation OverviewInputs Processes Outputs Outcomes Impacts

A simple example: program = a mathematics class• Inputs: students, teacher, lesson texts, paper and pencils, classroom

setting• Processes: lessons and exercises taught by teacher• Outputs: exercises completed by students• Outcomes (short-term): Students learn new concepts, have new

understandings (measures: tests)• Impacts (long-term): Students apply new concepts and

understandings, students advance to next level of learning For simplicity, this is set out as a linear track of 5 stages; in practice, it is

often more complex with multiple tracks because of different components of the program.

2. The Theoretical Component Theory from Evaluation Science

• Evaluation Science: bringing the tools of science and discovery to the assessment and evaluation of programs or policies.

• Tracking and monitoring inputs and processes; identifying and documenting outputs; measuring outcomes; tracking and measuring impacts.

Theory from the Program or Policy Area• Program or Policy Area: bringing the past learning from the area

to advance the area • Building upon past experimental and theoretical work in the

area, developing program components based on past research and practice.

Using both types of theories, the program components and the evaluation components are developed.

3. The Empirical Component

Program Design • Specify planned inputs and planned processes • Specify expected outputs, outcomes and impacts• Specify evaluation questions

Evaluation Design• Specify evaluation design(s) to answer the evaluation

questions. Design = the plan for data collection • Specify evaluation measures to obtain necessary data

4. Case Example

Area: Health – cancer control and prevention Populations: Chinese and Koreans in

California, USA Program focus: women’s cancers initially, later

men’s cancers Program aim: disease destigmatization

Overview: Chinese Program Components

Materials in Chinese language Large free luncheons with expert lecturers Free screenings for cancers Cancer survivors involved in the programs High visibility in community

Overview: Korean Program Components

Materials in Korean language Used networks of Korean Christian ministers to

destigmatize; cancer survivors featured Free screenings for cancers Prominent role at public events and fairs

The Program Inputs Processes Outputs Outcomes Impacts

Planned Inputs: materials in native language, large gatherings of community members, religious leaders (Korean), cancer experts (Chinese)

Planned processes: dissemination of materials and information to 4,000 people, cancer screenings, discussions among community members

Planned outputs: increased knowledge about cancer, increased understanding of consequences of disease stigmatization

Expected outcomes: increased (by 10%) cancer screenings, decreased disease stigmatization

Expected Impacts: 100 community cancer survivors serving as volunteers and outreach workers for the program

The EvaluationInputs Processes Outputs Outcomes Impacts

Examples of evaluation components: Inputs: tracked numbers of attendees at events Processes: observed community events – materials

distributed, lectures given, discussions held Outputs: assessed knowledge about cancer Outcomes: tracked cancer screenings, tracked decreased

disease stigmatization Impacts: tracked cancer survivors serving as volunteers for

the program

Selected Evaluation Results

Outputs and Outcomes Chinese Korean Totals

Health Education Sessions Conducted

(attendance)*

4,912 8,026 12,938

Community Outreach Workers Trained

124 27** + 30 181

Cancer Screenings Conducted+

2,212 1,739 3,951

* Estimated duplicate counts: 40%** Ministers+ Includes breast, uterus, liver, prostate and colon cancer screenings

5. Conclusions

The program and the evaluation blended theoretical components from evaluation science and from the area of health education/disease prevention.

The program set measurable objectives. These anchors provided a basis to judge achievements.

The evaluation involved empirical assessments of inputs, processes, outputs, outcomes and impacts.

The program planners/implementers from the Chinese and Korean communities were partners with the evaluation team, from start to finish.

The programs continue today, with a smaller funding base

6. Audience Inputs and Comments

Questions about the program Questions about the evaluation Questions about the results General questions about evaluation Any other general questions for

discussion

Evaluation: Next Steps

New book available soon:

Оценка программ: методология и практика

Редакторы: А.Кузьмин, Р.О’Салливан, Н.Кошелева