Mehmet Dali Öztürk, Ph.D.

mehmeto@cos.edu

Layers of Evaluation & Meaningfulness of Results

3CSN and ELI Collaborate: Data Collection Workshop

September 6, 2013

Least Meaningful

More Meaningful

Most Meaningful

Output Analysis

Outcome Analysis

Impact Assessment

Meaningfulness of ResultsLayers of Evaluation

Layers of Evaluation• Impact assessment focuses on determination of

changes in outcomes that can be attributed to polices, programs, initiatives and projects for schools, individuals, families, or communities. For example, the change in school graduation rates due to a specific program in comparison to schools who did not receive the same program.

• Outcomes focus on intended and expected results of

polices programs, initiatives and projects. For example, the percent increase in schools meeting state standards.

• Outputs are the types and quantities of

engagements and efforts. For example, the number of mentoring hours produced by program personnel.

Program X

Output Outcome ImpactNumber of students in

program

Number of students

completing program

Percent gain in standard test scores in math

Significant changes in percent gain in standard math test scores at the

focus schools where program is implemented

Percent gain in standard test

scores in reading

Significant changes in percent gain in standard test scores in reading at the focus schools where program is implemented

Percent of students

demonstrating a grade increase in math on report

cards

Significant changes in percent of students

demonstrating a grade increase in math on

report cards at the focus schools where program

is implemented

Percent of students

demonstrating a grade increase in

reading on report cards

Significant changes in percent of students

demonstrating a grade increase in reading on

report cards at the focus schools where program

is implemented

1. Are there differences in outcome variables?

2. Are the differences significant?

3. Are the differences meaningful?

4. What do these differences mean with regard to the overall community?

Providing a Context for the Evaluation Process

6

Program Effectiveness/Scale Rating System, which was developed to be a tool to help policy-makers better understand the impact of programs, strategies, and innovations.

Program Effectiveness Scale/Rating System

By placing evaluation results into a format that is easy to understand, the Program Effectiveness Scale can help policy-makers make objective judgments on the worth of a program or strategy.