Program Development and Evaluation (David Diehl, Ph.D.)

Program Development and Evaluation: A View from On High Nonprofit Summit – June 30, 2011

David C. Diehl, Ph.D. Assistant Professor, Program Planning and Evaluation

University of Florida

Does your job ever feel like this?

David C. Diehl, Ph.D. B.S. Psychology, University of Wyoming

Ph.D. Human Development, Cornell University

Dissertation: Early Reading Project with Head Start Families

Current Projects:

– Under One Sky: Evaluation of Youth Foster Care and Adoption Program

– United Way Strengthening Families: Process Evaluation of Family Strengthening/Protective Factors with Local United Ways

– Specialty Crop Research Initiative: Evaluation of Effort to Get Better Tasting Fruits to Supermarkets

– Climate Change Project: Evaluation of Effort to Educate Farmers on Climate Change

– Cooperative Extension: Evaluation of Educational Outreach Efforts throughout Florida (financial education, nutrition, etc.)

3 Children, 1 Wife… (10 years of wedded bliss)

Feature model for Wii character at the right…

The important thing is not to stop questioning. Curiosity has its own reason for existing.

- Albert Einstein

Always start with questions.

- David Diehl

Why do you do the work you

do?

What are your greatest

evaluation challenges?

Where can you have the greatest impact?

• What is your passion?

• What do you love?

• What are your gifts/skills?

• When are you the most creative?

When you think

of „evaluation,‟

what are the

first words or

images that

come to mind?

Stream of

Consciousness (Do NOT think)

Some words I

would like you to

start thinking of…

Learning

Creativity

Usefulness

Under My Control

Interesting

and so on…

Why plan programs?

If it were easy, it would

already be done…

What was the US

poverty rate in 2010?

What was the US

poverty rate in 1968?

What was the US

poverty rate in 1959?

14.3%

12.8%

22.4%

Blacks 55% in 1959

Whites 22% in 1959

“It‟s been four decades

since Bobby Kennedy

crouched in a shack along

the Mississippi Delta and

looked into the wide,

listless eyes of a hungry

child. And when Kennedy

turned to the reporters

traveling with him, with

tears in his eyes, he asked a

simple question about

poverty, „How can a country

like our allow it?‟”

Why evaluate programs?

Outcomes-Focused Planning

1. Understand the Situation

2. Define Participant Outcomes

3. Identify “What Works”

4. Create a Program Logic

Model

5. Implement the Program

6. Evaluate and Monitor Progress

7. Learn, Modify, and Improve

Key Principles

Measure, Learn, Improve, Communicate

Aim for Lasting Community Change

Uncover Underlying Issues

Be Specific about Intended Results/Outcomes

Engage the Right People for the Purpose at Hand

Base Decisions on Facts

Source: United Way Worldwide (2009). Outcomes-Focused Planning

Principle 1:

Base decisions on facts

“Numerous studies of Scared

Straight have demonstrated that

the program does not deter

future criminal activities. In

some studies, rearrest rates

were similar between controls

and youths who participated in

Scared Straight. In others,

youths exposed to Scared

Straight actually had higher

rates of rearrest than youths not

involved in this intervention.” - From “Youth Violence: A Report of the

Surgeon General”

We’re #1!! We’re #1!! (although we now have dropped to #7)

Suppose I wanted to create a program to address alcohol use at UF…

(who should I engage in the planning?)

Principle 2:

Engage the right people for the purpose at hand

Principle 3: Be specific about intended results/outcomes

“What gets measured gets done, (which) is a very powerful logic. If we measure something, it makes it important, people pay attention to it, you get it accomplished… The implication of this, however, is dramatic because it means if you measure the wrong thing, the wrong thing gets done.”

— Michael Patton

Principle 4: Uncover underlying issues

“It's so much easier to suggest solutions when you don't know too much about the problem.”

- Malcolm Forbes

If …., then …

If we have these

resources, then we will be able to carry out

these activities

If we carry out these activities,

then we will be able to reach the following people

If we reach these

people in these ways, then we will affect their learning/ attitudes.

If people are learning

and changing

their attitudes, then they

will change their

behavior

If people change their

behavior, social

conditions will also change

Resources/

Inputs

Outputs/

Activities

Short-Term

Outcomes

Long-Term

Outcomes

Intermediate

Outcomes

Assumptions

External Factors

Principle 5: Aim for lasting community change

http://makerealchange.org/

“Experts in his field had figured out how to educate one disadvantaged child, or one classroom full of kids, but the benefits were localized, and usually temporary. And no one had any idea how to change a whole school system or a whole housing project, or for that matter a whole neighborhood. So, in the middle of the 1990's, that's what Geoffrey Canada decided he would do. And now, 10 years later, he has become a very different kind of do-gooder, one with a mission both radically ambitious and startlingly simple. He wants to prove that poor children, and especially poor black children, can succeed -- that is, achieve good reading scores, good grades and good graduation rates -- and not just the smartest or the most motivated or the ones with the most attentive parents, but all of them, in big numbers.”

- New York Times Magazine

Principle 6: Measure, learn, improve,

communicate

http://learningstore.uwex.edu/assets/pdfs/G365

8-1W.PDF

“Being an evaluator is not easy. I’m not referring to the technical problems we face in our work, but to how people react to us and why. Telling someone that you’re an evaluator is like telling them you’re a cross between a proctologist and an IRS auditor. The news evokes a combination of fear, loathing, and disgust, mixed with the pity reserved for people who go where others don’t want them to go.”

-David Chavis, President of the Association for the Study and Development of Community

“Hi, I‟m Dr. Diehl, your proctologist.”

How far you go in life depends on your being tender with the young, compassionate with the aged, sympathetic with the striving, and tolerant of the weak and strong. Because someday in your life you will have been all of these.

- George Washington Carver