Arizona State University College of Nursing and Health InnovationLinda.larkey@asu.edu

Linda K Larkey, PhDScottsdale Healthcare Professor of Biobehavioral Oncology Research

overview

evaluation paradigms

what is an “evaluation theory”?

what does a theory buy you?

how do you build theoretical constructs into evaluation?

practice

positivist approach

assumes there are “objective”, observable and measurable aspects of a program

preference for predominantly quantitative evidence needs assessment assessment of program theory assessment of program process impact assessment efficiency assessment

(Rossi, Lipsey and Freeman, 2004)

three basic paradigms

positivist

Interpretive

critical-emancipatory

you don’t have to join a club--- feel free to blend

Potter, C. (2006). Program Evaluation. In M. Terre Blanche, K. Durrheim & D. Painter (Eds.), Research in practice: Applied methods for the social sciences (2nd ed.)

interpretive approaches

purports it is essential that the evaluator develops an understanding of the perspective, experiences, needs and expectations of all stakeholders

key words, “understanding” “meaning” “rich description”

considered crucial before one is able to make judgments about the merit or value of a program.

The evaluator’s contact with the program is important, over time observation Interviews focus groups

critical-emancipatory approaches based on action research for the

purposes of social transformation

particularly useful in developing countries or underserved populations

concerns: empowering populations to create change from within, providing “voice”

so what?

it matters whether you think your goal is to achieve some measurable state or reality…

or, if you think that the process along the way and people’s opinions and perceptions are most important

or, if you see the program goals to empower, raise consciousness, give voice

What is the result expected? Typical logic: Process: We will reach X number of

female, single heads of household with food stamp information by….

Short-term outcome: We will distribute $X within Maricopa County by…

Longer-term outcome: We will increase the use of food stamps by $X

Impact: We will reduce food shortage by X%

what “causes” the desired end result? how would we… reduce food instability in our state

reduce flu epidemics

increase number of servings of F & V eaten by school age children in school lunch

increase cancer screening rates what else?

Pick one “impact” level goal What causes it? (Needs assessment,

what is the problem, what contributes?)

What do you think might fix it? One thing or many?

How would you know if it were fixed?

That is your Evaluation Theory Once you name the causes, purported

improvement strategies, and expected outcome, you have a LOGIC MODEL

Your construal of the problem (antecedents) is YOUR THEORY

This theory, PLUS your assumptions about what is important to change to get a result is your EVALUATION THEORY

Simpler “Logic Model”: ATM

Low rates of colorectal

cancer screening

Patients can’t access

or are discouraged

from accessing services

Long Waits for services

Facilities Lacking

Not enough trained providers

Lack of funding for uninsured

(screening/ treatment)

Inadequate Lobbying

(for policy and/or funding) Existing health

care plans inadequate

Cultural factors

Providers not making referralsProviders not clear

on guidelines (who/what/when/ri

sk)

Patients not demanding screening

Patients not aware of need/importance/valu

e of screening

New and complicated

message(not your average bumper sticker)

Lack of office reminder and/or flagging systems

Conflicting research evidence, that

screening saves lives, or best method

Uncomfortable topic (distaste for “things

scatological”)

Inappropriate Utilization

Competing priorities during patient/provider appointment: too little

time

Mixed/unclear financial impact (will referred

patients return? reimbursement?)

Such complex map of problems… what’s the solution?

Research Project DevelopmentAnd Evaluation

State/County DataPublic Messaging

Capacity for DeliveryCommunity-based Organizations

VolunteersCommunity Visibility

Legislative/PolicyDialogue Leadership

AZCC

(academic)

ADHS

(public

health)

ACS

(private)

Health Services Delivery

Figure 2. Transdisciplinary Model Approach for Increasing CRC Screening in Arizona

building your own evaluation plan… share example of one of your programs map “antecedent conditions”– what are

the causes? (theory of the problem) so, if those are the causes, what could

fix this problem? (theory of solution) what is the final outcome expected? then…

measure what along the way? At the end? things you can count? meaning/perception? what does the community say? Remember

the “critical” view

holding your vision while negotiating the terrain…