Evaluation in Health Promotion

Presentation by Irving Rootman to SFU Class on Principles and Practices of Health Promotion

November 1, 2010

Description of Evaluator

“The evaluator counts the ants at the picnic of progress” (Mohan Singh)

Outline

What distinguishes evaluation in health promotion from evaluation in other fields?

RE-AIM Framework

Definition of Evaluation in Health Promotion (EWG, 2001)

“Evaluation is about the systematic examination and assessment of features of a programme or other intervention in order to produce knowledge that can be used by stakeholders for a variety of purposes”

Principles for Evaluation of Health Promotion Initiatives (EWG, 2001)

Participation Appropriateness Multiple methods Capacity-building

Conclusion-Recommendation(EWG, 2001) Those who have a

direct interest in a health promotion initiative should have the opportunity to participate in all stages of its planning and evaluation

Encourage the adoption of participatory approaches to evaluation that provide meaningful opportunities for involvement of all those with a direct interest

Types of Participatory Research (Green et al., 1997) Participatory Action

Research (PAR)

Participative Research

Collaborative Inquiry

Participatory Rural Appraisal (PRA)

Appreciative Inquiry

Dialectical Research

Conscientizing Research

Emancipatory Research

Participatory Learning Research

Empowerment Evaluation

Definition of Participatory Action ResearchDefinition of Participatory Action Research

Systematic investigation…

Actively involving people in a co-learning process…

For the purpose of action conducive to health**

--not just involving people more intensively as subjects of research or evaluation

*

(Green et al., 1997)

To generate knowledge about persons without their full participation in deciding how to generate it, is to misrepresent their personhood and to abuse by neglect, their capacity for autonomous intentionality. It is fundamentally unethical.

Heron, J. (1996) Co-operative Inquiry, London, Sage

Conclusion-Recommendations(EWG, 2001) The use of randomized

control trials to evaluate health promotion initiatives is in most cases inappropriate, misleading and unnecessarily expensive

Support the use of multiple methods

Support further research into the development of appropriate approaches to evaluating health promotion initiatives

Additional Conclusions about Evaluation in Health Promotion (EWG, 2001)

1. Evolving

2. Valuable

3. Not enough

4. Many planning models

5. Multi-disciplinary

6. Evaluators play many roles

7. Theory is essential

Dimensions of RE-AIM Model (Glasgow, 2004) Reach: What % of potentially eligible participants will take part and how

representative are they? (Individual Level)

Effectiveness: What impact did the intervention have? (Individual Level)

Adoption: What % of settings and intervention agents will participate and how representative are they? (Setting Level)

Implementation: To what extent are the intervention components delivered as intended? (Setting/Staff Level)

Maintenance: What are the long-term effects (Individual Level); To what extent are intervention components continued or institutionalized? (Setting Level)

(Glasgow& Linnan, 2008)

Example of Use of RE-AIM Model: Diabetes Self-Management Projects Both used SCT and Social Ecological Model,

similar measures, adult P.C. diabetes patients, similar recruitment methods

Differed in intensity, implementation and other features

Example of Use of RE-AIM Model (Cont.)Program 1: In-Office Self-Management Touch-screen computer program prior to office visit

Followed by R.N. review of action plan

Follow-up phone call

Took 30-45 minutes; 5minutes for R.N.

Used regular staff in 30 clinics in Colorado

Example of Use of RE-AIM Model (Cont.)

Program 2: Linked Health Coach S.M. 2 two- hour visits to health educator

Patient worked through problem-solving , computer-administered program to produce action plan

Example of Use of RE-AIM Model (Cont.)

Comparison of Programs: Program 1 produced better “reach” Program #2 produced slightly less

improvement on P.A. and H.E. but larger QOL Change

Largest difference in “adoption” with 20% of MD’s willing to participate in #2 v.s. 6% in #1

Both produced excellent “implementation”

Example of Use of RE-AIM Model (Cont.)Comparison (Cont.): #1 cost $222, #2 $547 per participant Both improved self-efficacy, perceived support in

comparison to controls Few relations between mediators and outcomes

Conclusions: Likely more health plans would adopt #1 because of

lower cost and greater cost-effectiveness Lower adoption rates of #1 by MD’s needs to be

addressed

“Reach” Challenges and Remedies (Glasgow and Linnan, 2008)

Challenge: Sampling

Remedies: Population-Based

Recruitment

Over-recruitment

Report on representativeness

Limit exclusion criteria

“Effectiveness” Challenges and Remedies (Glasgow and Linnan, 2008)

Challenges: Understanding

Outcomes Knowledge of mediators

Conflicting/ambiguous results

Inadequate control conditions

Remedies: Assess broad set of

outcomes Include mediator

measures Sub-group analyses

Design control condition to fit question

“Adoption” Challenges and Remedies (Glasgow and Linnan, 2004)

Challenges: Program only studied in

optimal conditions

Program not adopted

Remedies: Involve potential

adoptees

Approach settings early

“Implementation” Challenges and Remedies (Glasgow and Linnan, 2004)

Challenges: Protocols not delivered as

intended

Unable to answer questions about cost, time or staff requirements

Deciding if program adaptation is good or bad

Remedies: Assess nature of treatment Involve practitioners in

program design

Vary staff characteristics, and evaluate staff impact and costs

Specify critical theoretical components and elements that can be adapted

“Maintenance” Challenges and Remedies (Glasgow and Linnan, 2004)

Challenges: Program effects not

maintained

Attrition of settings, delivery staff and participants

Remedies: Include maintenance

phase in plan

Plan for institutionalization and sustainability

Take steps to evaluate and report on attrition

RE-AIM: Strengths/Weaknesses

Strengths: Comprehensive Focus on population impact Multi-level Relatively easy to use Useful for program and policy development Builds on other theories at different levels

Limitations: Not much research using model Not necessarily “participatory”

RE-AIM: Implications for Aging

Has been used in studies of older adults in relation to: Physical Activity Chronic disease management Nutrition Heart Disease prevention

Example: Home-based Exercise

Purpose: Evaluate home-based exercise program for older adults

Methods: 105 frail homebound older adults recruited from 10 Faith in Action Sites; Volunteer trainers assisted subjects; Surveys at baseline and after 4 months in program; RE-AIM used as conceptual framework; focused on “adoption” and “implementation”

Findings: Participants exercised average of 2.2. times per week; improved social functioning

Conclusion: Evidence for A and I components of modelEtkin, C.D. et al., 2006

References Etkin, E.D., et al. (2006). Feasibility of Implementing the Strong

for Life Program in Community Settings, The Gerontologist, 46: 284-292.

Glasgow, R.E. and Linnon, L.A. (2008). Evaluation of Theory-Based Interventions. In Glanz, et al., Health Behavior and Health Education: Theory, Research and Practice, Forth Edition, San Francisco: Jossey-Bass.

Green et al., Participatory Research…Ottawa: Royal Society of Canada, 1997. www.lgreen.net/guidelines.html

Rootman, I., Goodstadt, M., Hyndman, B., McQueen, D., Potvin, L., Springett, J., Ziglio, E. (Eds.),(2001). Evaluation in Health Promotion: Principles and Perspectives, Copenhagen: European Regional Office of the World Health Organization.