INSTITUTE OF MEDICINE Obesity Evaluation Toolkit: Resources for Evaluating Community-Level Obesity Prevention Efforts Webinar August 25, 2015 INSTITUTE

INSTITUTE OF MEDICINE

Obesity Evaluation Toolkit: Resources for Evaluating Community-Level Obesity

Prevention EffortsWebinar

August 25, 2015



2

1. Introduction

Leslie Sim, M.P.H. Institute of Medicine of The National Academies of Science, Engineering, and Medicine

2. Overview of report, Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress

Lawrence W. Green, Dr. P.H., M.P.H. University of California, San Francisco


3

3. Steps in evaluating community-level obesity prevention efforts

Stephen Fawcett, Ph.D.Work Group for Community Health and Development, University of Kansas

4. Web-based resources to support your efforts

Christina Holt, M.A.Community Tool Box, Work Group for Community Health and Development, University of Kansas

5. Questions


4

Submit questions - [email protected]

Issues connecting? - 1-866-770-8162


5

Evaluating Obesity Prevention Efforts:

A Plan for Measuring Progress

Lawrence W. Green, Chair; and 13 members of

The Committee on Evaluating Progress of Obesity Prevention Efforts


6

Committee on Evaluating Progress of Obesity Prevention Efforts


7

AN URGENT

NEED FOR

EVALUATION


8

What we need to know from evaluation…

• Where are we in making progress and with whom? (current status)

• How are we doing in making progress? (trends over time in assessing needs and implementation of policies and strategies)

• What works in which populations?• What are the unintended consequences?


9

Committee on Evaluating Progress of Obesity Prevention Efforts

Study Charge: “to develop a concise and actionable plan for

measuring progress in obesity prevention efforts for the nation and adaptable

guidelines for community assessments and evaluation.”


10


11

The Concise and Actionable Plan

• Enhanced Leadership• Enhanced Roadmap• Enhanced Capacity and Infrastructure

Included resources: • Indicators for the evaluation plans• Tools and methods for assessing progress

in populations at greater risk for obesity• Community health assessment,

surveillance, and monitoring of interventions


12

Interdependence of National and Community Obesity Evaluation

Plans

Community Obesity

Evaluation Plan

National Obesity

Evaluation Plan

Core indicators, Data sources, Resources,

Methodologies

Contextual data, Feasibility, Local

innovation


13

Recommendations

1. Improve Leadership and Coordination

2. Improve Data Collection

3. Provide Common Guidance

4. Improve Access to and Dissemination of Information

5. Improve Workforce Capacity

6. Address Disparities and Health Equity

7. Support a Systems Approach


14

To access the full report and related dissemination materials(url: http://iom.nationalacademies.org/Reports/2013/Evaluating-Obesity-Prevention-Efforts-A-Plan-for-Measuring-Progress.aspx)

4-page report brief

Pull-out summary of indicators

Interactive indicator widget


15

Indicators of Progress (Excerpt from Table 4-1)

Indicator Topic ObjectiveOVERARCHING/SYSTEM-LEVEL INDICATORSObesity-adult Reduce the proportion of adults who are obese (body mass

index (BMI) ≥ 30)

APOP GOAL AREA 1: PHYSICAL ACTIVITY ENVIRONMENT

Adult physical activity Increase the proportion of adults who meet current federal

physical activity guidelines for aerobic physical activity and for muscle-strengthening activity

…


16

National and State Level Indicators (excerpt, Table 6-4)


17

Community Level Indicators (excerpt, Table 7-2)

Indicator Topica Data Source Current Availability by Community Sizeb

Larger Smaller

Overarching/System-Level

Obesity-adult BRFSS

Overweight-adult BRFSS

Obesity-adolescent YRBSS, School reports

Goal Area 2: Food and Beverage Environment

Sugar-sweetened beverage policies in schools (school district)

SHPPS

Sugar-sweetened beverage consumption YRBSS (adolescent)

School policies to facilitate access to clean drinking water SHPPS

Note: Larger – population >50,000; Smaller – population < 50,000


18

Some Steps in Evaluating Community-Level Obesity

Prevention Efforts

Stephen Fawcett, Work Group for Community Health and Development, University of Kansas


19

Vision for IOM Report

Assure collection and analysis oftimely and meaningful data

to inform and improve obesityprevention efforts at national,

state,and community levels


20

CONTEXT for Evaluation

• Multi-component, multi-sector, and multi-level interventions

• Indicators/measures of varying quality & utility

• Varying capacity, capabilities, leadership, and resources


21

ACTIVITIES: Develop Resources

for Training, Technical Assistance,

and Dissemination


22

Steps/Components of a Community Evaluation Plan (Box

8-1)1. Design stakeholder involvement. 2. Identify resources for the monitoring and

summative evaluation.3. Describe the intervention’s framework,

logic model, or theory of change.4. Focus the monitoring and summative

evaluation plan.5. Plan for credible methods.6. Synthesize and generalize.


23

1. Design stakeholder involvement• Identify stakeholders

• Consider the extent of stakeholder involvement

• Assess desired outcomes of monitoring and summative evaluation

• Define stakeholder roles in the monitoring and summative evaluation


24

Participatory Evaluation (CBPR)


25

2. Identify resources for monitoring and summative

evaluation• Person-power resources• Data collection resources


26

Some Web-Based Data Sources(Table 7-5)

• Community Commons & CHNA.org

• County Health Rankings

• USDA Food Atlas

• CDC Diabetes Interactive Atlas

• Census ACS and County/Zip Business Patterns

• HHS Community Health Status Indicators


27

What indicators available at local level?Available for all/ many communities:

• Adult obesity/overweight

• Activity: Active transport by walking, bicycling, density of recreational facilities, leisure-time PA

• Nutrition: Adult F/V, food outlet density, farmers’ market density, food deserts, SNAP/WIC store density

Some larger communities also have:• Youth obesity/overweight

• Activity: Youth PA, screen time, school PA participation

• Nutrition: SSB consumption, youth F/V


Data Availability (Table 7-2 – Example)

Large Small


29

3. Describe the intervention’s framework or logic model, or

theory of change.• Purpose or mission• Context or conditions• Inputs: resources and barriers• Activities or interventions• Outputs of activities• Intended effects or outcomes


30

Generic logic model for community obesity prevention (Figure 8-1, adapted).


31

4. Focus the monitoring and summative evaluation plan.

• Purpose or uses: What does the monitoring and summative evaluation aim to accomplish?

• Set priorities by end-user questions, resources, context

• What questions will the monitoring and summative evaluation answer?

• Ethical implications (benefit outweighs risk)


32

End-User Focused Evaluation Questions—Some examples:

• How fully was intervention implemented?

• Did the intervention have desired effects?

• What was the impact on participants/ population?• With whom? • Under what conditions?


33

5. Plan for credible methods.

• Stakeholder agreement on methods• Indicators of success• Credibility of evidence


34

Some Emerging Methods forData Collection

Environmental change data• Documentation of initiatives• Unobtrusive observations• Secondary data (e.g., GIS)

Policy change data• Documentation of initiatives• Surveillance

Systems change data• Mapping changing relationships


35

Indicators of Success• Translate expected effects (logic model) into specific

measurable units • Examples include:

– Program Outputs—units of activities delivered– Intermediate Outcomes—changes in communities and

systems (program, policy, environment)– Behavioral Outcomes—changes in diet and physical

activity – Population-level Outcomes—reduced prevalence of

obesity


36

Evaluation Designs – Match to Goal/Context

Qualitative methods: interviews, focus groups, photo-voice, etc.


37

6. Synthesize and generalize.

• Disseminating and compiling studies• Learning more from implementation• Ways to assist generalization• Shared sense-making and cultural

competence• Disentangling effects of interventions


38

Logic Model Design and Shared Sensemaking

Obesity


39

Obesity Evaluation Toolkit:Web-based Resources

for Community Evaluation

CONTEXT: Distributed evaluation workforce • People we will never see• In places we will never be

TOOLKIT: Just-in-time resources for:• Training• Technical Assistance


40

Web-based resources to support your efforts

Christina Holt, M.A.Community Tool Box, Work Group for Community Health and Development, University of Kansas


41

Navigating to the Toolkit

The Obesity Evaluation Toolkit is available online:http://iom.nationalacademies.org/activities/nutrition/obesityprevprogress/resources-evaluating-community-level-obesity-prevention-efforts


42


43

What you will find


44


45


46


47

Evaluation Toolkit


48

Community Tool Box – Example Resource


49

Troubleshooting Guide


50


51


52


53


54


55


56


57

Questions/ Discussion

Documents

INSTITUTE OF MEDICINE Obesity Evaluation Toolkit: Resources for Evaluating Community-Level Obesity Prevention Efforts Webinar August 25, 2015 INSTITUTE