Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
1
A non-profit service and advocacy organization © 2011 National Council on Aging
Evidence-Based Programs 101 - Webinar Instructions
Thank you for joining today, please wait while others sign in.
The audio portion of this call will be heard through your
computer speakers.
Please make sure your speakers are on and the volume is
turned up!
You have joined the call in listen-only mode.
If you want to ask a question, please type your question into
the chat box.
Evidence-Based Programs 101 Tips for Selecting, Implementing, and Evaluating EBPs
Marcia Ory, PhD, MPH Ashley Wilson, MPH
Program on Healthy Aging Texas A&M Health Science Center
School of Rural Public Health
July 24th, 2012
Topic for Discussion
EBPs definitions and criteria
Advantages and disadvantages to EBPs
Steps in choosing an EBP for your organization
Texas Example
Program Evaluation Tips
Conclusions
Questions & Answers
Why the Recent Interest in EBPs?
Magnitude and seriousness of health and social problems in our communities
Awareness of preventability of many problems
Emergence of evidence-based practices and programs
Spend limited resources more efficiently
Our Assumptions
Most of you have probably delivered self-management programs to
adults.
Some of these programs have been ―evidence-
based.‖
What are Evidence-based Programs?
• Evidence-Based Programs are interventions based on evidence that is generated by scientific studies published in peer-reviewed journals.
• Model Programs are a type of evidence-based program
What are evidence-based programs?
• Previously implemented
• Found to make positive differences in the lives of participants
Evidence-based programs have been:
Tiered Set of Criteria for Defining EBPs Implemented Through OAA
FY2012 Congressional appropriations
require Title IIID funding to be
Evidence-Based
http://www.aoa.gov/AoARoot/AoA_Programs/HPW/Title_IIID/index.aspx
Highest-level Criteria
Dissemination products have been developed and are available to the public.
Full translation has occurred in a community site
Undergone Experimental or Quasi-Experimental Design
Adapted from Brownson et al, 2003
Type 1:
•Evidence that a health issue exists
Type 2:
•Evidence that a program is effective
Type 3: • Evidence about
design, context, and attractiveness of program
Three Types of Evidence
Target Population
Goals
Rationale
Benefits
Program Structure & Timeframe
Staffing
Facilities & Equipment
Program Evaluation
Fidelity Checklist
Elements Common to EBPs:
Practice Question
What are some advantages and disadvantages of evidence-based programs versus ―home grown‖ ones?
Perceived Disadvantages of EBPs
Requires knowing where to find and how to understand/judge the evidence
Feels like standardization of program rather than site-specific tailoring
Tools and processes are unfamiliar
Difficult to build community support—many prefer ―home grown‖ to ―off the shelf‖
Can be expensive
Online Training Modules: Evidence-based Health Promotion Programs for Older Adults
Increase in the likelihood of positive outcomes
Lead to efficient use of resources
Facilitate the spread of programs
Facilitate the use of common performance measures
Support continuous quality improvement
Help to establish partnerships
Make it easier to justify funding
Online Training Modules: Evidence-based Health Promotion Programs for Older Adults
Perceived Advantages to EBPs
How Does One Chose a EBP?
Choosing an EBP
Step 7: Sustain the program
Step 6: Evaluate the program
Step 5: Translate the intervention into a program
Step 4: Select an intervention
Step 3: Establish broad-based partnerships
Step 2: Identify effective interventions
Step 1: Identify an important health issue among the Seniors you serve
The tasks do not always follow a sequential progression. The list provides a guide, not a strict set of steps.
Step 1: Identify an important health issue among the Seniors you serve
Review health statistics and other data to identify relevant health conditions and risk factors
Ask questions of seniors in your constituency
Determine if the identified health issue affects all of the Seniors you serve or only a subset
Step 2: Identify effective interventions
National Council on Aging
• http://www.healthyagingprograms.org/content.asp?sectionid=32
National Cancer Institute
• http://rtips.cancer.gov/rtips/index.do
Substance Abuse and Mental Health Services Administration
• http://nrepp.samhsa.gov/Search.aspx
Aging Texas Well
• http://www.agingtexaswell.org/ebased/index.cfm
Partnering for Success
Identify organizations with a common mission
• Align goals
• Tap into the organizational cultures and core values of partners
Communicate
•Encourage honesty
• Clarify expectations
• Make sure everyone is given the same information
Work together
• Share praise, glory, and recognition
• Realize that no partner is perfect
• Think creatively
Share Resources
•Enhance participant recruitment
•Maximize program-to-program referrals and dissemination
•Share facilities and materials
•Cross train staff and volunteers
•Increase data collection and
•data analysis capacity
Step 3: Establish broad-based partnerships
Step 4: Select an intervention
Program Match
• Does the program fit your mission?
• Is leadership supportive of the program?
• Are clients interested in the program?
• Can you implement the program as designed?
Costs and Resources
• Is there a licensing fee?
• What type of facilitator is needed?
• Is facilitator training required?
• What equipment and supplies are needed?
• Is the program ongoing or periodic?
Cultural and Contextual Relevance
• Is the program culturally appropriate?
• Are materials written at the correct literacy level and in the correct language?
• Does the program have regional appeal?
Looking for Evidence-Based Programs
Based on best practices and principles
Subjected to rigorous research design
Program is efficacious on outcomes of interest
Program is manualized for replicability
Tested in multiple settings and populations
Published in peer review literature
Endorsed by professional body
Feasible to move to scale
Is Your Community Ready to Implement Evidence-Based Health Promotion Programs?
There is buy-in from leadership
There is access both to personnel with the expertise to do these programs and to the population that needs these programs
There is funding for the program
Agency/partner-ship is willing to do evidence-based health programs and stay true to the model being implemented
National Council on Aging Organizational Readiness Checklist
Step 5: Translate the intervention into a program
• The process of taking a program originally implemented in a controlled ―laboratory-like‖ setting and making it suitable for implementation in the community
Implementation Suitability
• The faithful and accurate adherence to the core elements of an intervention, must be maintained
• Preserves the evidence-base
Maintaining Fidelity
Step 5: Translate the intervention into a program (cont.)
• Program components that generated the positive outcomes
• Goal setting, identifying barriers, monitoring behavior change, etc.
Core elements
• Attributes that make the program ―fit‖ the population
• Size of fonts and reading level of materials, marketing techniques, etc.
• Key characteristics can be adapted, but core elements cannot be adapted (jeopardizes fidelity)
Key characteristics
Step 6: Evaluate the program
Use findings in future program planning
Evaluate program delivery and outcomes to assess program success
Monitor program operations and make adjustments to evaluation plan (if needed)
Create or find the instruments and protocols needed for data collection
Plan goals for process and outcome evaluation before program implementation
Step 7: Sustain the program
Elements of a sustainable infrastructure and delivery system
1 • Effective leadership
2 • Adequate delivery infrastructure
3 • Partnerships
4 • Centralized and coordinated logistical processes
5 • Business planning and financial sustainability
6 • Quality assurance and fidelity to interventions
Targets & Outcomes
Recruiting Partners and Participants
Quality Assurance and Fidelity
Challenges and Solutions
Texas Example
Texas Goals
Dates
April 1, 2010 –
March 31, 2012
Goals
Enroll 4,098 participants
72.5% Completion (2,975 Completers)
*Received extension through June 30th, 2012
Texas Healthy Lifestyles
Bexar AAA -San Antonio Area Central Texas AAA -Temple, Belton, Killeen East Texas AAA -45 counties Rio Grande AAA -El Paso Area Tarrant Count AAA -Fort Worth
Region Target Enrollees Target
Completers
Tarrant County 960 697
Central Texas 900 653
Rio Grande 888 645
East Texas Coalition 750 544
Bexar County 600 436
Total 4,098 2,975
Texas Targets by Region
Region Target Enrollees Target
Completers
Tarrant County 1,255 912
Central Texas 623 416
Rio Grande 985 739
East Texas Coalition 685 477
Bexar County 913 642
Total 4,461 3,186
Final Numbers by Region
Reach
• Questions to ask:
• Are we reaching our intended population?
• How do we attract underserved populations?
• How do we minimize dropout?
One of the first ways we evaluate programs is by their reach and population representativeness.
Recruiting: What Works?
Quality Leaders & Classes
Go to where seniors WORK, PLAY, PRAY, LIVE & EAT
Perseverance
Incentives
Food
Recruiting Program Participants
Networking
• Get out in your community
• Presentations
• Word of mouth
Senior Luncheons
Senior Expos, Health Fairs
City Senior Department
How to Recruit New Partners
Identify common organizational mission
• MOU
Make expectations clear
• Go beyond usual set of partners
• Identify various contributions
Think creatively
Keep a log of partner contacts
Recruiting Community Partners
Approaching Potential Partners
• Networking
• Program can be mutually beneficial to existing organizations
What to expect from a partner
• Space
• Participants
• Leaders
What to provide partner
• Program
• Leaders
• Materials
Quality Assurance & Program Fidelity
Quality Assurance refers to the entire scope of a state’s/organization’s program implementation.
• National Level
• State Level
• Regional Level
• Organizational Level
Fidelity refers very specifically to the degree to which a particular program is delivered as intended.
• Program developers
Both include: planning, monitoring, evaluating and making corrective actions
• Have a plan
Steps to Program Fidelity 4. Create a communications mechanism for feedback
3. Provide copies of plan and set training schedule
2. Prepare state-specific fidelity plan
1. Read Implementation Manual for each program
For reference, view NCOA Module 4: “Assuring Program Quality” at www.ncoa.org/chamodules
Example Fidelity Plan
Challenges
Participant Attrition
Leader Attrition
Program Fidelity
Data Collection
Solutions •Recruit & provide quality classes in community
•Provide transportation
•Know community trends
•Rural communities = fewer participants
Participant Attrition
•Provide leaders classes to teach immediately after training
•Involve leaders in scheduling classes
•Stipends if available Leader Attrition
• Identify and operationalize essential components of original program
• Know if adaptations are consistent with original study
• Dedicated Program Coordinator
• Protocol for fidelity checks
Program Fidelity
•Identify key research questions
•Simplify data collection
•Consider academic partners
•Document successes
Data Collection
Program Evaluation
• What do we want to know?
• Create instruments
• Data collection process?
• Who collects & manages the data?
• Database creation
• How to provide feedback to community & stakeholders?
Why is this
important?
Example Evaluation Manual
Program Evaluation Tips
Do you have other data collection requirements?
•AAA Client intake forms
Know target population’s abilities
•Age
•Reading Level
•Eye sight
Plan for time for program assessment
•Orientation class
•Partnering with University
Listen to and provide feedback to Key Stakeholders
Texas Size Successes
Exceeded our goal of 4,098 enrollees
Developed training infrastructure
Sustainability mechanisms
•Diabetes accreditation
•Partnering with health care organizations
Offering a menu of EBPs
Conclusions
EBPs have many benefits for improving population health
There are inventories of EBPs—no need to start from scratch
Partnerships are critical for recruitment, adoption, maintenance
For maximum impact, plan for sustainability early
Want to Learn More?
Take the online training modules: www.ncoa.org/CHAModules
Visit the Center’s website: www.ncoa.org/CHA
For more information:
Program on Healthy Aging
1266 TAMU
College Station, TX
77843-1266
Phone: 979-458-4202
Fax: 979-458-4264
www.programonhealthyaging.org