Upload
gpnp
View
665
Download
0
Embed Size (px)
DESCRIPTION
This set of slides contains supplemental information shared by Pam Meadowcroft of Meadowcroft & Associates.
Citation preview
Measuring ChangeMoving from Outputs to Outcomes to Impact
10:00am – 11:15amF ilit t Li K Ri h d Ki M ll F d tiFacilitator: Lisa Kuzma, Richard King Mellon Foundation
Speakers: Mike Bangser, MDRC / Junlei Li, Office of Child Development / Art Maxwell and Pam Meadowcroft, Meadowcroft
& Associates / Tim Weidemann, Rondout Consulting & Associates / Tim Weidemann, Rondout Consulting
Converting YOUR Program to a V lid EBP Fid li MValid EBP: Fidelity Management
Meadowcroft & Associates and Wesley Spectrum ServicesFor more information, please contact Pamela Meadowcroft, Ph.D. at [email protected] 412.683.7275
5/20/2011Meadowcroft&Associates and
Wesley Spectrum (c) 2
Evolution of what payers want AND ways id i dproviders improved programs
Old days: Then Came:OUTCOMES
Old days:PROCESS
N F t EBP+ Then EBP:Near Future: EBP+ continuous improvement (assess-plan-monitor-i )
Then EBP:process+outcomes
5/20/2011Meadowcroft&Associates and
Wesley Spectrum (c) 3
improve)
What is the Difference?Levels of confidence in the outcomes
Evidence-based practice (rigorously evaluated; most often proven via RCT)E id i f d ti / h b dEvidence-informed practice/research-based (existing research support)Best Practices (expert opinion)Best Practices (expert opinion)Promising practice (acceptable treatments, anecdotal) )InnovationsIntuition, “the way it’s always done”
5/20/2011Meadowcroft&Associates and
Wesley Spectrum (c) 4
, y y
WE KNOW A LOT ABOUT WHAT WORKS!!!!!
Meta-analyses on thousands of studiesMany programs ARE using research-based practicesThey just haven’t MEASURED and TRACKED their work!!!
5/20/2011Meadowcroft&Associates and
Wesley Spectrum (c) 5
Mark Lipsey, “Evidence-based Practice More than One Approach.” MST and FFT (two brand-names) show positive results the dark boxes but evenand FFT (two brand-names) show positive results, the dark boxes, but even
“generic” interventions showed better results.
5/20/2011Meadowcroft&Associates and
Wesley Spectrum (c) 6
From http://cjjr.georgetown.edu/pdfs/ebp/ebppaper.pdf
Wesley Spectrum In Home: History of Tracking Outcomes
91% 93% 92% 93%89%
100%
90% 90%
100%100%
Clients Discharged to Same or Less Restrictive Environments
82% 86%86% 89% 90% 90%
60%
70%
80%
90%
30%
40%
50%
60%
Allegheny
Westmoreland
0%
10%
20%
FY 2009 Qtr 1 FY 2009 Qtr 2 FY 2009 Qtr 3 FY 2009 Qtr 4 FY 2010 Qtr 1 FY 2010 Qtr 2
5/20/2011Meadowcroft&Associates and
Wesley Spectrum (c) 75/20/2011 7
FY 2009 Qtr. 1 FY 2009 Qtr. 2 FY 2009 Qtr. 3 FY 2009 Qtr. 4 FY 2010 Qtr. 1 FY 2010 Qtr. 2
B t Wh G d O t ?But… Why Good Outcomes?
Easier population? ORSomething we are DOING (our
/ d l)interventions/program model)?In other words: TRACKING OUTCOMES IS NOT ENOUGHOUTCOMES IS NOT ENOUGH
5/20/2011Meadowcroft&Associates and
Wesley Spectrum (c) 8
Id l R ltIdeal Results
High “fidelity” to the model leads to High fidelity to the model leads to the best outcomes
45 00
30.00
35.00
40.00
45.00
15.00
20.00
25.00 Model Adherence Scores
Change in Outcomes Scores
0.00
5.00
10.00
Family 1 Family 2 Family 3 Family 4
5/20/2011Meadowcroft&Associates and
Wesley Spectrum (c) 95/20/2011 9
Family 1 Family 2 Family 3 Family 4
Steps for Building a local EBP: Fidelity Management
fDefine the programDevelop and Track Model Fidelity ( )(outputs)Develop and Monitor OutcomesValidate the Locally-Developed Program Model (link outputs to
)outcomes)Build-in CQI
5/20/2011Meadowcroft&Associates and
Wesley Spectrum (c) 10
T l f A i M d l Fid litTools for Assuring Model Fidelity
Th i d S i Ch kli Therapist and Supervisor Checklists (Intake, Monthly, Discharge) include:
Who we are serving (population assessments)Who we are serving (population assessments)What are we doing (outputs related to key activities, intensity of services)How’d we do (client outcomes)How’d we do (client outcomes)
Consumer Satisfaction SurveysItems relate to key program activities; e s e a e o ey p og a ac es;additional output measures
Embed in CQI (participating in QII)
5/20/2011Meadowcroft&Associates and
Wesley Spectrum (c) 115/20/2011 11
Model Adherence Scores for Wesley Spectrum In Home Model Elements: Two Sites Compared to Be Used for CQI4
3.754
3.75 3.75 3.753 6
4
3.5 3.5 3.5
3
3.53.25
3.5 3.6
2 5
3
3.5
1.5
2
2.5
Westmoreland n=4
0.5
1Allegheny n=10
0
5/20/2011Meadowcroft&Associates and
Wesley Spectrum (c) 125/20/2011 12
Higher Model Fidelity Improved Child Well-Being: Strong Relationship g g pbetween Outputs and Outcomes
NCFAS-G Child Well-Being Change Scores
0.6
0.7
NCFAS-G Child Well-Being Change Scores
0 3
0.4
0.5
Lower Adherence Group
0.1
0.2
0.3Higher Adherence Group
0Overall Child Well-Being
Child's Behavior
School Performance
Child's Relationship with Parents
Motivation to Maintain Family
5/20/2011Meadowcroft&Associates and
Wesley Spectrum (c) 13
Overall Child Well-Being and Child's Behavior significant at p<.05 level
Wesley Spectrum In Home VS Multi-Systemic Therapy (name-brand EBP) with y py ( )Comparable Populations
Percent of Consumers Who Achieved Outcomes
0.92
0.96
0.8
0.84
0.88
MST
WSIH-All Cases
0.72
0.76
0.8
WSIH-Higher Adherence Cases
0.68Completed
Therapy / Planned Discharge
No Arrests / Planned Discharge
Child in Home Child in School
5/20/2011Meadowcroft&Associates and
Wesley Spectrum (c) 14
Child In School significant for MST vs. WSIH Higher (p<.05).
Brand-name EBP vs Local-developed EBP
Purchased EBP Model$millions for research and evaluation
Home Grown Model Building ProcessLow-cost research and evaluation in short timeand evaluation
Many decades research/developmentHighly prescribed
evaluation in short-timeModerate level program requirementsLower program cost
Low adaptabilityHigh effortOngoing high program C t (
Greater utility across populationsEmbedded in CQITools for incorporating Cost (e.g.,
recertification)Tools for incorporating new practicesStaff commitment
5/20/2011Meadowcroft&Associates and
Wesley Spectrum (c) 15
K C l iKey Conclusions
E id b d d l li it ti th tEvidence based models pose limitations that our model building process does not
Our model building process is replicable so other programs could do the same
The process gives programs supervision and monitoring tools for continuous improvement ANDmonitoring tools for continuous improvement AND for making the case of value to stakeholders
5/20/2011Meadowcroft&Associates and
Wesley Spectrum (c) 16