View
2
Download
0
Category
Preview:
Citation preview
From Big Data to Real Policy: Making EHR Data Matter in
Health Care
R. Adams Dudley, MD, MBAProfessor of Medicine and Health Policy
Director, UCSF Center for Healthcare Value
Twitter: @RAdamsDudleyMD 1
Financial disclosures
No relevant financial disclosures
Educational Objectives
• To learn how measurements made in the health care setting become policy
• To understand how this came to be—then faded away—in California intensive care units (ICUs)
• To recognize how new versions of these tools should be developed to increase the probability of uptake
3
How I Think about My Research
My patient: The health care system
Its diagnoses:• Quality is variable,• Cost is worse than that, • But at least we don’t insure everyone.
How Payors Try ToInfluence The System
Many of the weaknesses of the system reflect the history of how care is paid for:
• Payors have been told doctors know better than they what the patient needs,
• They have been told they therefore should pay doctors for what work they do, that is, a fee-for-service.
How Payers Try ToInfluence The System
Payors at the national level have only recently recognized fee-for-service incentivizes volume, not quality or value:
• Starting to use incentives and public reporting to stimulate performance
• In California, this started in 2004
California History with Incentives: How I Succeeded and Then Failed• The California Hospital Assessment and Reporting Task Force
was founded in 2004: Representatives from all stakeholder groups, including insurers, hospitals, doctors and nurses, employers/labor, consumers
7
History: Built ICU Outcome Models Using Hand-Collected Data
8
9
…and then to a highly accurate, multi-hospital model.
PATIENTRISK
32.5%
History: Built ICU Outcome Models Using Hand-Collected Data
10
Yields pretty good predictions (AUC for mortality prediction 0.82-0.89), but data collection was very
expensive
Much More Variation in Mortality Rates than Expected by Chance in 2006 Pilot
0
1
2
Ob
serv
ed D
eath
s/E
xpec
ted
Dea
ths
Observed ICU Deaths/Expected ICU Deaths (at each of the 110 hospitals that submitted data on ≥100 patientsO/E < 1 indicates low mortality, O/E > 1 indicates high mortality)
3-fold difference in death rates between red oval (worse) and green oval (better) hospitals.
Source: www.CalHospitalCompare.org, run by the California Hospital Assessment and Reporting Taskforce (CHART), CHCF, and UCSF
11
California Hospitals Improve Outcomes: Risk-adjusted ICU Mortality Rate Fell >2% in Three Years
10.0%
10.5%
11.0%
11.5%
12.0%
12.5%
13.0%
13.5%
14.0%
2007 2008 2009 2010
Risk-adjusted ICU Mortality
12
What Happened Next?
13
14
What Happens When You Lose Track of the Incentives?
You become irrelevant
University of CaliforniaSan Francisco
Electronic ICU Outcomes Model (e-ICOM)
An eight year scramble to use big data and machine learning to climb back to relevance
University of CaliforniaSan Francisco
Big Data Example from ICU Severity of Illness Models
Old way: Making ‘big data’ littleIn the past, ICU models would take “single worst value” for GCS, which is the same in these 3 patients. But their prognoses are very different.
New way: Making ‘big data’ big again: If data collection and analysis are cheap and easy, use all GCS values and calculate the change of the GCS over time.
University of CaliforniaSan Francisco
Estimating Severity of Illness/Risk of Death in an ICU Patient Sample
University of CaliforniaSan Francisco
Using the “Bigness” of the Data:As We Use More of the Lab and Vital Sign
Observations, the Severity Model Improves
University of CaliforniaSan Francisco
Conclusion: Using Big Data Improves Severity Assessments in Ways that Are Pretty Intuitive
University of CaliforniaSan Francisco
Adding Natural Language Processing and Machine Learning Makes It Even Better
University of CaliforniaSan Francisco
Electronic ICU Outcomes Model (e-ICOM)
Most importantly: The approach is consistent with CMS’ directive for pulling performance measures from the EHR andhospitals’ insistence on low data collection costs
University of CaliforniaSan Francisco
Overcoming the “Black Box”Which words in the text have predictive power?
Favors survival Favors mortality- +
Weights give strength of association with patient mortality
University of CaliforniaSan Francisco
Topic Modeling Interpretation and contribution of human-understandable concepts
University of CaliforniaSan Francisco
Topic Modeling Interpretation and contribution of human-understandable concepts
University of CaliforniaSan Francisco
Sensitivity analysis for modeling process Extensive overlap in both categories and specific terms
University of CaliforniaSan Francisco
Sensitivity analysis for modeling process Extensive overlap in both categories and specific terms
University of CaliforniaSan Francisco
Making sense of topicsClinician-understandable topics lend intuitive meaning to “black box” predictions
University of CaliforniaSan Francisco
Making sense of topicsClinician-understandable topics lend intuitive meaning to “black box” predictions
Implications for you• Doesn’t matter how good your methods are if your product doesn’t fit in with the business interests of the stakeholders
• Although you need leading clinicians on your team to get payors to believe that what you are doing is clinically appropriate, that is still not enough for your work to have an impact (and, hence, a market)
• How providers get paid is where you should focus:- Not on anything insurers or government say they
want…unless they pay for it- Not on what consumers want
• This applies to my team, too: The methods I’ve described will only matter if CMS or insurers re-adopt incentives addressing critical care.
Recommended