Embed Size (px)
Citation preview
Performance and outcomes measurement
Andrew Auerbach MD MPH
Associate Professor of Medicine
UCSF Division of Hospital Medicine
Introduction
• Housekeeping• Course overview• Foundational concepts in performance and
outcome assessment• Structural and management measures• Process measures• Outcome measurement
Introduction
• Ways in which outcomes and performance measurements are used: Research:
• ‘Outcomes research’ and ‘health services research’• Comparative effectiveness research• Implementation research
Operational/Policy• Quality improvement• Public reporting
Housekeeping
• Andy Auerbach [email protected] 415-502-1412 (office) 415-443-6670 (pager)
• Ashok Krishnaswami [email protected] Homework due before class. Leave at Olivia’s desk.
Housekeeping
• Each session: Will have 1-2 readings A homework (not today)
• Grades Based on homework + final exam + participation in final
class (quality debate)
Curriculum
The method (model) behind the madness
Donabedian A. Evaluating the quality of medical care. Milbank Mem Fund Q. 1966;44(3):Suppl:166-206.
More detailed model
• Donabedian A. JAMA 1988;260:1743-8
Structure Process Outcomes
Community Characteristics
Delivery System Characteristics
Provider Characteristics
Population Characteristics
Health Care Providers- Technical processes- Care processes- Interpersonal processes
Public & Patients- Access- Equity- Adherence
Health Status
Functional Status
Satisfaction
Mortality
Cost
Because in the end your analyses will look like this….
Measure = Predictor + confounders + error term
Evaluative models can be divided similarly
• Structurally-focused models: Research: Compare different systems of care in terms of clinical
outcomes• AKA - Health services research
Quality improvement/policy: Incent specific characteristics• Eg. Leapfrog measures
Evaluative models can be divided similarly
• Process-focused models: Research: Compare treatments in terms of outcomes
• Adherence research, variations in care delivery, educational research
• Use of ‘Surrogate outcomes’ with proven or a priori connection to outcomes
• Comparative effectiveness research Quality improvement/policy: Hospitalcompare.org
Evaluative models can be divided similarly
• Outcomes focused models Research: Dartmouth atlas Policy/QI: National Surgical Quality Improvement Program
What are some key features of outcomes/performance research?
• What sorts of data are used?
• What sorts of study designs does it encompass?
• How do we differentiate it from cost-effectiveness?
Features of outcomes/performance research
• Generally not experimental in nature• Generally uses preexisting data sets• Prone to numerous kinds of confounders and sorts of
biases• ‘Effectiveness’, not causality
Outcome variations based on structure
• Beds• Availability of testing on site• Teaching hospital• Volume of cases seen• Closed ICU• Number of primary care providers• Proximity to hospital or ED
Other structural factors
• Four important domains in management Strategic Cultural Technical Structural
Shortell SM, Bennett CL, Byck GR. Assessing the impact of continuous quality improvement on clinical practice: what it will take to accelerate progress. Milbank Q. 1998;76(4):593-624, 510.
Management factors required for quality improvement
• Strategic : Is there understanding about the processes and conditions
that are most important to the organization?• Cultural
Beliefs, values, norms, and behaviors in the organization which inhibit/support QI work
Increasing interest in ‘accountability’, in context of individual report cards, incentives
Management factors required for quality improvement
• Technical Training : Do people have skills necessary to carry out QI
efforts? Information support: Do people have the information
required to manage quality?• Structural
Are mechanisms which facilitate learning/dissemination of ‘learning’ available
DDx of quality problemsStrategic Cultural Technical Structural Results
NO YES YES YES • No results on anything important
YES NO YES YES • Small, temporary impact with backsliding• ‘Not how things are done’• No one notices that change is happening, good or bad
YES YES NO YES • Frustration/false starts
YES YES YES NO • Inability to capture learning and spread throughout organization
YES YES YES YES • Desired state
Shortell SM, Bennett CL, Byck GR. Assessing the impact of continuous quality improvement on clinical practice: what it will take to accelerate progress. Milbank Q. 1998;76(4):593-624, 510.
Process of care-focused research
Structure of care initiatives• Leapfrog group
Purchaser-catalyzed organization founded 1998 Collected and reported hospital data in 2001 Encourages change through public reports and purchasers 4 Leaps
• Computer Physician Order Entry (CPOE):• Evidence-Based Hospital Referral (EHR):• ICU Physician Staffing (IPS): Staffing ICUs with doctors
who have special training in critical care medicine• Leapfrog Safe Practices Score: based on NQF measures
2008 Leapfrog survey
• Leap 1: Computerized Order entry• Leap 2: Evidence-based referral volumes
Report – volumes of cases Some estimates of case-mix Geometric mean length of stay Mortality (if publicly reported)
Leapfrog 2008
Safe Practice
Creating and Sustaining a Culture of Safety
Element 1: Leadership Structures and Systems
Element 2: Culture Measurement for Performance
Element 3: Teamwork Training and Skill Building
Element 4: Identification and Mitigation of Risks and Hazards
Informed Consent
Life Sustaining Treatment
Nursing Workforce
Leapfrog 2008
Safe Practice
Communication of Critical Information
Labeling of Diagnostic Studies
Discharge Systems
Medication Reconciliation
Prevention of Aspiration and Ventilator Associated Pneumonia a
Central Venous Catheter Related Bloodstream Infection Prevention a
Hand Hygiene
DVT/VTE Prevention b
Anticoagulation Therapy b
Leapfrog 2001 and 2008
• Transition from primarily structural measures to more process focus – Why?1. Some measures hard to implement due to resource
limitations• Closed ICU – national intensivist shortage• CPOE
2. Some goals hard to achieve because of secular trendsIn CABG, national rates of the procedure are fallingEffects of volume are being subsumed by other national
initiatives
Leapfrog and outcomes
Big problem:How do you measure structure?
• Site visits• Voluntary surveys
Of which people? Patients? Physicians on the front line? Executives? Mystery shoppers?
Process-focused performance measurement
• General criteria for an optimal process measure Must target a common condition Measureable Definable ‘optimal’ patient population Requires evidence linking process to improved outcome
• Advantages: Generally, processes are more common than outcomes
statistical advantages Provide a clear performance target for clinicians Maybe: No risk adjustment once optimal patients defined
Process-focused performance measurement
• Disadvantages Medications, devices generally preferred process targets Documented practices or interpersonal/contextual factors
less so ‘Tyranny of the RCT’ Focus on only a few diseases, patient populations
Process focused performance comparisons – the major caveat
• Biases
In observational data, people who get a treatment are fundamentally different than those who don’t
Process of care initiatives
• Surgical Care Improvement Project Grew out of the Oklahoma Medicare PRO in the late 90’s,
focused initially on surgical infection prevention. Voluntary public reporting 2002-3 Medicare 2% withold for non-participation in 2004
SCIP
• Measures used in surgery Appropriate drug choice Antibiotics within 1 hour of incision Discontinuation in 24 hours VTE prevention practices
• Appropriate orders written• Appropriate therapy received in timely fashion
In cardiac surgery: Glucose < 200 at 6a on first postop day
Shortcomings of process measures
• Cautionary tale about SCIP Few data, as yet, to prove that adherence to individual
measures is associated with improved outcomes in any surgery
Growing thinking that even process measures need risk adjustment
Measuring processes
• Where do you collect process measures from?
Shortcomings of process measures
• Ceiling effects • What do we do when everyone is at 100%?• Weighting schemes for measures thought to be more
important• Incent continued excellence (e.g. number of months at
100%)• ‘All or none’ measurement
• Only ‘adherent’ if all quality measures met
Should quality be an all or none proposition?
Auerbach A, Ann Intern Med.2009;151.
Public reporting : Does it improve care?
• 45 papers reviewed, 27 since 1999• Few (or no) studies examining impact of reporting on patient or
provider level Receipt of care processes Outcomes
• When data are reported, moderate effect at system (care plan, hospital level) In case of NY CABG reporting – shift towards fewer high-risk cases being
performed No data about how public reporting influences the quality structure per
se
Ann Intern Med.2008;148:111-123.
Does paying for performance improve care?
• Natural experiment of 255 hospitals in P4P matched to 406 control hospitals
• Hospitals in the top decile on a composite measure of quality for a given year received a 2% bonus in addition to the usual reimbursement rate. Hospitals in the second decile
received a 1% bonus.
• Bonuses averaged $71,960 per year and ranged from $914 to $847,227.
N Engl J Med 2007;356:486-96.
Between 4.1 and 2.6% improvement in quality of care with P4P.But everyone improved
Do process measures need risk adjustment?
• Accounting for factors outside the site’s (or provider’s) control may be important Proportion of patients whose first HGBA1C is elevated Proportion of patients with morbid obesity This is somewhat controversial
Important caveat: Risk adjustment should not adjust away gaps related to disparities in care (e.g. gender, age, race)
Outcomes
• Most performance measurement will include a clinical outcome as the dependent variable
What are the major issues when you start comparing peoples’ (or systems’) performance in terms of their patients’ mortality, satisfaction, etc?
Outcomes focused initiatives
• Northern New England Cardiovascular Cooperative• Society of Thoracic Surgeons• Vermont Oxford Neonatal Networks• ACC PCI registries• VA NSQIP• Private Sector NSQIP
Common elements
• Collection of clinical risk adjustment and outcomes data via chart abstraction NSQIP – via nurses
• Robust risk adjustment and benchmarking• Some collaborative function via central steering
group and regional or site directors• Implicit or explicit agreements to undertake site
visits or audits of outliers
VA NSQIP
• Marked reduction in mortality for commonly performed procedures in the VA Overall mortality fell from 3% in 1995 to 1% in 2005 Consistent themes of high performers:
• Protocols, pathways, and guidelines with emphasis on standardization
• Interdisciplinary focus on systems improvement
VA NSQIP
• Not just data: Annual reports from all participants, attendance in all
group activities PRN reports from sites in upper 20% of mortality (worse
mortality) Mandatory site visits to worst performers (upper 5%) Aggressive efforts to identify poor performing physicians
Caveats of outcomes models
• Risk adjustment mandatory• Costly to collect data thought to be ‘better’
NSQIP (private sector) – about $40-50/case $50,000/year subscription fee, plus costs of data collection
nurse (1-2FTE) = 150-250K/year investment• Low event rates limit power
2005 data suggest that <60% of hospitals did enough CABG to provide good sample size to compare mortality
Advantages
• High face validity• Truly the ‘end product’• Culturally concordant and most impactful• Often contain specialty-specific complications
Tailored to needs of participating clinicians further maximizes buy-in
Conclusions
• Performance measurement can include structures, processes, or outcomes as comparators
Structural factors have strong connections to management theory, sociology, industrial design/IT
Conclusions
• Performance measurement can include structures, processes, or outcomes as comparators
Process related factors generally represent treatments physicians administer
Comparing these treatments in terms of outcomes = Comparative effectiveness research
Conclusions
• Outcomes The clearest ‘performance measure’, though not the entire
picture Risk adjustment, statistical methods paramount
