Selecting measures for purchasing:Selecting measures for purchasing:Quality measurementQuality measurement

Patrick S. Romano, MD MPHPatrick S. Romano, MD MPHUC Davis School of MedicineUC Davis School of Medicine

Washington State Conference on Quality-based PurchasingWashington State Conference on Quality-based PurchasingDecember 4, 2006December 4, 2006

Romano, 12/4/2006 2

OverviewOverview

Types of quality indicatorsTypes of quality indicators Strengths and limitations of different types of Strengths and limitations of different types of

quality indicatorsquality indicators Potential evaluation criteriaPotential evaluation criteria Examples from the fieldExamples from the field Consider unintended consequencesConsider unintended consequences Conclusions and recommendationsConclusions and recommendations

Romano, 12/4/2006 3

Definitions of qualityDefinitions of quality Donabedian (1980):Donabedian (1980):

““The quality of medical care (is)…the management that is expected to The quality of medical care (is)…the management that is expected to achieve the best balance of health benefits and risks…(taking) into achieve the best balance of health benefits and risks…(taking) into account the patient’s wishes, expectations, valuations, and means…account the patient’s wishes, expectations, valuations, and means…(and) the distribution of that benefit within the population.”(and) the distribution of that benefit within the population.”

Institute of Medicine (1990):Institute of Medicine (1990):““Quality of care is the degree to which health services for individuals Quality of care is the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge.”are consistent with current professional knowledge.”

Brook and McGlynn (1991):Brook and McGlynn (1991):““High quality care…produces positive changes, or slows the decline, in High quality care…produces positive changes, or slows the decline, in health; low quality care fails to prevent or actually accelerates a decline health; low quality care fails to prevent or actually accelerates a decline in a person’s health.”in a person’s health.”

Romano, 12/4/2006 4

Types of quality measuresTypes of quality measuresDonabedian, 2003Donabedian, 2003

Structure:Structure: the conditions under which care is provided the conditions under which care is provided Material resources (facilities, equipment)Material resources (facilities, equipment) Human resources (ratios, qualifications, experience)Human resources (ratios, qualifications, experience) Organizational characteristics (size, volume, systems)Organizational characteristics (size, volume, systems)

Process:Process: the activities that constitute health care (adherence to the activities that constitute health care (adherence to guidelines)guidelines) Screening and diagnosisScreening and diagnosis Treatment and rehabilitationTreatment and rehabilitation Education and preventionEducation and prevention

Outcome:Outcome: changes attributable to health care changes attributable to health care Mortality, morbidity (complications, readmissions), functional statusMortality, morbidity (complications, readmissions), functional status Knowledge, attitudes, and behaviorsKnowledge, attitudes, and behaviors Satisfaction (including patient experiences)Satisfaction (including patient experiences)

Romano, 12/4/2006 5

Structural measures for QBP: Structural measures for QBP: Background and ProblemsBackground and Problems

Structural measures are enabling factors that make it easier Structural measures are enabling factors that make it easier (or harder) for professionals to provide high-quality care(or harder) for professionals to provide high-quality care

Usually explain little of the observed variability in processes Usually explain little of the observed variability in processes and outcomesand outcomes

Few randomized trials, so causal relationships are often Few randomized trials, so causal relationships are often unclear. Do better structures lead to better processes, or do unclear. Do better structures lead to better processes, or do better processes create demand for different structures (e.g., better processes create demand for different structures (e.g., selective referral, CPOE)?selective referral, CPOE)?

Often easy to measure, but hard to modify and even harder to Often easy to measure, but hard to modify and even harder to evaluate. Few randomized intervention studies. evaluate. Few randomized intervention studies.

Romano, 12/4/2006 6

Structural measures for QBP: Structural measures for QBP: ImplicationsImplications

Structural indicators should be viewed as markers or Structural indicators should be viewed as markers or facilitators of quality rather than as true measuresfacilitators of quality rather than as true measures

QBP programs have relied on structural indicators when QBP programs have relied on structural indicators when acceptable process or outcome measures were not yet acceptable process or outcome measures were not yet available (transitional practice to avoid “free ride”)available (transitional practice to avoid “free ride”)

Focus on structural indicators that are modifiable (e.g., Focus on structural indicators that are modifiable (e.g., accreditation, training of key physicians)accreditation, training of key physicians)

Avoid structural indicators for which hasty Avoid structural indicators for which hasty implementation may lead to worse outcomes (CPOE)implementation may lead to worse outcomes (CPOE)

Use non-modifiable measures only if you are willing to Use non-modifiable measures only if you are willing to close down organizations that cannot changeclose down organizations that cannot change

Copyright ©2005 American Academy of Pediatrics

Han, Y. Y. et al. Pediatrics 2005;116:1506-1512

Fig 1. Observed mortality rates (presented as a normalized % of predicted mortality) during the 18-month study period are plotted according to quarter of year

Romano, 12/4/2006 8

“Role of computerized physician order entry systems in facilitating medication errors”

Romano, 12/4/2006 9

Example error typesExample error types

Entering order for wrong patient due to interruptionEntering order for wrong patient due to interruption Delays in orders when patients not yet entered into systemDelays in orders when patients not yet entered into system

– One fatal example reported in previous One fatal example reported in previous JAMA JAMA piecepiece Incorrect default dosing or protocolIncorrect default dosing or protocol Overloading users with alerts and reminders for Overloading users with alerts and reminders for

completenesscompleteness– Ignoring/over-riding all alerts and requestsIgnoring/over-riding all alerts and requests

Medications discontinued without clinicians being aware Medications discontinued without clinicians being aware Koppel et al. Role of CPOE in facilitating medication errors. JAMA 2005.

Ash J et al. Unintended Consequences of IT in Health Care J Am Med Inform Assoc 2004

Romano, 12/4/2006 10

Process measures for QBP:Process measures for QBP:BackgroundBackground

Process measures are directly actionable by health Process measures are directly actionable by health care providers (“opportunities for intervention”)care providers (“opportunities for intervention”)

Process measures are highly responsive to changeProcess measures are highly responsive to change Process measures have generally been tested and Process measures have generally been tested and

validated (or could be validated) in randomized validated (or could be validated) in randomized controlled trialscontrolled trials

Process measures provide the pathways by which Process measures provide the pathways by which QBP leads to improved patient outcomesQBP leads to improved patient outcomes

Romano, 12/4/2006 11

Process measures for QBP:Process measures for QBP:ProblemsProblems

Often costly or difficult to collectOften costly or difficult to collectPharmacy/laboratory utilization (complete data capture?)Pharmacy/laboratory utilization (complete data capture?)Chart review (information bias?)Chart review (information bias?)Patient surveys (recall bias?)Patient surveys (recall bias?)Participant observation (Hawthorne effect?)Participant observation (Hawthorne effect?)Provider surveys/vignettes (social desirability bias?)Provider surveys/vignettes (social desirability bias?)Simulated patients (reliability?)Simulated patients (reliability?)

Romano, 12/4/2006 12

Process measures for QBP:Process measures for QBP:ProblemsProblems

Implicit process measures often lack reliabilityImplicit process measures often lack reliabilityMultiple peer reviewers are required (at least 5?)Multiple peer reviewers are required (at least 5?)Unblinded reviewers are biased by adverse outcomesUnblinded reviewers are biased by adverse outcomesMay not be actionable (too global)May not be actionable (too global)Evaluation criteria may be context-specificEvaluation criteria may be context-specific

Explicit process measures often lack validityExplicit process measures often lack validityAre they really evidence-based (vs. “expert opinion”)?Are they really evidence-based (vs. “expert opinion”)?Some processes that seem important may NOT beSome processes that seem important may NOT beMany important processes have not yet been recognizedMany important processes have not yet been recognizedMeasures may not generalize across settingsMeasures may not generalize across settings

Romano, 12/4/2006 13

Process measures for QBP: Process measures for QBP: ImplicationsImplications

Process measures rely on exclusions instead of risk-Process measures rely on exclusions instead of risk-adjustment, so exclusions should be clearly definedadjustment, so exclusions should be clearly defined– ““Patients with moderate-to-severe asthma should not receive beta-Patients with moderate-to-severe asthma should not receive beta-

blocker…” (but how is that defined?)blocker…” (but how is that defined?) The validity of process measures depends on evidence, so The validity of process measures depends on evidence, so

focus on measures with a strong evidence basefocus on measures with a strong evidence base– ““Proportion of eligible registry patients who have documentation…of a Proportion of eligible registry patients who have documentation…of a

physician…statement regarding the patient’s symptoms that coincides physician…statement regarding the patient’s symptoms that coincides with NAEPP terminology…” (who cares?)with NAEPP terminology…” (who cares?)

Focus on actual care rather than documentation of care, and Focus on actual care rather than documentation of care, and establish systems for auditing data or ensuring data accuracyestablish systems for auditing data or ensuring data accuracy

Process measures tend to be provider-centered, so consider Process measures tend to be provider-centered, so consider including user-centered measures as wellincluding user-centered measures as well

Romano, 12/4/2006 14

Process measures for QBP:Process measures for QBP:Promise and PotentialPromise and Potential

The cost of collecting process measures can be The cost of collecting process measures can be reduced with clinical information systems:reduced with clinical information systems: Electronic medical records Electronic medical records Linked pharmacy and laboratory claimsLinked pharmacy and laboratory claims

Patient surveys now reliably measure patient-Patient surveys now reliably measure patient-centered processes of care, such as education, centered processes of care, such as education, communication, and pain management.communication, and pain management.

The evidence-based medicine paradigm has led to The evidence-based medicine paradigm has led to greater reliance on RCTs and systematic reviews greater reliance on RCTs and systematic reviews to identify useful processes of careto identify useful processes of care

Romano, 12/4/2006 15

Outcome measures:Outcome measures:BackgroundBackground

Outcomes are what really matter to patients, Outcomes are what really matter to patients, families, and communitiesfamilies, and communities

Outcomes are intrinsically meaningful and generally Outcomes are intrinsically meaningful and generally easy to understandeasy to understand

Outcomes reflect not just what was done but how Outcomes reflect not just what was done but how well it was done (which is hard to measure directly)well it was done (which is hard to measure directly)

Outcomes may be ascertainable using Outcomes may be ascertainable using administrative data, if such data existadministrative data, if such data exist

Romano, 12/4/2006 16

Outcome measures:Outcome measures:ProblemsProblems

Data systems depend on reporting by provider organizationsData systems depend on reporting by provider organizations Morbidity measures tend to be documented and reported Morbidity measures tend to be documented and reported

inconsistently (poor physician documentation and/or coding)inconsistently (poor physician documentation and/or coding) Mortality measures may be confounded by variation in use of Mortality measures may be confounded by variation in use of

observation units, inter-hospital transfers, and LOSobservation units, inter-hospital transfers, and LOS Severity of illness varies widely across providers; most Severity of illness varies widely across providers; most

existing data systems capture little of this variationexisting data systems capture little of this variation Many adverse outcomes are rare or delayed (e.g., little short-Many adverse outcomes are rare or delayed (e.g., little short-

term responsiveness to change, lots of random noise)term responsiveness to change, lots of random noise) Are outcomes sufficiently under providers’ control?Are outcomes sufficiently under providers’ control?

Romano, 12/4/2006 17

Outcome measures:Outcome measures:Promise and PotentialPromise and Potential

Internal and external (e.g., vital statistics) data linkages may Internal and external (e.g., vital statistics) data linkages may minimize confounding due to variation in transfer rates and LOS.minimize confounding due to variation in transfer rates and LOS.

Many states now capture data from emergency departments (EDs) Many states now capture data from emergency departments (EDs) and/or ambulatory surgery centers; readmissions can be identified and/or ambulatory surgery centers; readmissions can be identified using Medicare data or linked state data.using Medicare data or linked state data.

Some data sets (NY, CA, soon FL) distinguish comorbidities from Some data sets (NY, CA, soon FL) distinguish comorbidities from complications, or add “clinical” data elements (e.g., “key clinical complications, or add “clinical” data elements (e.g., “key clinical findings” in PA; DNR in CA and NJ).findings” in PA; DNR in CA and NJ).

Mail/telephone patient satisfaction surveys (CAHPS, H-CAHPS) Mail/telephone patient satisfaction surveys (CAHPS, H-CAHPS) have been developed and validated. have been developed and validated.

Some outcomes monitoring systems now have clear definitions, Some outcomes monitoring systems now have clear definitions, detailed guidance for data collectors, and external auditing.detailed guidance for data collectors, and external auditing.

Romano, 12/4/2006 18

Outcome measures for QBP: Outcome measures for QBP: ImplicationsImplications

Outcome measures rely on risk-adjustment, so methods Outcome measures rely on risk-adjustment, so methods should be open (not “black-box”) and validatedshould be open (not “black-box”) and validated

The utility of outcome measures depends on the existence of The utility of outcome measures depends on the existence of treatments that work, so focus on measures with a strong treatments that work, so focus on measures with a strong evidence base from prior intervention studiesevidence base from prior intervention studies

Outcome measures are relatively easy to game (by not Outcome measures are relatively easy to game (by not reporting complications or over-reporting comorbidities), so reporting complications or over-reporting comorbidities), so focus on “harder” outcomes and establish systems for auditing focus on “harder” outcomes and establish systems for auditing data or ensuring data accuracydata or ensuring data accuracy

Romano, 12/4/2006 19

Consider processes and outcomes togetherConsider processes and outcomes together

Integrating outcome and process measures Integrating outcome and process measures provides a more complete assessment of quality provides a more complete assessment of quality and avoids perverse incentivesand avoids perverse incentives

Agreement among process and outcome Agreement among process and outcome measures confirms the validity of eachmeasures confirms the validity of each

Disagreement suggests bad data (information Disagreement suggests bad data (information bias), unmeasured severity of illness (confounding bias), unmeasured severity of illness (confounding bias), selection factors, or an incorrect conceptual bias), selection factors, or an incorrect conceptual model linking processes and outcomesmodel linking processes and outcomes

Romano, 12/4/2006 20

Selecting quality measures for QBPSelecting quality measures for QBPEvaluation criteria: NQF and othersEvaluation criteria: NQF and others

Importance or relevanceImportance or relevance Scientific acceptability or soundnessScientific acceptability or soundness Usability Usability FeasibilityFeasibility

Note that all of these criteria may depend on Note that all of these criteria may depend on local circumstances and priorities…local circumstances and priorities…

NQF IOM/NHQR JCAHO NCQA

Importance/Relevance

Leverage point for improving quality

Impact on health Targets improvement in the health of populations

Strategically importantClinically important

Meaningfulness to policymakers, consumers

Meaningful to consumers, purchasers, plans, providers

Performance in the area is suboptimal

Potential for improvement

Aspect of quality is under provider control.*

Susceptibility to being influenced by health care

Under provider control Controllable

Considerable variation in quality of care exists

Variance among plans/providers

Financially important

Romano, 12/4/2006 22

Estimating the impact of implementing Estimating the impact of implementing Leapfrog hospital volume standards (NIS)Leapfrog hospital volume standards (NIS)

Birkmeyer et al., Surgery 2001;130:415-22Birkmeyer et al., Surgery 2001;130:415-22

Volume indicatorVolume indicatorRR mortalityRR mortalityLVH vs HVHLVH vs HVH

Patients at LVHs Patients at LVHs in MSAsin MSAs

Potential lives saved Potential lives saved by volume standardsby volume standards

CABGCABG 1.381.38 164,261164,261 1,4861,486

Coronary Coronary angioplasty/PCIangioplasty/PCI 1.331.33 121,292121,292 345345

AAA repairAAA repair 1.601.60 18,53418,534 464464

Carotid endarterectomyCarotid endarterectomy 1.281.28 82,54482,544 118118

EsophagectomyEsophagectomy 3.013.01 1,6961,696 168168

Romano, 12/4/2006 23

Leapfrog Hospital Rewards Program:Leapfrog Hospital Rewards Program:Focused clinical areas chosen to maximize commercial employer impactFocused clinical areas chosen to maximize commercial employer impact

5 of the ten CFGs have 5 of the ten CFGs have NQF-approved measures NQF-approved measures collected by JCAHOcollected by JCAHO

Benchmarked against Benchmarked against Medstat’s MarketScan, the 5 Medstat’s MarketScan, the 5 CFGs represent 33% of CFGs represent 33% of admissions and 20% of a admissions and 20% of a commercial payer’s inpatient commercial payer’s inpatient spendingspending

Ran

k

Clinical Focus Group

Total Cost to Commercial

Insurer

Total Potential

Opportunity

1CORONARY ARTERY BYPASS GRAFT $691,772,784 $62,666,869

2PERCUTANEOUS CORONARY INTERVENTION $717,954,275 $58,157,873

3ACUTE MYOCARDIAL INFARCTION $607,227,166 $53,616,015

4 COLON SURGERY $396,004,245 $38,389,6735 HEART FAILURE $224,919,006 $34,983,226

6COMMUNITY ACQUIRED PNEUMONIA $355,686,956 $29,536,322

7 OTHER CARDIAC SURGERY $211,578,764 $25,767,1918 PREGNANCY AND NEWBORNS $1,781,273,763 $23,368,7219 VASCULAR SURGERY $133,287,531 $16,412,194

10 SPINE - OTHER $422,595,301 $12,925,843

Potential savings from reduced complication and re-admission rates

Romano, 12/4/2006 24

Estimating the impact of preventing each PSI Estimating the impact of preventing each PSI event on mortality, LOS, charges (ROI)event on mortality, LOS, charges (ROI)

NIS 2000 analysis by Zhan & Miller, JAMA 2003;290:1868-74NIS 2000 analysis by Zhan & Miller, JAMA 2003;290:1868-74

IndicatorIndicator ΔΔ Mort (%) Mort (%) ΔΔ LOS (d) LOS (d) ΔΔ Charge Charge ($) ($)

Postoperative septicemiaPostoperative septicemia 21.921.9 10.910.9 $57,700$57,700

Postoperative thromboembolismPostoperative thromboembolism 6.66.6 5.45.4 21,70021,700

Postoperative respiratory failurePostoperative respiratory failure 21.821.8 9.19.1 53,50053,500

Postoperative physiologic or metabolic Postoperative physiologic or metabolic derangementderangement

19.819.8 8.98.9 54,80054,800

Decubitus ulcerDecubitus ulcer 7.27.2 4.04.0 10,80010,800

Selected infections due to medical careSelected infections due to medical care 4.34.3 9.69.6 38,70038,700

Postoperative hip fracturePostoperative hip fracture 4.54.5 5.25.2 13,40013,400

Accidental puncture or lacerationAccidental puncture or laceration 2.22.2 1.31.3 8,3008,300

Iatrogenic pneumothoraxIatrogenic pneumothorax 7.07.0 4.44.4 17,30017,300

Postoperative hemorrhage/hematomaPostoperative hemorrhage/hematoma 3.03.0 3.93.9 21,40021,400

Romano, 12/4/2006 25

Estimating the impact of preventing each PSI Estimating the impact of preventing each PSI event on mortality, LOS, charges (ROI)event on mortality, LOS, charges (ROI)

NIS 2000 analysis by Zhan & Miller, JAMA 2003;290:1868-74NIS 2000 analysis by Zhan & Miller, JAMA 2003;290:1868-74

IndicatorIndicator ΔΔ Mort (%) Mort (%) ΔΔ LOS (d) LOS (d) ΔΔ Charge Charge ($) ($)

Birth traumaBirth trauma -0.1 (NS)-0.1 (NS) -0.1 (NS)-0.1 (NS) 300 (NS)300 (NS)

Obstetric trauma –cesareanObstetric trauma –cesarean -0.0 (NS)-0.0 (NS) 0.40.4 2,7002,700

Obstetric trauma - vaginal w/out instrumentationObstetric trauma - vaginal w/out instrumentation 0.0 (NS)0.0 (NS) 0.050.05 -100 (NS)-100 (NS)

Obstetric trauma - vaginal w instrumentationObstetric trauma - vaginal w instrumentation 0.0 (NS)0.0 (NS) 0.070.07 220220

Postoperative abdominopelvic wound Postoperative abdominopelvic wound dehiscencedehiscence

9.69.6 9.49.4 40,30040,300

Transfusion reaction*Transfusion reaction* -1.0 (NS)-1.0 (NS) 3.4 (NS)3.4 (NS) 18,900 (NS)18,900 (NS)

Complications of anesthesia*Complications of anesthesia* 0.2 (NS)0.2 (NS) 0.2 (NS)0.2 (NS) 1,6001,600

Foreign body left during procedureForeign body left during procedure†† 2.12.1 2.12.1 13,30013,300

* All differences NS for transfusion reaction and complications of anesthesia in VA/PTF.

† Mortality difference NS for foreign body in VA/PTF.

Romano, 12/4/2006 26

RAND QA Tools:RAND QA Tools:A comprehensive assessment of qualityA comprehensive assessment of quality

N Engl J Med. 2003 Jun 26;348(26):2635-45N Engl J Med. 2003 Jun 26;348(26):2635-45

Selected 30 clinical areas representing about half of reasons Selected 30 clinical areas representing about half of reasons people seek carepeople seek care

Developed specific standards or indicators within each clinical area Developed specific standards or indicators within each clinical area based on literature reviewsbased on literature reviews

Convened 45 experts nominated by specialty societies to evaluate Convened 45 experts nominated by specialty societies to evaluate proposed standards proposed standards

Sampled households from 12 metro areas around the nationSampled households from 12 metro areas around the nation Conducted telephone interviews (demographics, health history, Conducted telephone interviews (demographics, health history,

some process measures)some process measures) Obtained and abstracted medical records from all providers for the Obtained and abstracted medical records from all providers for the

two years preceding the date of the telephone interviewtwo years preceding the date of the telephone interview 79+45 measures translated to CPT/ICD-9-CM codes for use with 79+45 measures translated to CPT/ICD-9-CM codes for use with

billing data (Care Focused Purchasing initiative led by Mercer)billing data (Care Focused Purchasing initiative led by Mercer)

Romano, 12/4/2006 27

Potential for improvement may vary Potential for improvement may vary across diseases and treatmentsacross diseases and treatments

0 20 40 60 80 100

Alcohol dependence

Ulcers

Diabetes

Headache

BPH

Osteoarthritis

Depression

Low back pain

Cataracts

% of standards passed

Romano, 12/4/2006 28

Jha, A. K. et al. N Engl J Med 2005;353:265-274

Potential for improvement may vary across regions and communities

Romano, 12/4/2006 29

Jha, A. K. et al. N Engl J Med 2005;353:265-274

Which JCAHO/CMS Core Measures had the greatest variation across hospitals (for Medicare patients

admitted with AMI) in January-June 2004?

Romano, 12/4/2006 30

Jha, A. K. et al. N Engl J Med 2005;353:265-274

Which JCAHO/CMS Core Measures had the greatest variation across hospitals (for Medicare patients with

CHF or pneumonia) in January-June 2004?

Romano, 12/4/2006 31

Williams, S. C. et al. N Engl J Med 2005;353:255-264

What is the potential “value-added” from using an existing indicator for QBP?

Trends for AMI and pneumonia at US hospitals, 7/02-6/04

NQF AHRQ/NHQR JCAHO NCQA

Scientific Acceptability/Soundness

Well-defined and precisely specified

Precisely defined and specified

Precisely specified (under “Feasibility”)

Reliable Reliability (“stable results”)

Reliable (“identify consistently”)

Reproducible

Valid (“accurately representing the concept”)

Validity (“measure what it is intended to measure”)

Valid (“capture what it was intended to measure”)

Valid (face, construct, content)

Precise, adequate discrimination

Accurate (“reasonable level of precision”)

Adaptable to patient preferences and variety of settings

Comparability of data sources

Adequate, specified risk-adjustment

Risk-adjusted or stratified (if needed)

Risk-adjustable

Evidence linking process measures to outcomes

Explicitness of the evidence base

Degree of professional agreement

Romano, 12/4/2006 33

Reliability of PSIs: hospital-level signal ratioReliability of PSIs: hospital-level signal ratio

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0Foreign Body

Tranf. Reaction

Postop AP wound dehis

Postop hemorr/hemat

Postop physio/metab

Iatrogenic PTX

Postop hip fracture

Ob trauma –cesarean

Failure to rescue

Anesth complications

Postop resp failure

Postop sepsis

Postop DVT/PE

Death low mort DRGs

Ob trauma - vag forc/vac

Accid puncture/lac

Selected infection

Decubitus ulcer

Ob trauma - vag w/out

Birth trauma

Source: 2002 State Inpatient Data. Average signal ratio across hospitals after risk-adjustment (N=4,428)

Romano, 12/4/2006 34

Year-to-year correlation of hospital effects for Year-to-year correlation of hospital effects for PSIsPSIs

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0Foreign Body

Tranf. ReactionPostop AP wound dehis

Postop hemorr/hemat

Postop physio/metabIatrogenic PTX

Postop hip fracture

Ob trauma –cesareanFailure to rescue

Anesth complicationsPostop resp failure

Postop sepsis

Postop DVT/PEDeath low mort DRGs

Ob trauma - vag forc/vac

Accid puncture/lacSelected infection

Decubitus ulcer

Ob trauma - vag w/outBirth trauma

Source: 2001-2002 State Inpatient Data, hospitals with at least 1,000 discharges (N=4,428). Risk-adjusted unsmoothed rates.

Romano, 12/4/2006 35

Face (consensual) validity of PSIs: Face (consensual) validity of PSIs:

Clinical panel reviewClinical panel review Modified RAND/UCLA Appropriateness MethodModified RAND/UCLA Appropriateness Method Physicians of various specialties/subspecialties, nurses, other Physicians of various specialties/subspecialties, nurses, other

specialized professionals (e.g., midwife, pharmacist)specialized professionals (e.g., midwife, pharmacist) Potential indicators were rated by 8 multispecialty panels; surgical Potential indicators were rated by 8 multispecialty panels; surgical

indicators were also rated by 3 surgical panelsindicators were also rated by 3 surgical panels Pre-conference ratings, focused discussion, post-conference ratingsPre-conference ratings, focused discussion, post-conference ratings All panelists rated all assigned indicators (1-9) on: All panelists rated all assigned indicators (1-9) on:

– Overall usefulnessOverall usefulness– Likelihood of identifying the occurrence of an adverse event or Likelihood of identifying the occurrence of an adverse event or

complication (i.e., not present at admission)complication (i.e., not present at admission)– Likelihood of being preventable (i.e., not an expected result of underlying Likelihood of being preventable (i.e., not an expected result of underlying

conditions) conditions) – Likelihood of being due to medical error or negligence (i.e., not just lack of Likelihood of being due to medical error or negligence (i.e., not just lack of

ideal or perfect care)ideal or perfect care)– Likelihood of being clearly charted Likelihood of being clearly charted – Extent to which indicator is subject to case mix biasExtent to which indicator is subject to case mix bias

Romano, 12/4/2006 36

Expert panel ratings of PSI “preventability”Expert panel ratings of PSI “preventability” Acceptable Median 7-9; <2 rated <7

Acceptable (-) Median 7-9; <2 rated <4

Unclear Median 5-7 or disagree

Unclear (-) Median 4-5

Decubitus ulcer Complications of anesthesia

Death in low mortality DRG

Failure to rescue

Foreign body left in Selected infections due to medical care

Postop hemorhage/ hematoma

Postop physiologic/ metabolic derange

Iatrogenic pneumothoraxa

Postop PE or DVTb Postop respiratory failure

Postop hip fracturea Transfusion reaction Postop abdominopelvic wound dehiscence

Technical difficulty with procedure

Birth trauma Postop sepsis

Obstetric trauma (all delivery types)

a Panel ratings were based on definitions different than final definitions. For “Iatrogenic pneumothorax,” the rated denominator was restricted to patients receiving thoracentesis or central lines; the final definition expands the denominator to all patients (with same exclusions). For “In-hospital fracture” panelists rated the broader Experimental indicator, which was replaced in the Accepted set by “Postoperative hip fracture” due to operational concerns. b Vascular complications were rated as Unclear (-) by surgical panel; multispecialty panel rating is shown here.

Romano, 12/4/2006 37

International expert panel ratings of PSIsInternational expert panel ratings of PSIsOrganization for Economic Cooperation and DevelopmentOrganization for Economic Cooperation and Development

PSIs recommended

PSIs not recommended

Experimental or rejected PSIs recommended

Selected infections due to medical care

Death in low mortality DRG Postop wound infection

Decubitus ulcer Postop hemorhage/ hematoma In-hospital hip fracture or fall Complications of anesthesia Iatrogenic pneumothorax Postop PE or DVT Postop abdominopelvic wound

dehiscence

Postop sepsis Failure to rescue Technical difficulty with procedure

Postop physiologic/ metabolic derangement

Transfusion reaction Postop respiratory failure Foreign body left in Postop hip fracture Birth trauma Obstetric trauma (all delivery types)

Romano, 12/4/2006 38

Can Solucient’s Expected Complication Rate Index be used for QBP?Can Solucient’s Expected Complication Rate Index be used for QBP?Criterion validity of Iezzoni’s Complications Screening ProgramCriterion validity of Iezzoni’s Complications Screening ProgramMed Care 2000;38:785-806,868-76; Int J Qual Health Care 1999;11:107-18Med Care 2000;38:785-806,868-76; Int J Qual Health Care 1999;11:107-18

CSP IndicatorCSP Indicator

Coder (%):Coder (%):ComplicationComplication

PresentPresent11

RNRN (%) (%):: Process Process problemproblem

identifiedidentified

MDMD (%) (%)::ComplicationComplication

presentpresent

MDMD (%) (%)::QualityQuality

problemproblemconfirmedconfirmed

Postprocedural hemorrhage/hematomaPostprocedural hemorrhage/hematoma 83 (surg)83 (surg)49 (med)49 (med)

66 vs 4666 vs 4613 vs 513 vs 5

57 (surg)57 (surg)55 (med)55 (med)

37 vs 237 vs 231 vs 231 vs 2

Postop pulmonary compromisePostop pulmonary compromise 7272 52 vs 4652 vs 46 7575 20 vs 220 vs 2

DVT/PEDVT/PE 59 (surg)59 (surg)32 (med)32 (med)

72 vs 4672 vs 4669 vs 569 vs 5

70 (surg)70 (surg)28 (med)28 (med)

50 vs 250 vs 220 vs 220 vs 2

In-hosp hip frx and fallsIn-hosp hip frx and falls 57 (surg)57 (surg)11 (med)11 (med)

76 vs 4676 vs 4654 vs 554 vs 5

71 (surg)71 (surg)11 (med)11 (med)

24 vs 224 vs 25 vs 25 vs 2

1 Contrast between cases flagged with this CSP indicator and cases unflagged by any CSP indicator.

Romano, 12/4/2006 39

Criterion validity in CA hospital discharge dataCriterion validity in CA hospital discharge datavaries with different definitions of obstetric complicationsvaries with different definitions of obstetric complications

Romano PS, et al. Obstet Gynecol 2005;106(4):717-725Romano PS, et al. Obstet Gynecol 2005;106(4):717-725

IndicatorIndicatorSensitivitySensitivity PPVPPV

UnweightedUnweighted WeightedWeighted UnweightedUnweighted WeightedWeighted

FORMERFORMERAHRQ PSI: Obstetric trauma,AHRQ PSI: Obstetric trauma,Cesarean deliveryCesarean delivery

11%11% 5%5% 67%67% 94%94%

HealthGrades: major comps,HealthGrades: major comps,Vaginal deliveryVaginal delivery

67%67% 58%58% 91%91% 91%91%

HealthGrades: major comps, HealthGrades: major comps, Cesarean deliveryCesarean delivery

55%55% 47%47% 64%64% 79%79%

AHRQ/JCAHO: 3AHRQ/JCAHO: 3rdrd or 4 or 4thth degree lacerationdegree laceration

90%90% 93%93% 90%90% 73%73%

Romano, 12/4/2006 40

Construct validity based on literature Construct validity based on literature review (MEDLINE/EMBASE)review (MEDLINE/EMBASE)

Approaches to assessing construct validityApproaches to assessing construct validity– Is the outcome indicator associated with explicit processes Is the outcome indicator associated with explicit processes

of care (e.g., appropriate use of medications)?of care (e.g., appropriate use of medications)?– Is the outcome indicator associated with implicit process Is the outcome indicator associated with implicit process

of care (e.g., global ratings of quality)?of care (e.g., global ratings of quality)?– Is the process indicator associated with a clinically Is the process indicator associated with a clinically

meaningful outcome?meaningful outcome?– Is the outcome (process) indicator associated with nurse Is the outcome (process) indicator associated with nurse

staffing or skill mix, physician skill mix, or other aspects of staffing or skill mix, physician skill mix, or other aspects of hospital structure?hospital structure?

Summary of published construct validity evidence for PSIsSummary of published construct validity evidence for PSIsIndicatorIndicator Explicit processExplicit process Implicit processImplicit process StaffingStaffing

Complications of anesthesiaComplications of anesthesia

Death in low mortality DRGsDeath in low mortality DRGs ++

Decubitus ulcerDecubitus ulcer ±±

Failure to rescueFailure to rescue ++++

Foreign body left during procedureForeign body left during procedure

Iatrogenic pneumothoraxIatrogenic pneumothorax

Selected infections due to medical careSelected infections due to medical care

Postop hip fracturePostop hip fracture ++ ++

Postop hemorrhage or hematomaPostop hemorrhage or hematoma ±± ++

Postop physiologic/metabolic derangementsPostop physiologic/metabolic derangements ––--

Postop respiratory failurePostop respiratory failure ±± ++ ±±

Postop thromboembolismPostop thromboembolism ++ ++ ±±

Postop sepsisPostop sepsis ––--

Accidental puncture or lacerationAccidental puncture or laceration

Transfusion reactionTransfusion reaction

Postop abdominopelvic wound dehiscencePostop abdominopelvic wound dehiscence

Birth traumaBirth trauma

Obstetric trauma – vaginal birth w instrumentationObstetric trauma – vaginal birth w instrumentation

Obstetric trauma – vaginal w/out instrumentationObstetric trauma – vaginal w/out instrumentation

Obstetric trauma – cesarean birthObstetric trauma – cesarean birth

Romano, 12/4/2006 42

Developing data on accuracy and relevance: Developing data on accuracy and relevance: AHRQ PSIs in Children’s HospitalsAHRQ PSIs in Children’s Hospitals

Sedman A, et al. Sedman A, et al. PediatricsPediatrics 2005;115(1):135-145 2005;115(1):135-145

PSIPSI No. reviewedNo. reviewed(total events)(total events)

PreventablePreventable(PPV %)(PPV %)

NonpreventableNonpreventable UnclearUnclear

Complications of anesthesia 74 (503) 11 (15%) 37 25

Death in low-mortality DRG 121 (1282) 16 (13%) 89 16

Decubitus ulcer 130 (2300) 71 (55%) 47 10

Failure to rescue 187 (5271) 15 (8%) 148 11

Foreign body left in 49 (235) 25 (51%) 14 10

Postop hemorrhage or hematoma 114 (1571) 40 (35%) 51 23

Iatrogenic pneumothorax 114 (1113) 51 (45%) 42 21

Selected infection 2° to med care 152 (7291) 63 (41%) 45 39

Postop DVT/PE 126 (1956) 36 (29%) 61 29Postop wound dehiscence 41 (232) 19 (46%) 16 6Accidental puncture or laceration 133 (4020) 86 (65%) 19 26

Romano, 12/4/2006 43

Do mortality measures have adequate discrimination? Do mortality measures have adequate discrimination? Minimum hospital volume to detect mortality doubling Minimum hospital volume to detect mortality doubling

((αα=0.05, =0.05, ββ=0.2)=0.2)

Dimick, et al. Dimick, et al. JAMA.JAMA. 2004;292:847-851. 2004;292:847-851.

NQF IOM/NHQR JCAHO NCQA

Usability

Can be used by at least one stakeholder audience for decision-making

Useful to supplement or enhance the accreditation process

Performance differences are statistically meaningful

Performance differences are clinically meaningful

Risk stratification or adjustment can be applied

Capacity to support subgroup analyses (under “Feasibility”)

Effective presentation and dissemination strategies exist

Can be interpreted by data users

Information about appropriate conditions is given

Methods for aggregating measure are defined

NQF IOM/NHQR JCAHO NCQA

Feasibility

Point of data collection tied to care delivery, when feasible

Logistically feasible

Timing and frequency of measure collection are specified

Benefit of measurement is evaluated against financial and administrative burden

Cost and burden of measurement

Data collection effort is assessed (availability, accessibility, effort, cost)

Reasonable cost

Auditing strategy is designed and can be implemented

Auditable

Confidentiality concerns can be addressed

Confidential

Public availability (access to measure construct and calculation algorithm)

Existence of prototypes (in use)

0 .0 00 0 00 0 .2 0 00 0 0 0. 40 0 00 0 0 .6 00 0 00 0 .8 0 00 0 0

Percen ta ge exc eptio n re porte d, A sthma

0. 30 0 00 0

0. 40 0 00 0

0. 50 0 00 0

0. 60 0 00 0

0. 70 0 00 0

0. 80 0 00 0

0. 90 0 00 0

1 .0 00 0 0 0

As

thm

a ac

hie

vem

en

t

Correlation: achievement against exception reporting, Asthma

Romano, 12/4/2006 47

Examples from the fieldExamples from the field

For more informationFor more informationGo to the National Quality Measures Clearinghouse at Go to the National Quality Measures Clearinghouse at

http://www.qualitymeasures.ahrq.gov/http://www.qualitymeasures.ahrq.gov/ What indicators have P4P programs used so far?What indicators have P4P programs used so far?

Medicaid managed care in WisconsinMedicaid managed care in WisconsinPremier Hospital Quality Incentive DemonstrationPremier Hospital Quality Incentive DemonstrationNCQA’s Bridges to ExcellenceNCQA’s Bridges to ExcellenceIntegrated HealthCare Association (CA)Integrated HealthCare Association (CA)Care-focused Purchasing coalitionCare-focused Purchasing coalition

Romano, 12/4/2006 48

Med-Vantage survey of P4P Med-Vantage survey of P4P programs in 2003 and 2004programs in 2003 and 2004

Romano, 12/4/2006 49

AHRQ Prevention Quality IndicatorsAHRQ Prevention Quality IndicatorsHighlighted measures recommended for state Medicaid programs by FACCTHighlighted measures recommended for state Medicaid programs by FACCT

Ambulatory care sensitive conditions (hospitalizations)Ambulatory care sensitive conditions (hospitalizations) DehydrationDehydration Bacterial pneumoniaBacterial pneumonia Urinary tract infectionUrinary tract infection AnginaAngina Adult asthma/Adult asthma/pediatric asthmapediatric asthma Chronic obstructive pulmonary diseaseChronic obstructive pulmonary disease Congestive heart failureCongestive heart failure Diabetes (short-term and long-term complications, uncontrolled)Diabetes (short-term and long-term complications, uncontrolled) Lower extremity amputation with diabetesLower extremity amputation with diabetes HypertensionHypertension Pediatric gastroenteritisPediatric gastroenteritis

Other avoidable conditionsOther avoidable conditions Perforated appendixPerforated appendix Low birth weightLow birth weight

Romano, 12/4/2006 50

Evaluating Medicaid managed care Evaluating Medicaid managed care programs in Wisconsinprograms in Wisconsin

% ICare Enrollees with CHF Hospitalized for CHF

20.8

15.1

0

5

10

15

20

25

1998 2000

Pe

rce

nt

% ICare Enrollees with Asthma Hospitalized for Asthma: 1998 & 2000

3.92.8

0

2

4

6

8

10

12

1998 2000

Pe

rce

nt

% ICare Enrollees with COPD Hospitalized for COPD

5.84.7

0

2

4

6

8

10

12

1998 2000

Pe

rce

nt

% ICare Enrollees with Diabetes Hospitalized for Diabetes: 1998 & 2000

2.9

2.5

0

0.5

1

1.5

2

2.5

3

3.5

1998 2000

Pe

rce

nt

Romano, 12/4/2006 51

Center for Medicare and Medicaid ServicesCenter for Medicare and Medicaid ServicesPremier Hospital Quality Incentive DemonstrationPremier Hospital Quality Incentive Demonstration

Premier Hospital Quality Incentive DemonstrationPremier Hospital Quality Incentive Demonstration33 measures for pay-for-performance within Premier, Inc.33 measures for pay-for-performance within Premier, Inc.Started with 27 NQF-endorsed measures, 4 PSI-based Started with 27 NQF-endorsed measures, 4 PSI-based

measuresmeasuresAdded CABG inpatient mortality and ASA at discharge, Added CABG inpatient mortality and ASA at discharge,

THA/TKA 30-day readmits; dropped use of IMA for CABGTHA/TKA 30-day readmits; dropped use of IMA for CABG266 hospitals accepted invitation to participate266 hospitals accepted invitation to participateHospitals performing in top two deciles received modest Hospitals performing in top two deciles received modest

bonus payments (2%/1%) in Year 2bonus payments (2%/1%) in Year 2Hospitals performing in bottom decile penalized in Year 3 Hospitals performing in bottom decile penalized in Year 3

Romano, 12/4/2006 53

NCQA’s Bridges to ExcellenceNCQA’s Bridges to Excellence

Romano, 12/4/2006 54

CA’s Integrated Healthcare AssociationCA’s Integrated Healthcare Association

Romano, 12/4/2006 55

CA’s Integrated Healthcare AssociationCA’s Integrated Healthcare Association

Copyright restrictions may apply.

Rosenthal, M. B. et al. JAMA 2005;294:1788-1793.

Romano, 12/4/2006 57

Care-focused PurchasingCare-focused PurchasingMercer Human Resources ConsultingMercer Human Resources Consulting

Clinical Quality: Structure, process and outcome based measures of safety, effectiveness, timeliness, and equity

Service Quality: (patient

experience) Survey based measures of patient experience and equity, i.e.timeliness, courtesy, respect, education, treatment options and risks, follow-up.

Efficiency: Risk-adjusted, longitudinal average and best practice total costs to achieve target levels of quality. Comparisons to among providers AND to other treatment options.

Clin

ica

l Qu

alit

y

Service Quality

Efficiency

Romano, 12/4/2006 58

Care-focused PurchasingCare-focused PurchasingVersion 1.0Version 1.0

Hospital Efficiency: Hospital Efficiency: Risk adjusted “proxy cost” per admission for acute APR-DRGsRisk adjusted “proxy cost” per admission for acute APR-DRGsHospital Quality:Hospital Quality: CMS “voluntary” measurementsCMS “voluntary” measurements Available JCAHO core measuresAvailable JCAHO core measures Leapfrog Group measuresLeapfrog Group measures State-specific hospital performance reporting programsState-specific hospital performance reporting programs AHRQ QIs where “warning label” is removedAHRQ QIs where “warning label” is removed Medpar complication rates for 53 hospital service lines (CACR, CareScience)Medpar complication rates for 53 hospital service lines (CACR, CareScience)Physician Efficiency:Physician Efficiency: Severity and risk-adjusted episode based resource consumption (Symmetry ETG)Severity and risk-adjusted episode based resource consumption (Symmetry ETG)Physician Quality:Physician Quality: Compliance with evidence based guidelines (ActiveHealth Management, RAND, or Compliance with evidence based guidelines (ActiveHealth Management, RAND, or

Resolution Health Inc.)Resolution Health Inc.) NCQA’s Physician Recognition Programs (PRP) in cardiovascular disease, diabetes, NCQA’s Physician Recognition Programs (PRP) in cardiovascular disease, diabetes,

and office systemnessand office systemness

Romano, 12/4/2006 59

Consider unintended consequencesConsider unintended consequences Quality-based purchasing is a potentially powerful tool to Quality-based purchasing is a potentially powerful tool to

stimulate behavior change among providersstimulate behavior change among providers You will get what you pay for – make sure that’s what you want!You will get what you pay for – make sure that’s what you want! Perverse incentives to improve “measures” without actually Perverse incentives to improve “measures” without actually

improving quality of care (e.g., survival with poor quality of life, improving quality of care (e.g., survival with poor quality of life, survival to discharge with death a week later, selection of low-survival to discharge with death a week later, selection of low-risk patients, avoidance of high-risk patients, switch high-risk risk patients, avoidance of high-risk patients, switch high-risk cases to uncovered settings)cases to uncovered settings)

Perverse incentives to improve measured variables without Perverse incentives to improve measured variables without improving unmeasured variablesimproving unmeasured variables

““Free ride” versus “Sisyphus syndrome” – keep “raising the bar” Free ride” versus “Sisyphus syndrome” – keep “raising the bar” but not too high too quicklybut not too high too quickly

Romano, 12/4/2006 60

Early results of NHS reforms – Scotland% of maximum available points scored

010

2030

4050

% o

f pra

ctic

es

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100Total points scored

Romano, 12/4/2006 61

Measurement for quality-based purchasing:Measurement for quality-based purchasing:Conclusions and recommendationsConclusions and recommendations

QBP aligns incentives so providers are motivated to do what’s QBP aligns incentives so providers are motivated to do what’s right: improve quality, reduce disparities, improve IT, teamworkright: improve quality, reduce disparities, improve IT, teamwork

Select measures based on local priorities and available/obtainable Select measures based on local priorities and available/obtainable surveillance datasurveillance data

Consider your key audiences and objectives. How important is Consider your key audiences and objectives. How important is provider buy-in? How important is purchaser buy-in?provider buy-in? How important is purchaser buy-in?

Consider private feedback before public reporting and QBPConsider private feedback before public reporting and QBP Define and collect measures in a manner that earns the confidence Define and collect measures in a manner that earns the confidence

of key stakeholders (definitions manual, auditing, monitoring of key stakeholders (definitions manual, auditing, monitoring undesirable consequences, maximizing transparency)undesirable consequences, maximizing transparency)

Outcome measures: consider stratification/risk-adjustmentOutcome measures: consider stratification/risk-adjustment Process measures: consider eligibility criteriaProcess measures: consider eligibility criteria

Romano, 12/4/2006 62

Measurement for quality-based purchasing:Measurement for quality-based purchasing:Designing a measure setDesigning a measure set

Select enough measures to represent multiple domains of Select enough measures to represent multiple domains of care, but not so many that providers are overwhelmedcare, but not so many that providers are overwhelmed

Weight measures according to importance – but think about Weight measures according to importance – but think about how much effort will be required of providershow much effort will be required of providers

Think incrementally – start small (where you can get “most Think incrementally – start small (where you can get “most bang for the buck”), build up, improve data quality as you gobang for the buck”), build up, improve data quality as you go

Don’t reinvent the wheel – use existing measures if possible, Don’t reinvent the wheel – use existing measures if possible, but be a pioneer if you need tobut be a pioneer if you need to

Involve multiple stakeholders, listen to everyoneInvolve multiple stakeholders, listen to everyone Use more measures, cross-cutting measures, pooled data for Use more measures, cross-cutting measures, pooled data for

evaluation at physician/practice level (vs. group/plan level)evaluation at physician/practice level (vs. group/plan level)

“I think that I should warn you that the flip side of our generous bonus

incentive scheme is capital punishment”