Leapfrog’s “Survival Predictor”: Composite Measures for Predicting

Hospital Surgical Mortality

May 7, 2008

2

Townhall Call Overview• Introductions• Need for better measures• What is the “Survival Predictor”?• Methods• Reporting to Leapfrog & Scoring Details• Resources• Q & A

3

Need for a better measures

• Individual quality measures have limitations – Mortality rates are often too “noisy” – Hospital volume is a weak proxy for performance

• Growing interest in composite measures but methods are not well established– Not clear on how to best weight measures when creating

composite or interpret them when they conflict

4

What is the “Survival Predictor”?

• A simple composite score based on a combination of surgical mortality and hospital volume

• Designed to optimize the prediction of future risk-adjusted mortality based on these two inputs

• Applicable to six of Leapfrog’s EBHR high-risk surgeries (bariatric surgery excluded)

5

Validation Study:Data Sources & Study Population

• Used data from national Medicare population (MEDPAR files)

• Identified all patients aged 65 to 99 undergoing six operations– Coronary artery bypass– Percutaneous coronary interventions– Abdominal aortic aneurysm repair– Aortic valve replacement– Pancreatic cancer resection– Esophageal cancer resection

6

Development of Measure

• Used empirical Bayes approach to combine mortality rates with hospital volume

• Observed mortality rate is weighted according to how reliably it is estimated, with the remaining weight placed on the information regarding hospital volume

Composite mortality prediction = (weight)*(observed mortality) + (1-weight)*(volume-predicted mortality)

7

• Weight = Reliability (precision) or the mortality measure

• More weight placed on observed mortality rate when a hospital has higher number of cases (estimated with more reliability)

• Less weight on observed mortality rate for lower number of cases (lower reliability)

Development of Measure

8

Validation of Composite Measure

Determine value of measure:

Step 1: Explaining variation in risk-adjusted mortality

Technical implications: “Is it a better measure?”

Step 2: Prediction of future performance

Practical implications: “How much can I improve my chances by choosing this hospital?”

9

Table 1. Hospital caseload and the weight applied to each of the two inputs in the composite measure. The weight applied to the mortality rate is the reliability and the weight applied to hospital volume is 1-reliability.

Hospital caseloads, mean (SD)

Weight applied to observed mortality,

mean (SD)

Weight applied to hospital volume,

mean (SD)

Coronary artery bypass grafting 321 (319) .46 (.21) .54 (.21)

Aortic valve replacement 59 (72) .26 (.17) .73 (.17)

Percutaneous coronary interventions 442 (499) .48 (.27) .52 (.27)

Elective abdominal aortic aneurysm repair 31 (46) .21 (.18) .79 (.18)

Pancreatic cancer resection 4 (10) .23 (.17) .77 (.17)

Esophageal cancer resection 4 (8) .14 (.13) .86 (.13)

10

Table 2. Relative ability of each measure to explain hospital-level differences in risk-adjusted mortality rates

Proportion of hospital-level variation in mortality rates explained by each performance measure (%)

Hospital volume Observed mortality Simple composite measure

Coronary artery bypass grafting

7 44 55

Aortic valve replacement

22 30 45

Percutaneous coronary interventions

37 53 71

Elective abdominal aortic aneurysm repair

58 14 64

Pancreatic cancer resection

69 19 76

Esophageal cancer resection

65 14 70

11

Table 3. Relative ability of historical measures (2003-04) to predict subsequent risk-adjusted mortality (2005-2006).

Adjusted odds ratio for risk-adjusted mortality (2003-04), best vs. worst quartile (95% CI)

Hospital volume alone (2003-04)

Observed mortality alone (2003-04)

Simple composite measure

(2003-04)

Coronary artery bypass grafting

1.15(1.04-1.27) 1.71 (1.57-1.87) 1.72 (1.57-1.88)

Aortic valve replacement

1.48 (1.29-1.67) 1.73 (1.54-1.94) 1.97 (1.75-2.23)

Percutaneous coronary interventions

1.49 (1.35-1.64) 1.84 (1.69-2.01) 1.95 (1.79-2.13)

Abdominal aortic aneurysm repair

1.48 (1.26-1.73) 1.20 (1.05-1.38) 1.69 (1.44-1.98)

Pancreatic cancer resection

3.93 (2.85-5.42) 1.36 (1.05-1.76) 4.23 (3.07-5.84)

Esophageal cancer resection

2.89 (2.15,3.89) .90 (.72,1.12) 2.71 (2.03-3.60)

12

Future risk-adjusted mortality rates (2003-04) for quartiles of hospital rankings based on historical (2005-06) hospital volume, risk-adjusted mortality rates, and composite measures. Future risk-adjusted mortality is shown for the composite measure created using both risk-adjusted and unadjusted mortality rates.

Esophagectomy

13.1

10.0 10.3

4.9

10.5

8.3

5.2

11.5

13.2

11.1

8.6

5.3

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

Quartiles of Performance Measures (2003-2004)

Ris

k A

djus

ted

Mor

talit

y 05

-06

(%)

Hospital volume Historical mortality Composite

Abdominal Aortic Aneurysm Repair

4.0

3.43.2

2.7

4.1

3.0 2.9

3.4

4.4

3.33.1

2.6

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Quartiles of Performance Measures (2003-2004)

Ris

k A

djus

ted

Mor

talit

y 05

-06

(%)

Hospital volume Historical mortality Composite

13

Risk-Adjustment

• Evaluated the extent to which risk-adjustment is important in improving the predictive ability of the composite measure

• Risk-adjustment performed using a logistic regression to estimate expected mortality rates (age, gender, race, urgency of operation, income, & coexisting diseases)

• Compared the ability of risk-adjusted and unadjusted composite measures to predict subsequent performance

14

Conclusions:

• Simple composite measures are better at explaining variations in mortality and predicting future performance

• Empirical Bayes approach is a valid and useful method for combining multiple domains of quality

• Composite measures based on unadjusted and risk-adjusted mortality rates were equally good at predicting future risk-adjusted mortality rates

15

Future Steps

• Incorporate other inputs into composite measure– e.g., outcomes with other related procedures

• Perform validation study for bariatric surgery

16

Reporting to Leapfrog

• For each of the six EBHR surgeries, hospitals asked to report on hospital volume and number of patient deaths following procedure

• No additional burden as compared to 2007 Leapfrog Hospital Survey

17

Scoring Details

• A hospital’s result on the composite measure for each procedure is publicly released and displayed on Leapfrog’s website in one of five categories:

• Best Chance of Survival (full circle) means the hospital is in the best quartile for the composite measure for that procedure.

• Good Chance of Survival (3/4-circle) means the hospital is above the midpoint (median), but not in the best quartile for the composite measure for that procedure.

• Fair Chance of Survival (1/2-circle) means the hospital is below the midpoint (median), but not in the worst quartile for the composite measure for that procedure.

• Worst Chance of Survival (1/4-circle) means the hospital is in the worst quartile for the composite measure for that procedure.

• Did not disclose this information (empty circle) means the hospital did not respond to this section of the survey, or the hospital was asked to complete the survey but has not submitted one.

• N/A -- Standard does not apply means the hospital does not perform the procedure electively.

• Scoring based on rank relative to all U.S. hospitals

• Results will be posted on Leapfrog’s website in early Fall

18

Resources

• White paper available on Leapfrog’s website (under “News & Events”)

• Recording of today’s townhall call is available for one month after today’s call

19

Questions?