The Use of Funnel Plots & Multi-Year Cumulative Data to

Track Hospital Performance

Herbert MA, Hamman BH, Roper KL, Ring WS, Edgerton JR, Texas Quality Initiative

The American Association of Thoracic SurgeonsApril 26, 2015

Seattle, Washington

Nothing to Disclose

The Texas Quality Initiative• 27 Hospitals in North Texas agreed to share

clinical and administrative data• All participated in the STS Database• 26,634 cardiac procedures from 1/2008 -12/2012• 13,379 isolated CABG were analyzed for observed

to expected (O/E) operative mortality• There was a need to graphically represent the

data– Simple– Easy to understand

The Funnel Plot

Methods

• A funnel plot is centered on a benchmark with 95% confidence intervals drawn on the graph.

• To assess operative mortality and allow for risk correction, the observed to expected (O/E) ratio is used.

• The case volume is plotted on the horizontal axis • O/E ratio on the vertical axis; • either annual data or multi-year data can be

shown.

The Funnel Plot• X Axis: Volume of Cases (CABG)• Y Axis: O/E Ratio for isolated CABG• An O/E of 1 is expected• 95% Confidence intervals surround “1”

The Funnel Plot

Worse than 1, but not statistically different

Better than 1, but not statistically different

Outlier for poor performance

Outlier for good performance

The Funnel Plot• At low volume it is very hard to become an outlier• In fact, at less than 200 cases, you cannot become an outlier for

good performance• The Problem:• Most hospitals analyze their data on an annual basis• Most Hospitals do less than 200 cases per year • They cannot reveal themselves as outlier due to wide confidence interval at low volume• Year after year they find that their results are “OK”

The Funnel Plot

PROBLEMNo Problem

How can we account for this problem?

• Plotting running totals moves the result to the right, where the funnel is narrower

• The results can begin to show statistically significant differences from “1”

• More important…..• Trends become visually apparent• Poorly performing hospitals can be identified– Even before the results reach statistical significance– Urgent interventions can be put in place

Plotting Running 5 Year Totals

Year

1

Year

1 + 2

+ 3

Year

1+ 2

Year

1 +

2 +

3 +

4Ye

ar 1

+ 2 +

3 + 4

+ 5

Plotting Running 5 Year Totals Year 6

Year

2

Year

2 + 3

+ 4

Year

2+ 3

Year

2 + 3

+ 4 +

5Ye

ar 2

+ 3 +

4 + 5

+ 6

How Does This Help

• Let’s see some examples from the TQI Data• These are real data from real hospitals• Some of the examples are from different time

intervals, because…• I Selected graphs to illustrate different scenarios• Colored dots represent annual data• Green line represents the running 5 year total

Even with excellent outcomes, a hospital with case volumes under 200 cannot become an outlier for good performance

However, Cumulative data will reveal excellence (in 1 more year)

Annual Data tightly clustered: O/E doesn’t change muchWith Cumulative Data …

The Curve is flat, but at higher volume becomes an outlier

Annual Data is all within the funnelHospital perceives “No Problem”

Cumulative Data Unmasks Outlier for Poor Performance

Annual Data is Inconclusive: 3 out of 5 years are within the funnelCumulative Data Slope of the Curve is Predictive of Poor Performance

At Year 3 Intervention is Needed, This is even before hospital becomes an outlier in year 5

Annual Data: 4 of 5 years O/E is > 1Cumulative Data Shows a downward slopeWe have no concerns about this hospital

Conclusions• The use of funnel plots allows easy comparison

of individual programs• Analyzing only annual data can lead to a false

sense of satisfaction • The plotting of a five year running total will

provide sufficient volume to reveal an accurate assessment

• the trend (slope) may give an indication of effectiveness of quality improvement programs in place.