Long-Term Outcomes After Severe Sepsis

Please Stop Wasting RCT DataTheodore J. Iwashyna, MD, PhDUniversity of MichiganAnn Arbor VA Center for Clinical Management Researchon sabbatical at ANZIC-RC at Monash University2 December 2015 Victorian Intensive Care Network

116 July 20159th Alfred ICU Adanced Mechanical Ventilation Conference

DisclosuresKey Funding:U.S. NIH K08 HL091249 (TJI)U.S. NIH R21 AG044752 (TJI)U.S. VA HSR&D IIR 11-109 (TJI)University of Michigan (for sabbatical funds)

Much of this is joint work with Jim Burke, Jeremy Sussman, Hallie Prescott, Rod Hayward, and Derek Angus. It would have been impossible without them. Errors are, however, mine.

This work does not necessarily represent the views of the U.S. Government or Department of Veterans Affairs

I have no relevant financial conflicts of interest to disclose

This talk is based on PMID: 26177009

The Simple Dream:I will provide this treatment if the benefits outweigh the harms.

Net Benefit = Benefit Harm

If Net Benefit > 0, I treat.

Guyatt et al (1994) JAMA 271:59.

Guyatt et al (1994) JAMA 271:59.

Courtesy of HC Prescott; North American Symptomatic Carotid Endarterctomy Trial Collaboration (1991) NEJM 325:445.

Rothwell (1995) The Lancet 345:8965.

The Simple Dream:If Net Benefit > 0, I treat.

If a patient would have been enrolled in a clinical trial, then my best guess should be thatNet Benefit > 0 The Extenders:If Net Benefit > 0, I treat.

If a patient would have been enrolled in a clinical trial, then we need more information to know if Net Benefit > 0

The Simple Dream:If Net Benefit > 0, I treat.

If a patient would have been enrolled in a clinical trial, then my best guess should be thatNet Benefit > 0 The Extenders:If Net Benefit > 0, I treat.

If a patient would have been enrolled in a clinical trial, then we need more information to know if Net Benefit > 0

The Implications of Heterogeneity of Treatment Effect by Baseline Risk

Why we should just about always expect HTEHTE and positive trials in acute respiratory failureHTE and negative trials in acute respiratory failureBut, maybeSo what the heck am I supposed to do with this?

Risk of DeathIf Never TreatedUntreated Risk of Death

Risk of DeathIf Never TreatedUntreated Risk of Death

Risk of DeathIf Treated (and no side-effects)Untreated Risk of Death

Risk of DeathIf Never TreatedUntreated Risk of Death

Risk of DeathIf Treated (and no side-effects)Untreated Risk of Death

Risk of DeathIf Never TreatedUntreated Risk of Death

Risk of DeathIf Treated (and no side-effects)Untreated Risk of Death

Reduction in Risk of DeathAbsolute Mortality Benefitof Treatment, assuming noside effectsUntreated Risk of Death

Risk of DeathIf Never TreatedUntreated Risk of Death

Risk of DeathIf Treated (and no side-effects)Untreated Risk of Death

Reduction in Risk of DeathAbsolute Mortality Benefitof Treatment, assuming noside effectsUntreated Risk of Death

Risk of DeathSide Effect Risk ofTreatmentUntreated Risk of Death

Risk of DeathPutting it all togetherUntreated Risk of Death

Absolute Mortality Benefitof Treatment, assuming noside effectsSide Effect Risk ofTreatment

Untreated Risk of Death

AA: Clearly good, these people should get this

3 Domains of Net Benefit

Untreated Risk of Death

Untreated Risk of Death

ABA: Clearly good, these people should get thisB: Clearly bad, these people should not get this3 Domains of Net Benefit

Untreated Risk of Death

Untreated Risk of Death

AB

Untreated Risk of Death

CA: Clearly good, these people should get thisB: Clearly bad, these people should not get thisC: Uncertain, requires a conversation3 Domains of Net Benefit

Untreated Risk of Death

Untreated Risk of Death

AB

Untreated Risk of Death

CA: Clearly good, these people should get thisB: Clearly bad, these people should not get thisC: Uncertain, requires a conversationKey question at the bedside:For this patient, for this treatment, where are we?

This all hinges on there being a big distributionof baseline risk. Are my patients that heterogeneous?

US Veterans AffairsNon-Post-Operative Mech Vent

Austalian APDNon-Post-Operative Mech Ventw/ LOS>24h, not Drug Overdose

The Implications of Heterogeneity of Treatment Effect by Baseline Risk

Why we should just about always expect HTEHTE and positive trials in acute respiratory failureHTE and negative trials in acute respiratory failureBut, maybeSo what the heck am I supposed to do with this?

Simulating a new therapy20% relative risk reduction if no adverse events

Risk of DeathIf Treated (and no adverse events)Untreated Risk of Death

Simulating a new therapy20% relative risk reduction if no adverse events3% risk of adverse events (known and unknown) if treated

Risk of DeathIf Treated (and no adverse events)Untreated Risk of Death

Risk of DeathAdverse EventRisk of TreatmentUntreated Risk of Death

Simulating a new therapy20% relative risk reduction if no adverse events3% risk of adverse events (known and unknown) if treatedN=2,500, run in an acute respiratory failure population

Risk of DeathIf Treated (and no adverse events)Untreated Risk of Death

Risk of DeathAdverse EventRisk of TreatmentUntreated Risk of Death

Simulating a new therapy20% relative risk reduction if no adverse events3% risk of adverse events (known and unknown) if treatedN=2,500, run in an acute respiratory failure population

Observed RRR = 0.85 (95% CI: 0.77, 0.94)Absolute risk reduction from 39.8% to 33.6%

Wooohooo!

Risk of DeathIf Treated (and no adverse events)Untreated Risk of Death

Risk of DeathAdverse EventRisk of TreatmentUntreated Risk of Death

87% power9th Alfred ICU Adanced Mechanical Ventilation Conference16 July 201527

Simulating a new therapy20% relative risk reduction if no adverse events3% risk of adverse events (known and unknown) if treatedN=2,500, run in an acute respiratory failure population

Observed RRR = 0.85 (95% CI: 0.77, 0.94)Absolute risk reduction from 39.8% to 33.6%

NNT in highest risk = 8NNT in decile 2 = 90Net harm in lowest risk patients.

The Implications of Heterogeneity of Treatment Effect by Baseline Risk

Why we should just about always expect HTEHTE and positive trials in acute respiratory failureHTE and negative trials in acute respiratory failureBut, maybeSo what the heck am I supposed to do with this?

Simulating a new therapy15% relative risk reduction if no adverse events

Risk of DeathIf Treated (and no adverse events)Untreated Risk of Death

Simulating a new therapy15% relative risk reduction if no adverse events3% risk of adverse events (known and unknown) if treated

Risk of DeathIf Treated (and no adverse events)Untreated Risk of Death

Risk of DeathAdverse EventRisk of TreatmentUntreated Risk of Death

Simulating a new therapy15% relative risk reduction if no adverse events3% risk of adverse events (known and unknown) if treatedN=2,500, run in an acute respiratory failure population

Risk of DeathIf Treated (and no adverse events)Untreated Risk of Death

Risk of DeathAdverse EventRisk of TreatmentUntreated Risk of Death

Simulating a new therapy15% relative risk reduction if no adverse events3% risk of adverse events (known and unknown) if treatedN=2,500, run in an acute respiratory failure population

45% of these trials are now negative (p>0.05 for differences between treated & controls)

Risk of DeathIf Treated (and no adverse events)Untreated Risk of Death

Risk of DeathAdverse EventRisk of TreatmentUntreated Risk of Death

Simulating a new therapy15% relative risk reduction if no adverse events3% risk of adverse events (known and unknown) if treatedN=2,500, run in an acute respiratory failure population

45% of these trials are now negative (p>0.05 for differences between treated & controls)

But, NNT in highest risk = 9NNT in 2nd highest risk = 14

Untreated Risk of Death

2 RCTs:SameNet Benefit ProfilebutModest DifferencesIn Baseline RiskAmong Enrolled

Risk of DeathUntreated Risk of Death

AB

Yay!Positive TrialSad!Negative TrialKent et al (2010) Trials 11:85; see also Hayward et al (2005) Health Affairs 24:1571. Kent et al (2010) Trials 11:85.

2 RCTs:SameNet Benefit ProfilebutModest DifferencesIn Baseline RiskAmong Enrolled

The Implications of Heterogeneity of Treatment Effect by Baseline Risk

Why we should just about always expect HTEHTE and positive trials in acute respiratory failureHTE and negative trials in acute respiratory failureBut, maybeSo what the heck am I supposed to do with this?

Arent people doing this already?

replicating Scenario #1, but with a strongly uneven adverse event ratean adverse event rate that was 0.5 in 100 for patients with a baseline risk of death of zero and increased linearly to a rate of 8 in 100 for those with a baseline risk of death of 1.0, resulting in an average adverse event rate of 3.5 in 100 (approximately that in Scenario #1). This resulted in trials with a median RR of 0.85 (95% CI: 0.77, 0.94) and 87% power. 9th Alfred ICU Adanced Mechanical Ventilation Conference16 July 201538

Arent people doing this already?

Sort of. ICNARC does it as part of their trials, hidden in the online Appendices. Not so much others.If you find others, please tell me!

replicating Scenario #1, but with a strongly uneven adverse event ratean adverse event rate that was 0.5 in 100 for patients with a baseline risk of death of zero and increased linearly to a rate of 8 in 100 for those with a baseline risk of death of 1.0, resulting in an average adverse event rate of 3.5 in 100 (approximately that in Scenario #1). This resulted in trials with a median RR of 0.85 (95% CI: 0.77, 0.94) and 87% power. 9th Alfred ICU Adanced Mechanical Ventilation Conference16 July 201539

But in reality, adverse events are not perfectly even. Sicker patients have more adverse events.

NNT in highest risk: 7NNT in lowest risk: 74

replicating Scenario #1, but with a strongly uneven adverse event ratean adverse event rate that was 0.5 in 100 for patients with a baseline risk of death of zero and increased linearly to a rate of 8 in 100 for those with a baseline risk of death of 1.0, resulting in an average adverse event rate of 3.5 in 100 (approximately that in Scenario #1). This resulted in trials with a median RR of 0.85 (95% CI: 0.77, 0.94) and 87% power. 9th Alfred ICU Adanced Mechanical Ventilation Conference16 July 201540

But in reality, adverse events are not perfectly even. Sicker patients have more adverse events.

NNT in highest risk: 7NNT in lowest risk: 74

replicating Scenario #1, but with a strongly uneven adverse event ratean adverse event rate that was 0.5 in 100 for patients with a baseline risk of death of zero and increased linearly to a rate of 8 in 100 for those with a baseline risk of death of 1.0, resulting in an average adverse event rate of 3.5 in 100 (approximately that in Scenario #1). This resulted in trials with a median RR of 0.85 (95% CI: 0.77, 0.94) and 87% power. 9th Alfred ICU Adanced Mechanical Ventilation Conference16 July 201541

But in reality, many of the things that kill patients are not even potentially responsive to treatment.

But if there is too much non-responsive risk, it becomes really hard to have a positive trial.

Holds adverse event rate constant at 3%9th Alfred ICU Adanced Mechanical Ventilation Conference16 July 201542

But in reality, many of the things that kill patients are not even potentially responsive to treatment.

But if there is too much non-responsive risk, it becomes really hard to have a positive trial.

Fraction of Risk that is Treatment-ResponsiveFraction of Risk from Other Causes of DeathTreatment-Responsive Relative Risk Reduction100%0%20%75%25%27.5%50%50%40%25%75%80%

Holds adverse event rate constant at 3%9th Alfred ICU Adanced Mechanical Ventilation Conference16 July 201543

Highest RiskLowest RiskDecile of Baseline Risk for Death100%-640239121734937261912675%107115190594434261913650%28410366463729232313825%13165453629231915131412345678910

Proportion of Risk that isTreatment ResponsiveBased on data visualization courtesy of HC Prescott.Kent et al (2010) Trials 11:85.

Holds adverse event rate constant at 3%9th Alfred ICU Adanced Mechanical Ventilation Conference16 July 201544

Highest RiskLowest RiskDecile of Baseline Risk for Death100%-640239121734937261912675%107115190594434261913650%28410366463729232313825%13165453629231915131412345678910

Proportion of Risk that isTreatment ResponsiveBased on data visualization courtesy of HC Prescott.Kent et al (2010) Trials 11:85.

But in reality, many of the things that kill patients are not even potentially responsive to treatment.

Even when this is true, and you have an incredibly potent therapy, there is still substantial variability in NNT in positive trials.

Holds adverse event rate constant at 3%9th Alfred ICU Adanced Mechanical Ventilation Conference16 July 201545

But Australia has many very low risk patientsits ICU population is way more skewed than that VA data you showed.

replicating Scenario #1, but with a strongly uneven adverse event ratean adverse event rate that was 0.5 in 100 for patients with a baseline risk of death of zero and increased linearly to a rate of 8 in 100 for those with a baseline risk of death of 1.0, resulting in an average adverse event rate of 3.5 in 100 (approximately that in Scenario #1). This resulted in trials with a median RR of 0.85 (95% CI: 0.77, 0.94) and 87% power. 9th Alfred ICU Adanced Mechanical Ventilation Conference16 July 201546

But Australia has many very low risk patientsits ICU population is way more skewed than that VA data you showed.

The more uneven the distribution, the worse the problem.

replicating Scenario #1, but with a strongly uneven adverse event ratean adverse event rate that was 0.5 in 100 for patients with a baseline risk of death of zero and increased linearly to a rate of 8 in 100 for those with a baseline risk of death of 1.0, resulting in an average adverse event rate of 3.5 in 100 (approximately that in Scenario #1). This resulted in trials with a median RR of 0.85 (95% CI: 0.77, 0.94) and 87% power. 9th Alfred ICU Adanced Mechanical Ventilation Conference16 July 201547

The Implications of Heterogeneity of Treatment Effect by Baseline Risk

Why we should just about always expect HTEHTE and positive trials in acute respiratory failureHTE and negative trials in acute respiratory failureBut, maybeSo what the heck am I supposed to do with this?

Storming of the Bastille, by Jean-Pierre-Louis-Laurent Houel, from Wikipedia.org.

Our journals are letting us down.

RCTs should, at a minimum, be published with subgroup analyses by baseline risk of death.

These should be interpreted cautiously, like any subgroup, but with a high prior likelihood of variation in effect size.

Demand better!

Overall baseline risk, not just specific physiology, fundamentally shapes each patients opportunity for benefit from any therapy.

Higher risk patients may often benefit substantially more from our therapies than low risk patientseven if high risk patients die more often anyway.

Our bedside psychology (availability & salience biases) may mislead us, emphasizing the deaths despite therapy in high risk patients more than the saves because of it and overemphasizing the saves despite therapy in low risk groups.

Be willing to withhold indicated therapy in very low risk patients, or in modestly low risk patients with higher likelihoods of adverse events.

This is not an excuse to willy-nilly ignore RCTs & guidelines.

Untreated Risk of Death

Untreated Risk of Death

AB

Untreated Risk of Death

CKey question at the bedside:For this patient, for this treatment, where are we?

Please email me at jack.iwashyna.on.sabbatical @ gmail.com for copies of my slides or to continue a conversation. I often tweet @iwashyna.