Systematic Reviews of the Literature and Meta-analyses: ….problems or panacea? Daren K. Heyland, MD, FRCPC, MSc Queen’s University, Kingston, Ontario

Systematic Reviews of the Literature and Meta-analyses:

….problems or panacea?

Daren K. Heyland, MD, FRCPC, MSc

Queen’s University, Kingston, Ontario

www.criticalcarenutrition.com

Updated Jan 2009

Summarizes >200 trials studying 21283 patients

34 topics 17 recommendations

Clinical Practice Guidelines

Validity Homogeneity

SafetyFeasibility

Cost

evidence integration of values+

practiceguidelines

In Search of Truth...…Does it work?

Begins with a hypothesis or question Does Drug X reduce the incidence of problem Y in patients

with condition Z

Application of experimental or observational methods to determine the answer

Results of our observations leads to conclusions that are correct (truth) or incorrect (due to bias or chance)

Levels of Evidence

Systematic reviews RCT’s Cohort Studies Case Control Case Series

less bias/strong inferences

more bias/weaker inferences

198 RCT’s Reviewed in Critical Care Nutrition Guidelines

PLOS 2008;5: e4

Will be able to appraise and incorporate results of systematic reviews into clinical decision making.

understand the role of systematic reviews in research and policy settings.

List the strengths and weakness of meta-analyses

Learning Objectives

Overview

Definition and Classification Usefulness Methodological Quality Making Inferences Conclusions

Systematic Review…

Form of scientific investigation to assess the effectiveness of healthcare interventions

Integrative research Subjects= original or primary studies Employs methods that limit bias and reduce random

error

Feature NarrativeReview

SystematicReview

Question No specific question,usually broad in scope

Focused clinicalquestion

Search Not usually specified,potentially biased

Comprehensive, explicitstrategy

Selection Not usually specified,potentially biased

Criterion-basedselection

Appraisal Variable Rigorous criticalappraisal

Synthesis Qualitative Qualitative + Quantitative

Inferences Sometimes evidence-based

Evidence-based

Systematic Reviews and Meta-analysis

Narrative Reviews

Systematic Reviews

Meta-analysis

Number of Systematic Reviews Published

The Frailties of Narrative Reviews

If the original studies of thrombolytics therapies had been subject to a systematic review, the treatment effect would have been apparent in the 1970s instead of 1980s.

Narrative reviews omitted effective therapies and endorsed ineffective therapies.

Antman JAMA1992;268;240 and Lau NEJM 1992;327:248

Clinical Decision Making and Systematic Reviews

Case Scenario 77 y.o. male with presumptive Dx of Urosepsis PMHX: MI, Prostate BMI 21 After initial resuscitation

FiO2 = 100%, PO2 = 55 MAP = 65, CVP 13, levophed 20 mcg/kgk/min rising Cr, 20 ml of urine, acidemic High NG drainage

Going to start on EN but not likely to tolerate Role for early supplemental PN?

Clinical Decision Making and Systematic Reviews

Problem 100s of citations across scores of journals published over

the last 20 years In diverse patient populations or diverse settings with variable or inconsistent results!

How do you make sense of this all?

Impact of Caloric Debt

Caloric debt associated with:

Longer ICU stay

Days on mechanical ventilation

Complications

Mortality

Adequacy of EN

Rubinson CCM 2004; Villet Clin Nutr 2005; Dvir Clin Nutr 2006; Petros Clin Nutr 2006

0200

400600

8001000

12001400

16001800

2000

1 3 5 7 9 11 13 15 17 19 21

Days

kcal

Prescribed Engergy

Energy Received From Enteral Feed

Caloric Debt

2007 International Nutrition Practice Survey

Point prevalence survey of nutrition practices in ICU’s around the world conducted Jan. 27, 2007

Enrolled 2772 patients from 158 ICU’s over 5 continents

Included ventilated adult patients who remained in ICU >72 hours

HypothesisHypothesis

There is a relationship between amount of energy and protein received and clinical outcomes (mortality and # of days on ventilator)

The relationship is influenced by nutritional risk BMI is used to define chronic nutritional risk

What Study Patients Actually Rec’d

Average Calories in all groups: 1034 kcals and 47 gm of protein

Result: Average caloric deficit in Lean Pts:

7500kcal/10days Average caloric deficit in Severely Obese:

12000kcal/10days

Relationship Between Increased Calories and 60 day Mortality

BMI Group Odds Ratio

95% Confidence

Limits

P-value

Overall 0.76 0.61 0.95 0.014

<20 0.52 0.29 0.95 0.033

20-<25 0.62 0.44 0.88 0.007

25-<30 1.05 0.75 1.49 0.768

30-<35 1.04 0.64 1.68 0.889

35-<40 0.36 0.16 0.80 0.012

>=40 0.63 0.32 1.24 0.180

Legend: Odds of 60-day Mortality per 1000 kcals received per day adjusting for nutrition days, BMI, age, admission category, admission diagnosis and APACHE II score.

RESULTS: WHO IS AT RISK?

RCT Level of Evidence that More EN= Improved Outcomes

RCTs of aggressive feeding protocols Results in better protein-energy intake Associated with reduced complications and improved survival

Taylor et al Crit Care Med 1999; Martin CMAJ 2004

Meta-analysis of Early vs Delayed EN Reduced infections: RR 0.76 (.59,0.98),p=0.04 Reduced Mortality: RR 0.68 (0.46, 1.01) p=0.06


More is Better!

Our Field of DreamIf you feed them (better!)

They will leave (sooner!)

ICU patients are not all created equal…should we expect the impact of nutrition therapy to be the

same across all patients?b

What if you can’t provide adequate nutrition enterally?

… to TPN or not to TPN,

that is the question!

Current practice in nutritional support in septic patients: Results of national,

prospective multicenter German Study

Point prevalence study 454 ICUs from 310 hospitals

in Germany 399 patients septic patients

included Median APACHE II 26 68% had no GI pathology 46% in shock Overall mortality 55.2%

EN onlyPN onlyEN +PNnone

Elke CCM 2008;36:1762

Current practice in nutritional support in septic patients: Results of national,

prospective multicenter German Study

Point prevalence study 454 ICUs from 310 hospitals

in Germany 399 patients septic patients

included Median APACHE II 26 68% had no GI pathology 46% in shock Overall mortality 55.2%

0

10

20

30

40

50

60

70

EN only PN only EN +PN

none

%mortality

P=0.005

Multivariate analysis:

PN independent predictor for mortality

(OR 2.09, 95% CI 1.29-3.37)

Early Supplemental PN is Associated with Increased Infection in

Critically Ill Trauma Patients

Retrospective, multicenter, cohort study of 597 severely injured patients

Compared those that rec’d PN within 7 to those who did not.

Also compared early PN group to subgroup of ‘EN tolerant’ (tolerated 1000 kcal any day during first week)

Adjusted for differences in key baseline demographics

Sena J Am Coll Surg 2008;207:459

Early Supplemental PN is Associated with Increased Infection in Critically Ill Trauma Patients

Differences not due to differences in glycemic control

No Early PN Early PN Odds RatioP value

Overall Adjusted

Nosocomial Infections 27% 56% 2.1 (1.3-3.5)P=0.003

Late ARDS 1% 8% 3.4 (1.0-11.0)P=0.04

Death 8% 23% 1.5 (0.8-3.0)P=0.24

EN tolerant analysis

Nosocomial Infections 42% 69% 2.5 (1.1-5.9)P=0.03

Late ARDS 2% 9% 5.4 (1.1-27.4)P=0.04

Death 8% 19% 2.7 (0.8-9.3)P=0.10

Prospective Studies of Supplemental PNEffect on Mortality


What if you can’t provide adequate nutrition enterally?

… to TPN or not to TPN,

that is the question!

Maximize EN delivery prior to initiating PN

Use of Supplemental PN in Sepsis?

Results of meta-analysis Results of single RCTs of Septic Patients Results of observational studies Consideration of Individual Patient

Characteristics

Using Systematic Reviews in Clinical Practice

Summarizes large body of knowledge Answers specific clinical question Less likely to be biased than narrative reviews More accurate and precise estimate of treatment

effect

Research Question: What is the effect of Glutamine and Antioxidant

supplementation on survival in critically ill patients?

Methods: A meta-analysis

Using Systematic Reviews in Research Setting

Effect of Glutamine in Critically Ill:

A Systematic Review of the Literature Comprehensive search

Selection criteria Randomized

Surgical or critically ill adults

Glutamine (EN or PN) vs. placebo

Clinically important outcomes

20 RCT’s

Effect of Glutamine:

A Systematic Review of the Literature

Updated Jan 2009, see www.criticalcarenutrition.com

Mortality


A Systematic Review of the LiteratureInfectious Complications



A Systematic Review of the LiteratureHospital Length of Stay


Results of Subgroup Analysis

PN>>>EN?

Mortality Infection

EN(n=9)

0.81 (0.48-1.34)P=0.41

0.83 (0.64-1.08)P=0.16

PN(n=17)

0.71 (0.55-0.92)P=0.008

0.76(0.62-0.93)P=0.008

1200 ICU patientsEvidence of

organ failureR

glutamine

placebo

Concealed

Stratified by site

R

R

antioxidants

placebo

Factorial 2x2 design

placebo

antioxidants

REducing Deaths from OXidative Stress:

The REDOXS study

Fed enterally

Using Systematic Reviews in Research Setting

Summarizes what is known; identifies gaps Background of grant proposals Generates hypotheses Estimate of treatment effect N Subgroup analysis

Using Systematic Reviews in Policy Making

As an ICU, should you make an arginine-supplemented diet available for general use in your institution?

Meta-analyses of Meta-analyses of Arginine-supplemented DietsArginine-supplemented Diets

o 22 RCTs of IEDs All arginine-containing IED, not just IMPACT/IMMUNAID Non english, more recently published studies Excluded duplicates Excluded single agents

Heyland JAMA 2001;286:944

Overall Effect on Mortality

RR 1.10 (0.93-1.31)

Overall Effect on Complications

RR 0.66 (0.54-0.80)

1.18 (0.88,1.58)

Effect of Arginine-supplemented Diets

in the Critically Ill Patient


Mortality

Infectious Complications




Hospital Length of Stay




Using Systematic Reviews in Policy Making

Greatest generalizability Consistent with perspective of policy makers Related to other forms of integrative research

Assessing the Validity of Systematic Reviews

Validity= fxn { inputs, process, results }

Inputs selection of studies

clinical homogeneity explicit, reproducible criteria

methodological quality of studies outdated/unmeasured co-interventions


Process comprehensive search strategy

publication/timing bias data excess language bias

judgements about inclusion explicit/reproducible data abstraction reproducible


Results few studies few clinical endpoints statistical heterogeneity


Methdological Quality of Methdological Quality of Meta-analysesMeta-analyses

lots of bias little bias

weak inferences

strong inferences

Strong clinical recommendations

Making Inferences from a Meta-Analysis of RCT’s

Small number of trials Weak trial methodology Outdated/unmeasured

co-interventions Surrogate endpoints Statistical heterogeneity Fixed effects model

Large number of trials Strong trial methodology Current/documented co-

interventions Clinically important

endpoints Statistical homogeneity Random effects model

Weaker Inferences Stronger Inferences

Meta-analysis vs. Large RCT’s

“…if no subsequent randomized, clinical trial, the meta-analysis would have led to the adoption of an ineffective treatment in 32% cases and rejection of useful treatment in 33% cases.”

LeLorier NEJM 1997;337:536

“I still prefer conventional narrative reviews …Editorial, NEJM


RCT #1 RCT #2

RCT #3 RCT #4

RCT #5


Argument is with Meta-analysis, not the concept of systematic reviews

Assumes the latest single large trial is the GOLD standard

Assumes RCT and Meta-analysis are measuring the same thing

Differences in Generalizability Bias exists in both TOOLS.

Resolving Discrepancies Between a Meta-analysis and a Subsequent Large

RCT

Recent meta-analysis found calcuim supplementation to be effective in preventing preeclampsia

Large RCT found no risk reduction in health nulliparous women

Exploration of heterogeneity across studies Stratify for high and low baseline risk

JAMA 1999;282:664

Resolving Discrepancies Between a Meta-analysis and a Subsequent Large

RCT

JAMA 1999;282:664

JAMA 2008;300:933

Role of Systematic Reviews in Medical Education

Good source of medical knowledge Promotes EBM practices Helps locate original articles Facilitates critical appraisal of original research Considered a scholarly research activity

Conclusions

Important tool to determine the effectiveness of therapeutic interventions

Need to understand the strengths, weaknesses and limitations

Useful in clinical and policy decision making and research setting

Encourage use of and generation of systematic reviews amongst learners.

Documents

Systematic Reviews of the Literature and Meta-analyses: ….problems or panacea? Daren K. Heyland, MD, FRCPC, MSc Queen’s University, Kingston, Ontario