Systematic Reviews and Meta-Analysis

Kuntjoro Harimurti

Center for Clinical Epidemiology and Evidence-Based Medicine (CE-EBM)Faculty of Medicine UI/Cipto Mangunkusumo Hospital

Jakarta

Reviews on medical literatures

• Overview (review article): • unsystematic

• Systematic review: • in gathering, evaluating, presenting evidence• no formal statistical methods

• Meta-analysis: • systematic review plus formal statistical analysis

Review articleSystematic review

Meta analysis

Integrative Literature

Systematic vs. Unsystematic Reviews

Unsystematic reviews• Not focused on one

interested topic broader• Mechanisms of disease• No criteria for selection of

articles to be reviewed bias

• No assessment of selected articles

• Low applicability in clinical practice

Systematic reviews• Focused on only one or two

topics of interest• Diagnosis, etiology, prognosis,

and treatment of disease• Explicit methods and criteria

for selection (database, keywords, types of articles)

• Assessment of selected articles

• Highly applicable

Narrative/traditional reviews(unsystematic)

Narrative/traditional reviews(unsystematic)

Systematic reviews

Systematic review/Meta-analysis

Why Systematic Reviews?• Information overload

• Not all studies in journals are good in quality (valid/unbiased)

• Result from many studies not conclusive (controversies exist)

Systematically search

Systematically assessed the quality of included studies

Systematically combined to arrive at conclusion

Medical evidence increasing at epidemic rates: we all need EBP skills to keep up-to-date

MEDLINE 20102,000 articles / day

approx 75 new trials published

every day

Bastian, Glasziou, Chalmers (2010) 75 Trials and 11Systematic Reviews a Day: How Will We Ever Keep Up? PLoS Med 7(9)

The research-to-awareness gap

0

500000

1000000

1500000

2000000

2500000

Trials MEDLINE BioMedical

Med

ical

Art

icle

s pe

r Yea

r

5,000?per day

1,500 per day55 per

day

Streptokinase for Myocardial Infarction

Prophylactic antiarrhythmic drug therapy in acute myocardial infarction

Teo et al (JAMA, 1993)• A systematic review of 51 RCTs of anti-

arrhythmic drugs in heart attack involving 23,229 patients

• 660 deaths in patients allocated drugs (lidocaine)

• 571 deaths in patients allocated to control• 89 deaths attributable to drugs

Deadly medicine: why tens of thousands of heart patients died

in America's worst drug disaster • At the peak of their use in the late 1980s, it has been

estimated that these drugs killed asm many Americans every year as were killed during the whole of Vietnam war.

• The vast majority of the victims of these drugs were treated outside controlled trials

Thomas J Moore, 1995

Steps on Systematic Reviews

• Formulating answerable question (PICO)• Extensive and comprehensive searching the evidence

– 2 or more major databases + handsearching– Specific keywords– Selection criteria (inclusion and exclusion): types of

(primary) articles, year of publication, language• Critically appraise the evidence quality assessment

– 2 or more assessors• Synthesis and analysis meta-analysis• Discuss and conclude the results

Components of SR/MA

• Introduction

• Methods

• Results

• Discussions and Conclusions

Introduction

• State the reasons why the review is needed– Based on problems in clinical setting (high volume,

high risk, high cost)• Any controversies in the literatures? In

treatment, choice of diagnostic modalities, determination of prognosis

• Ended with objective of SR based on clinical question (PICO)

Methods

• How the individual studies searched and selected

• How to appraise/assess the individual studies

• How to combine (synthesis)

Searching the literatures

• A comprehensive and reproducible literature search is the foundation of a systematic review

• Search for published studies at least in MEDLINE, EMBASE, and CINAHL.

• Search for unpublished clinical trials in the Cochrane Central Register of Controlled Trials (CENTRAL) or other clinical trial database (www.clinicaltrial.gov)

• Secondary/hand searching• No limitation on years and languages

Literature Search Challenges

• Database Bias - “No single database is likely to contain all published studies on a given subject.”

• Publication Bias - selective publication of articles that show positive treatment of effects and statistical significance.– Hence, it is important to search for unpublished

studies through a manual search of conference proceedings, correspondence with experts, and a search of clinical trials registries.

Literature Review Challenges (cont.)

• English-language bias - occurs when reviewers exclude papers published in languages other than English

• Citation bias - occurs when studies with significant or positive results are referenced in other publications, compared with studies with inconclusive or negative findings

Quality Assessment

• The validity of a systematic review ultimately depends on the scientific method of the retrieved studies and the reporting of data

• In systematic review to assess treatment effect, RCTs are considered to be more rigorous than observational studies– A review based on well-designed RCT will likely be more

valid and accurate than a review based on observational studies or case reports

Quality Assessment (cont.)

• Quality assessment should be performed by at least two assessors

• The most common way to assess and report study quality has been using a composite, numerical scoring instrument (Exp. Jadad score)

Jadad score• Randomization (2 points possible)

– 1 point if study described as randomized– Add 1 point if randomization method described and appropriate

(e.g. random numbers generated) – Deduct 1 point if randomization described and inappropriate

• Double-blinding (2 points possible)– 1 point if study described as double-blinded– Add 1 point if method of double-blinding described and

appropriate – Deduct 1 point if double-blinding described and inappropriate

• Withdrawals (1 point possible)– Give 1 point for a description of withdrawals and drop-outs

Results

• Result of search strategies (flow diagram)

• Result of quality assessment

• Synthesis (summarized the effects)

Flow diagramPotentially relevant RCTs (n= ..)

n RCTs excluded with reason

RCTs retrieved for more detailed information (n= ..)

n RCTs excluded with reason

Potentially appropriate RCTs (n= ..)

RCTs in meta-analysis (n=..)

n RCTs excluded with reason

Quality Assessment

Summarized the effect

• Forest plot:– Effect size with CI of each study represented by

weightened square – Combined effect (and CI) represented by diamond – Vertical line (line of no effect) “1” for OR/RR, “0” for

mean difference– Vertical dotted line crossing combined effect to assess

heterogeneity• Test for heterogeneity

– Cochrane Chi2 (p value), I2 (degree of heterogeneity) – Random effect model vs. Fixed effect model

Study I 1992Study II 1994Study III 1995Study IV 1995Study V 1996Study VI 1997Study VII1 1999Study VIII 2000Combined

0.1 10

OR = 1Favor drug Favor placebo

Meta-analysis of RCTs with nominal outcome

Study I 1992Study II 1994Study III 1995Study IV 1995Study V 1996Study VI 1997Study VII1999Study VIII 2000

Combined

-0.3 +0.3Xe-xc=0

Favor drug Favor placebo

Meta-analysis of RCTs, numerical outcome

Result of summarized effects

Reporting publication bias

• A funnel plot is used as a way to assess publication bias in meta-analysis

CRITICAL APPRAISAL OFSYSTEMATIC REVIEWS/META-ANALYSIS

Is the review any good?Q-FAST appraisal

• Question – Does the systematic review addres focused question (PICO)?

• Finding – Did the search find all relevant evidence?

• Appraisal – Have been the studies critically appraised?

• Synthesis – Have the results been synthesised with appropriate summary tables and plots?

• Transferability of results

Thank You