20141110-Detty-IKK-EBM-Systematic Review and Meta-analysis.pdf

8/18/2019 20141110-Detty-IKK-EBM-Systematic Review and Meta-analysis.pdf

1/79

SYSTEMATIC REVIEWAND META-ANALYSIS

Dr. Detty S Nurdiati, MPH, PhD, SpOG(K)1,2

1Clinical Epidemiology and Biostatistics Unit2Dept of Obstetrics & Gynecology

Faculty of Medicine, Universitas Gadjah Mada Yogyakarta, Indonesia

The 6th Board Meeting and Scientific Meeting of ICE-EBM NetworkFaculty of Medicine, Universitas Maranatha, Bandung, December 13-15, 2013

EVIDENCE-BASED MEDICINE WORKSHOP


2/79


3/79

Levels of Evidence,

in Context

Anderson, P.F. (2006).

Chain of Trust / Level of Evidence – Vertical.

Available at: http://www-personal.umich.edu/~pfa/

pro/courses/ChainOfTrustLoEVert2.pdf.

Top pyramid is from:

Medical Research Library of Brooklyn.

Guide to Research Methods, The Evidence Pyramid. Available

at: http://library.downstate.edu/ebm/2100.htm

Anderson PF. 2011. Systematic Reviews: Context & Methodology


4/79

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

You find this review

Mickan S. 2013. Systematic Reviews


5/79




6/79


How confident are you of the evidence?



7/79


Why systematic reviews?

• efficient way to access the body of research

• saves time required for searching

•critical appraisal

• interpretation of results

• explore differences between studies

•

reliable basis for decision making• unbiased selection of relevant information

• useful for health care, policy, future research


8/79

SYSTEMATIC REVIEW?

A review of a clearly formulated

question that uses systematic and

explicit methods to identify,

select and critically appraiserelevant research, and to collect

and analyse data from the studies

that are included in the review.


9/79


Narrative vs Systematic Review

Narrative

– Many questions

– No search methods

–No inclusion criteria

– No combining studies

– Prone to random and

systematic error

– Provide conflictingsummaries

Systematic

– One question

– Explicit search, reproducible

–Explicit inclusion criteria

– Combine study results

(with/without meta-analysis)

WHY do we need Systematic Reviews?



10/79


Purpose of systematic reviews• Provide up to date summary of all published

research literature

• Allow large amounts of data to be assimilated

• Provide an objective collation of results of

research

• Provide reliable recommendations



11/79


Useful Resources

• The Cochrane Collaboration www.thecochranelibrary.com/ – Cochrane Handbook for Systematic Reviews of

Interventions (version 5 updated March 2011)

• CRD www.crd.york.ac.uk/ – The Centre for Reviews and Dissemination is a

department of the University of York and is part ofthe National Institute for Health Research

• EPPI-Centre www.eppi.ioe.ac.uk/ – The Evidence for Policy and Practice Information and

Co-ordinating Centre, Social Science Research Unit,Institute of Education, University of London.


http://www.thecochranelibrary.com/http://www.crd.york.ac.uk/http://www.eppi.ioe.ac.uk/http://www.eppi.ioe.ac.uk/http://www.crd.york.ac.uk/http://www.thecochranelibrary.com/


12/79


Key features of a systematic review• clearly stated objectives

• pre-defined eligibility criteria

• explicit, reproducible methodology

• systematic search

• assessment of validity of included studies

• systematic synthesis and presentation offindings


13/79


The review team• review must be undertaken by more than one person

• allows double-checking

–

eligibility of included studies – data collection and entry

– risk of bias assessment

• different areas of expertise

–

clinical (multidisciplinary) – systematic review methods (including statistics)

– user perspective (consumer, professional, settings)

• consider establishing an advisory group


14/79


Steps of a systematic review1. Clear answerable question

2. Reproducible search strategy

3. Assessment of literature quality

4. Summary of the evidence

5. Statistical, sensitivity analyses

6. Interpretation

7. Conclusions, recommendations

8. Published protocol and review



15/79


Types of systematic review

• Different research questions require differentstudy designsgenerate different types ofreview

• Variations occur in – Research questions asked

– Primary study designs included

–Methods for synthesis

– Approaches to being systematic

– Types of evidence included



16/79


Best evidence for different questions

Treatment Prognosis Particularperspective

Systematic

Review of …

Systematic

Review of … Systematic

Review of …

Randomisedtrials InceptionCohorts Qualitativestudies



17/79


Best evidence for different questions

Treatment Prognosis Particularperspective

Systematic

Review of …

Systematic

Review of … Systematic

Review of …

Randomisedtrials InceptionCohorts Qualitativestudies

The most type of

SR published



18/79


Risk of bias

• use risk of biasassessment

• for studies that provideddata for sleep disruption:

– what is the overall risk ofbias of these studies

–does the risk of biasreduce our confidence inthe effect?

F lt f M di i U i it G dj h M d Y k t I d i


19/79


Review level

↓

Effect measure

Study A Effect measureOutcome data

Effect measureOutcome dataStudy B

Effect measureOutcome dataStudy C

Effect measureOutcome dataStudy D

Study level

↓

Source: Jo McKenzie & Miranda Cumpston



20/79


What is a meta-analysis?

• combines the results from two or more studies

• estimates an ‘average’ or ‘common’ effect

•

optional part of a systematic review

Source: Julian Higgins

Systematicreviews Meta-analyses



21/79


Why perform a meta-analysis?

• quantify treatment effects and theiruncertainty

• increase power

• increase precision

• explore differences between studies

• settle controversies from conflicting studies

• generate new hypotheses




22/79


When not to do a meta-analysis

• mixing apples with oranges

• each included study must address same question

• consider comparison and outcomes

• requires your subjective judgement

• combining a broad mix of studies answers broad

questions

• answer may be meaningless and genuine effectsmay be obscured if studies are too diverse


Faculty of Medicine Universitas Gadjah Mada Yogyakarta Indonesia


23/79


When not to do a meta-analysis

• garbage in – garbage out

• a meta-analysis is only as good as the studies in it

• if included studies are biased:

• meta-analysis result will also be incorrect

• will give more credibility and narrower confidenceinterval

• if serious reporting biases present:

• unrepresentative set of studies may give misleadingresult




24/79


When can you do a meta-analysis?

• more than one study has measured an effect

• the studies are sufficiently similar to produce

a meaningful and useful result• the outcome has been measured in similar

ways

•data are available in a format we can use



25/79


Steps in a meta-analysis

• identify comparisons to be made

• identify outcomes to be reported and

statistics to be used• collect data from each relevant study

• combine the results to obtain the summary

of effect• explore differences between the studies

• interpret the results



26/79


Selecting comparisons

• break your topic down into pair-wise comparisons

• each review may have one or many• use your judgement to decide what to group

together, and what should be a separatecomparison

Hypothetical review: Caffeine for daytime drowsiness

vscaffeinated coffee decaffeinated coffee



27/79


Selecting outcomes & effect measures

• for each comparison, select outcomes

• for each outcome, select an effect measure• may depend on the available data from included

studies

• asleep at end of trial (RR)

• irritability (MD/SMD)

• headaches (RR)

Hypothetical review: Caffeine for daytime drowsiness

vscaffeinated coffee decaffeinated coffee



28/79


Calculating the summary result

• collect a summary statistic from eachcontributing study

• how do we bring them together?

• treat as one big study – add intervention &control data?

• breaks randomisation, will give the wrong answer

•

simple average?• weights all studies equally – some studies closer to

the truth

• weighted average



29/79


Weighting studies

• more weight to the studies which give moreinformation

• more participants, more events, narrower

confidence interval

• calculated using the effect estimate and its variance

• inverse-variance method:

2

SE

1

estimateof variance

1weight

weightsof sum

)weightestimate(of sumestimatepooled



30/79


Headache Caffeine Decaf Weight

Amore-Coffea 2000 2/31 10/34

Deliciozza 2004 10/40 9/40

Mama-Kaffa 1999 12/53 9/61

Morrocona 1998 3/15 1/17

Norscafe 1998 19/68 9/64

Oohlahlazza 1998 4/35 2/37

Piazza-Allerta 2003 8/35 6/37

For example



31/79


Headache Caffeine Decaf Weight

Amore-Coffea 2000 2/31 10/34 6.6%

Deliciozza 2004 10/40 9/40 21.9%

Mama-Kaffa 1999 12/53 9/61 22.2%

Morrocona 1998 3/15 1/17 2.9%

Norscafe 1998 19/68 9/64 26.4%

Oohlahlazza 1998 4/35 2/37 5.1%

Piazza-Allerta 2003 8/35 6/37 14.9%

For example



32/79


• for dichotomous or continuous data

• inverse-variance

• straightforward, general method

• for dichotomous data only

• Mantel-Haenszel (default)

• good with few events – common in Cochrane reviews

•

weighting system depends on effect measure• Peto

• for odds ratios only

• good with few events and small effect sizes (OR close to

1)

Meta-analysis options



33/79


A forest of lines

Trees Joyce Kilmer Forest by charlescleonard http://www.flickr.com/photos/charlescleonard/3754931947/


http://i.creativecommons.org/l/by-nc/3.0/88x31.png


34/79

y , j , gy ,

Forest plots

Headache at 24 hours

• headings explain the comparison



35/79

y , j , gy ,

Forest plots


• list of included studies



36/79

y , j , gy ,


• raw data for each study

Forest plots



37/79

y j gy

Forest plots


• total data for all studies



38/79


• weight given to each study

Forest plots



39/79


• effect estimate for each study, with CI

Forest plots


40/79



41/79


• scale and direction of benefit

Forest plots



42/79


• pooled effect estimate for all studies, with CI

Forest plots



43/79

Interpreting confidence intervals

• always present estimate with a confidence interval

• precision

• point estimate is the best guess of the effect

• CI expresses uncertainty – range of values we can be

reasonably sure includes the true effect

• significance

• if the CI includes the null value

• rarely means evidence of no effect

• effect cannot be confirmed or refuted by the available evidence

• consider what level of change is clinically important



44/79

Heterogeneity• consider heterogeneity in the meta- analysis

– overlap in confidence intervals

– I2 statistic

• is there unexplained statistical heterogeneity?



45/79

Identifying heterogeneity

• visual inspection of the forest plots

• chi-squared (c2) test (Q test)

•I2 statistic to quantify heterogeneity



46/79

Visual inspection

Forest plot A Forest plot B



47/79

The chi-squared (c2) test

• tests the null hypothesis of homogeneity

• low power with few studies

•

may detect clinically unimportant differences withmany studies

• narrow question (yes/no) not useful if

heterogeneity is inevitable

• calculated automatically by RevMan



48/79

The I2 statistic

• I2 statistic describes the percentage of

variability due to heterogeneity rather than

chance (0% to 100%)

• low values indicate no, or little, heterogeneity

• high values indicate a lot of heterogeneity

• calculated automatically by RevMan

• be cautious in interpreting



49/79

What to do about heterogeneity

• check that the data are correct

• if heterogeneity is very high – interpret fixed-effect results with caution

•

consider sensitivity analysis – would random-effects havemade an important difference?

– may choose not to meta-analyse• average result may be meaningless in practice

• consider clinical & methodological comparability of studies

– avoid• changing your effect measure or analysis model

• excluding outlying studies

• explore heterogeneity



50/79

Reporting biases•

consider whether the outcome is at risk – no large studies

– industry sponsorship

–

few of your studies report this outcome – consider funnel plot results or statistical tests for

small study effects, if appropriate



51/79

External validity

• how well, and how

completely, do the studies

address your review

question?

• compare your inclusion

criteria to the ‘Characteristics

of included studies’ table – PICO



52/79

The Results section of your review

• a systematic, narrative summary of results

• forest plots

– key forest plots linked as figures

•

usually primary outcomes – all forest plots will be published as supplementary data

– avoid forest plots with only one study

• may also add other data tables

–results of single studies

• summary data for each group, effect estimates, confidence intervals

– non-standard data

• not helpful to report trivial outcomes or results at high risk of bias


53/79

RAPID CRITICAL APPRAISAL OF SYSTEMATIC REVIEW



54/79

Time Spent

P

r o j e c t s W o r k e d O

n

Step 1What

question

did thestudy ask?

Step 4

How do theresults

apply to the

care of mypatients

Rapid Critical Appraisal of SR

Step 3

What do theresultsmean? Step 2

How well

was the

study

done?


55/79

Step 1

What question did the study ask?

P :...................

I :...................

C :...................

O :...................


56/79

Step 2

How well was the study done?

... Internal Validity...


57/79

QUESTION

Does the SR address a focusedquestion?

... And use it to direct the searchand select articles for inclusion?



58/79

Getting started

KEY = systematic, rigorous, transparent, reproducible• Define the research question

– Clear background, scope, setting

– Research question determines method of review (PICO)

– Specify inclusion and exclusion criteria



59/79

FIND

Did the search find all therelevant evidence?



60/79

Find the published researchClear, comprehensive, reproducible search strategy

• Search terms

• Databases

• Other strategies for grey literature




61/79

Manage the research evidence

• Organise database, hand searching

– Use of forward citation searching, reference lists

•

Manage references – Reference Management software eg Endnote

• Screen studies to check fit

–2 reviewers, process of agreement

– Record decisions about whether studies meet

criteria




62/79



63/79

APPRAISEHave the studies been critically

appraised?

... And was the overall qualityadequate?



64/79

Assess quality of the literature•

Dual, independent assessment of design aspectslikely to cause bias – depends on study designs

• Resource: online www.cochrane.org/training/cochrane-handbook or www.equator-network.org/home/



http://www.cochrane.org/training/cochrane-handbookhttp://www.cochrane.org/training/cochrane-handbookhttp://www.equator-network.org/home/http://www.equator-network.org/home/http://www.equator-network.org/home/http://www.equator-network.org/home/http://www.cochrane.org/training/cochrane-handbookhttp://www.cochrane.org/training/cochrane-handbookhttp://www.cochrane.org/training/cochrane-handbook


65/79

The Cochrane risk of bias tool

Risk of bias Interpretation Within a study Across studies

Low risk of bias Plausible bias unlikely toseriously alter the

results.

Low risk of bias for all

key domains.

Most information is from

studies at low

risk of bias.

Unclear risk of bias Plausible bias that raisessome doubt about theresults

Unclear risk of bias forone or more key

domains.

Most information is fromstudies at low or unclear

risk of bias.

High risk of bias Plausible bias thatseriously weakens

confidence in the results.

High risk of bias for one

or more key

Domains.

The proportion of

information from studies

at high risk of bias issufficient to affect the

interpretation of the

results.




66/79

A visual representation - RCTs




67/79

Describe included studies• Design data extraction forms

– General descriptive information

– Research methods

– Key results

–2 reviewers, process of agreement



68/79

SYNTHESISE

Have the results been synthesisedwith appropriate tables and plots?

... And were the results similarbetween the studies?



69/79

Decide on process of synthesisFactors to consider

• Consistency of outcome measures

• Sub groups

• Heterogeneity

• Common sense test




70/79

Details of data synthesis

• Look for consistent measurement of data,with 95% confidence intervals




71/79

Primary outcome/s

•

Basis for meta-analysis




72/79

Sub group analysis•

Identify in protocol with justification• To enhance usefulness of research answers




73/79

Heterogeneity• Common sense test of study design, outcome

measurements, forest plot

• Are syntheses meaningful (apples vs oranges)

• Influences statistics within meta-analysis




74/79

Sensitivity analyses• determine whether the assumptions or decisions

made have a major effect on the results of thereview.



75/79

Step 3

What do the results mean?

What measure was used, howlarge was the effect (could ithave been due to chance)?

Other comments?



76/79


77/79

Step 4

How do the results apply to the care

of my patients? Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia


78/79

Is the review any good – FAITH?

FINDING

Did they find most studies?

APPRAISAL

Did they use appropriate inclusion criteria?

INCLUDE

Did they include valid studies – for question asked?

TOTAL UP

Did they synthesise similar outcomes?

HETEROGENEITY


79/79

The Cochrane Collaboration

International non-profit organisation that

prepares, maintains, and disseminates

systematic up-to-date reviews of

health care interventions

Documents

20141110-Detty-IKK-EBM-Systematic Review and Meta-analysis.pdf