How to conduct and assess a systematic review...2018/05/05 · a systematic review to integrate the results of the included studies. Statistical methods (meta-analysis) may or may

Trusted evidence. Informed decisions. Better health.



How to conduct and assess a systematic review

Antti Malmivaara, MD, PhD, Chief Physician

Centre for Health and Social Economics

National Institute for Health and Welfare

Finnish Institute for Health and Welfare, Centre for Health and

Social Economics, Chief Physician

Current Care Guidelines, Finnish Medical Association Duodecim,

Editor

Ministry for Social Affairs and Health, Council for Choices in

Health care in Finland, Expert

Hospital ORTON, Scientific Board, Member

European Academy of Rehabilitation Medicine

Cochrane Back and Neck Review Group, Editorial Board

Cochrane Rehabilitation Field, Member of Executive Committee;

Chair of the Methodology Committee

Potential conflicts of interests

8.5.2018 Antti Malmivaara

“It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or

sub-specialty, adapted periodically, of all relevant randomized controlled trials”

Archie Cochrane

Antti Malmivaara



International non-profit organization that prepares, maintains,

and disseminates systematic up-to-date reviews of health care

interventions

A systematic review is a review of a clearly formulated

question that uses systematic and explicit methods to

identify, select, and critically appraise relevant research,

and to collect and analyze data from the studies that are

included in the review.

Meta-analysis refers to the use of statistical techniques in

a systematic review to integrate the results of the included

studies.

Statistical methods (meta-analysis) may or may not be used to

analyze and summarize the results of the included studies.

Systematic review, definitions

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, et al. (2009) The PRISMA Statement for Reporting Systematic

Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration. BMJ

2009;339:b2700


1. Defining the question

2. Establishing inclusion/exclusion criteria

3. Literature search and choice of articles

4. Data extraction and quality estimation of

articles

5. Combining scientific evidence (descriptive or

meta-analysis)

6. Drawing conclusions based on evidence

Steps in a systematic review

Antti Malmivaara


Antti Malmivaara

Assessing a systematic review


1. What is the study question (PICOS)?

2. Validity - how high is the Risk of Bias

(RoB)?

3. What are the results and how precise they

are?

4. What is the generalizability of the results?

Steps in assessing systematic

reviews and meta-analyses

Antti Malmivaara


Patient (population)

Intervention

Control intervention

Outcome

Study design

1. What is the study question

(PICOS)

Antti Malmivaara


Was the study question well defined and were the studies

included in the meta-analysis clinically homogenous?

Was the study design appropriate for anwering the study

question?

Is it probable that all relevant studies are included in the

meta-analysis?

What was the validity of the included studies and were the

interpretations based on studies with low risk of bias?

Are the results of the systematic review/meta-analysis

repeatable?

2. Validity - how high is the Risk

of Bias (RoB)

Antti Malmivaara


1. What are the point estimates?

2. How precise are the point

estimates (by 95 per cent confidence

intervals) ?

Statistical significance and clinical

importance

3. What are the results and how

precise they are?

Antti Malmivaara


21

5

10 1

1 2

8

7

8

12

4

3

1

1

2

8

7

2

1

1

1

2

8

1

5

15

6

No

t M

en

tio

ne

d

Ro

uti

ne

Ex

pe

rim

en

tal

Rare

/Ne

ve

r

Sp

ec

ific

M

M

M

M

M

M

Textbook/Review

Recommendations

Odds Ratio (Log Scale)

0.5 1.0 2.0

Favours Treatment Favours Control

RCTs Pts

1 23

2 65

3 149

4 316

7 1793

10 2544 11 2651 15 3311 17 3929 22 5452

P<.01

23 5767

27 6125

30 6346

33 6571

43 21 059 54 22 051

67 47 531 65 47 185

70 48 154

P<.001

P<.00001

Cumulative

Year

1960

1965

1970

1980

1985

1990

Trombolysis for MI

Antti Malmivaara 8.5.2018 Antti Malmivaara

Example Figure: Overall failure (defined as failure of assigned regimen or

relapse) with tetracycline-rifampicin versus tetracycline-streptomycin.

1. How applicable the results are for my

own patients? • Consider the PICO: how similar are the patients, interventions, control

interventions and outcomes in relation to my own patients.

• Putting the question other way round: are my patients so different from those

in the systematic review, that the results are not applicabe to my patients

2. Have we enough resources and

competence for the intervention?

4. What is the generalizability of

the results?

Antti Malmivaara


3. Were all important benefits and harms taken into

consideration

• Although systematic reviews/meta-analyses provide more decisive

information of results than original studies, they may not have

covered all benefits or harms important for my patient.

4. What are my patient’s values and preferences

concerning the benefits and harms

Antti Malmivaara

4. What is the generalizability of the results? (cont’d)


Antti Malmivaara

8.5.2018

Interpretation of systematic reviews

and meta-analyses should be based

on:

Clinically homogeneous studies

Studies with a low risk of bias

Lack of publication bias

Antti Malmivaara

Malmivaara A. Methdological considerations of the GRADE method. Ann Med

2015;47:1-5.

Conclusions

8.5.2018

Antti Malmivaara

Funnel plot – no publication bias

8.5.2018

Turlik M. Evaluating the results of a

Systematic Review/Meta-Analysis:

Foot and Ankle Online Journal

2009;2:5

Antti Malmivaara

8.5.2018

Funnel plot – a publication bias

Turlik M. Evaluating the results of a

Systematic Review/Meta-Analysis:

Foot and Ankle Online Journal

2009;2:5

Direct comparison, undirect

comparison, network meta-analysis


Direct comparison Undirect

comparison

Direct and undirect

comparison = Network

meta-analysis

Antti Malmivaara

Antti Malmivaara

Antti Malmivaara

Evidence from direct and undirect comparison can be

combined in network analysis

Statistical power and precision of results increases as the

patient material increases

The risk of bias in network meta-analyses is greater than in

systematic reviews based on direct comparisons, because

some of the comparisons are not based on a randomized

design

Summary, network meta-analysis


Antti Malmivaara

Conducting a systematic review

Follow the PRISMA statement


Antti Malmivaara


Antti Malmivaara

8.5.2018

PRISMA flow chart


Antti Malmivaara

8.5.2018

PRISMA checklist (1)

Antti Malmivaara

Antti Malmivaara

8.5.2018


Antti Malmivaara


Antti Malmivaara

8.5.2018


Antti Malmivaara

Antti Malmivaara

8.5.2018

QUOROM vs PRISMA

Antti Malmivaara




GET INVOLVED! FOLLOW US

http://rehabilitation.cochrane.org

@CochraneRehab

CONTACT US

[email protected]

http://rehabilitation.cochrane.org/

http://rehabilitation.cochrane.org/

Thank you!

8.5.2018

Antti Malmivaara

Documents

How to conduct and assess a systematic review...2018/05/05 · a systematic review to integrate the results of the included studies. Statistical methods (meta-analysis) may or may