Systematic Reviews and Meta-Analysis

Systematic Reviews and Meta-Analysis

Methodologies for a new era summer school

School of Applied Social Studies, University College Cork

20 June 2011

Dr Paul Montgomery

Aims

1) Discuss the advantages and main features of systematic reviews

2) Introduce basic principles of meta-analysis

Course feedback

The Problem

Millions of articles published in thousands of journals each year

Practitioners and researchers are busy

Subjective summaries may misrepresent research

Reviews

Systematic Reviews Aim to answer specific questions, reduce

uncertainty, identify outstanding questions

Common methods include narrative synthesis, meta-analysis (meta-regression)

Traditional ‘journalistic’ reviews Aim to persuade, draw attention to a

topic, synthesise information, etc. Narrative synthesis most common

Systematic Review:

“the application of scientific strategies that limit bias to the systematic assembly, critical appraisal, and synthesis of all relevant studies on a specific topic."

Cook DJ, Sakett DL, Spitzer WO. Methodological guidelines for systematic reviews of randomized contro trials in health care from the Potsdam Consultation

on Meta-Analysis. J. Clin. Epidemiol. 1995;48:167-71

Systematic Reviews

Clear Question Define the population, problem,

intervention, alternative interventions, and outcomes

Replicable Method Search strategy Inclusion criteria Analytical strategy

Transparent Process

Advantages

Explicit methods limit bias in identifying and rejecting studies

Information can be understood quicklyReduced delay between discoveries

and implementation Results can be formally comparedHeterogeneity can be identified and

new hypotheses generatedQuantitative reviews increase precision

Producers

CochraneCampbellEPPIDARENICE Interested practitioners/ academics

Cochrane Review ProcessRegister titles and check for overlapProtocols developed and peer reviewedSearches performed widely on all main

databases, grey literature searches, personal contacts

Abstracts reviewed by two authorsData collected and trial quality assessedData synthesis and analysisWrite-up Reviewed by Cochrane/ Campbell editors,

then peer reviewed

Systematic reviews

Key components:1. Ask a good question2. Identify studies3. Extract data4. Synthesise data5. Interpreting the results

Who Should Review?

“Experts, who have been steeped in a subject for years and know what the answer ‘ought’ to be, are less able to produce an objective review of the literature in their subject than non-experts. This would be of little consequence if experts' opinions could be relied on to be congruent with the results of independent systematic reviews, but they cannot.”

(Trisha Greenhalgh)

PICO Mad-libs

ForP ____________ doesI ____________ compared toC ____________ improve/reduceO ____________ ?

Highly Sensitive Search

Electronic SearchesDatabases/IndexesAdditional Electronic Searches

Hand SearchesPersonal Contacts

Electronic databases:

BA

Medline

PsycInfo

Medline covers 23% of the core 505 ‘psychiatric’

journals, plus most of the major biomedical

journals. Biological Abstracts covers

48%, plus lots of life sciences stuff.

Embase covers 67% plus lots of

European journals that Medline

misses.

Embase

PsycInfo covers 73% and has a

psychological focus

Searching PsycLit and Embase will cover 92%

of the core 505 ‘psychiatric’ journals.

Electronic Searches

Sensitivity vs. SpecificityEven if 2 terms and 3 databases return almost all literature on a subject, the goal of a systematic review is to find everything.

Electronic Searches

Specific AuthorsReverse CitationAgencies / Non-ProfitsFunding BodiesAcademic Groups / Research CentersGoogle

Additional Searches

Previous ReviewsBibliographies of Related ArticlesHand Search Journals (that aren’t

indexed)Conference Reports

(many are electronically published)

Personal Communication

Call or Email AuthorsAttend ConferencesWrite to:

Agencies / Non-ProfitsProviders / Manufacturers /

DistributorsFunding BodiesAcademic Groups / Research Centers

Questions to Ask

Which programs will be studied?Compared to what?What study designs are acceptable?What must a study measure?How must it be measured?Must researchers be blind at

allocation, during the trial, etc?How will dropouts be handled?What about missing data?

Inclusion and Exclusion

Types of studiesTypes of studies Types of participantsTypes of participants Types of comparisonsTypes of comparisons

Specify:Specify:Types of outcomesTypes of outcomes

Multiplicity (time, comparisons, measures, Multiplicity (time, comparisons, measures, statistics)statistics)

Transparency

Be clear about all definitions, searches, inclusion and exclusion criteria, etc.

Report ongoing trialsList excluded studies, particularly if:

The trials contain valuable information Exclusion was a close call You discovered something about a trial

Evaluating a Review

Even if a review is Even if a review is ‘systematic’ it may ‘systematic’ it may

not be well-not be well-conducted. How do conducted. How do

we tell the we tell the difference?difference?

Validity

1. Did the review address a clearly focussed question?

2. Were the right sort of studies selected?

3. Was the search strategy explicit and comprehensive?

4. Did the reviewers assess the quality of the identified studies?

Importance:

1. Were the results similar from study to study?

2. What is the overall result of the review?

3. How precise are the results?

Potential Sources of Bias

Describe aspects of study design that might have influenced the magnitude or direction of results

Use of rating scales with fixed cut-offs potentially misleading

Consider external validity

Juni P, Witschi A, Bloch R, Egger M. The hazards of

scoring the quality of clinical trials for meta-

analysis. JAMA 1999; 282: 1054-1060

Tower of Babel

Studies that find a treatment effect are more likely to be published in English-language journals.

Opposing studies may be published in non-English-language journals.

Gregoire G, Derderan F, Le Lorier J. Selecting the language of the publications included in a meta-analysis: is there a Tower of Babel Bias? J.Clin.Epidemiol.

1995;48:159-163

Publication Bias

“the tendency of investigators, reviewers and editors to differentially submit or accept manuscripts for publication on the direction or strength of the study findings.”

Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A et al. Should unpublished data be included in meta-analyses? Current convictions and controversies.

JAMA 1993; 269: 2749-2753

Unpublished data

ControversialUnpublished data may not be a full or

representative sample (Cook 1993)Publication is no guarantee of scientific quality

(Oxman 1991)Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A et al. Should

unpublished data be included in meta-analyses? Current convictions and controversies. JAMA 1993; 269: 2749-2753

Oxman AD, Guyatt GH, Singer J, Goldsmith CH, Hutchison BG, Milner RA et al. Agreement among reviewers of review articles. J.Clin.Epidemiol. 1991;44:91-98.

Meta-analyses:

“A systematic review that employs statistical methods to combine and summarise the results of several studies.”

Cook DJ, Sakett DL, Spitzer WO. Methodological guidelines for systematic reviews of randomized contro trials in health care from the Potsdam Consultation

on Meta-Analysis. J. Clin. Epidemiol. 1995;48:167-71

Summarising trials

Reviews

Systematic reviews

Meta-analyses

Meta-analyses

Mathematically combine the results of different studies

For dichotomous or continuous outcomes

From analytical (treatment) or observational (aetiology, diagnosis, prognosis) studies

‘Weighted’ by study size (usually 1/se2) and/or quality

Benefits of meta-analysis:

1. To increase statistical power for primary end points and for subgroups.

2. To improve estimates of effect size.3. To resolve uncertainty when reports disagree4. To answer questions not posed at the start of

individual trials.

Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC. Meta-analyses of randomized controlled trials. N.Engl.J.Med. 1987;316:450-455

Outcome Measures

Continuous / Dichotomous (/ Ordinal)

Objective / Subjective

Meta-analysis

Some outcomes are measured on scales – e.g. depression or continuously e.g. sleep minutes

Continuous outcomes can be calculated using the scale on which they were measured (WMD)

If changes in depression are measured on different scales it is still possible to combined them but on a standardised scale

Meta-analysis

Alternatively we might be interested in binary data - two mutually exclusive states

Dead/alive; hospitalised/not hospitalisedThese data will be measured in a

different way to continuous (scale) dataReported as ‘event rates’

Meta-analysis

Central Tendency: Mean (Cohen’s d, Hedges’s g) Odds Ratio / Relative Risk / Rate Ratio

Variance (Confidence Interval)Clinical Significance (NNT/NNH)Heterogeneity (I2, Q, Chi2)

Dichotomous Outcomes

Odds are calculated by dividing the number of events by non-events (ie clients experiencing the event divided by clients not experiencing an event)

Risk/Rate is more widely reported in reviews as it tends to be easier to communicate

Weighting

Some studies contribute more weight to the ‘average’ result than do others

The more precise the effect estimate, the more weight is given

Wide variation is sometimes associated with small studies

Weighting

Clinical trials are rarely conducted according to identical protocols

Severity of the problem, intensity of the intervention, duration, setting of trial, age may account for differences in response

Apples and oranges?Sources of Heterogeneity:

Study participants Comparisons Intervention design Delivery Duration of follow-up Outcome measures Methods

Heterogeneity

Estimates from individual trials vary more than can be explained by the play of chance alone

N.B. Meta-analysis should NOT overlook important material differences in subgroup response

Heterogeneity – approaches

Qualitative v. quantitative Qualitative – reconsider pooling Does it makes sense to average

effects from the studies? Fixed v. random effects

Subgroup Analysis

If together there is excessive variation, when analysed separately there is a uniform response to treatment in each subgroup

Hypothesis generating

Sensitivity analysis:

Sensitivity analyses investigate how the conclusions of a review change when one or more of the decisions or assumptions are altered.

Testing for heterogeneity

Look at plots of resultsFormal tests of homogeneity

I2

Q Chi2

Assess qualitative differences in study design or implementation

.

Weeks Study name Comparison Outcome Statistics for each study Sample size Hedges's g and 95% CI

Hedges's Lower Upper Relative g limit limit Media Comp weight

4 Hassan 1992 Wait List Combined 4.30 2.55 6.05 10 8 0.68

5 Bickel 2007 Wait List Combined 0.81 -0.33 1.96 8 5 1.40

4 Milne 1998 Wait List (TaU) Combined -0.33 -1.35 0.70 7 6 1.67

8 Rosen 1976 Wait List Combined -0.73 -1.72 0.27 16 6 1.74

1 Klein 2001 Wait List Combined 0.66 -0.18 1.51 10 12 2.20

8 Abramowitz 2009 Wait List Combined 0.39 -0.45 1.22 11 10 2.23

8 Lidren 1994 Wait List Combined 0.91 0.09 1.74 12 12 2.29

8 Richards 2006 Wait List Combined 0.63 -0.14 1.41 23 9 2.46

12 Fletcher 2005 Wait List HADS - Anxiety 0.10 -0.65 0.86 11 15 2.56

1 Heading 2001 Wait List Combined 0.17 -0.58 0.92 13 13 2.57

13 Kiely 2002 Wait List (TaU) Combined 0.85 0.12 1.58 16 14 2.66

8 Lewis 1978 Monitoring Combined 0.58 -0.13 1.29 38 10 2.77

4 Jones 2002 Wait List (TaU) Combined 0.58 -0.06 1.21 19 20 3.15

8 Grime 2004 Wait List HADS - Anxiety 0.41 -0.22 1.04 16 23 3.16

10 Carlbring 2001 Wait List Combined 0.81 0.18 1.44 21 20 3.18

8 Sorby 1991 No Int (Plus TaU) Combined 0.62 -0.00 1.24 25 17 3.22

6 Smith 1997 Attention Combined 0.33 -0.29 0.94 30 15 3.25

10 Titov 2009 Wait List Combined 0.98 0.37 1.59 24 21 3.27

11 Arpin-Cribbie 2007 No Int Combined 0.81 0.24 1.38 29 22 3.50

10 Berger 2009 Wait List Combined 0.75 0.18 1.31 31 21 3.55



14 Hazen 1996 Wait List Combined 0.43 -0.10 0.97 27 27 3.75

13 Zetterqvist 2003 Wait List Combined 0.39 -0.07 0.84 37 45 4.28

12 Van Boeijen 2005 Treatment as Usual Combined -0.13 -0.59 0.32 53 28 4.29

10 Titov 2008b Wait List Combined 0.79 0.34 1.24 41 40 4.32

10 Titov 2008c Wait List Combined 0.45 0.02 0.87 61 34 4.54

10 Titov 2008a Wait List Combined 0.83 0.42 1.23 50 49 4.64

13 Mead 2005 Wait List HADS 0.18 -0.20 0.56 50 53 4.83

12 Rapee 2007 Wait List Combined 0.38 0.00 0.76 56 52 4.87

9 Proudfoot 2004 Treatment as Usual BAI 0.38 0.10 0.66 99 98 5.65

0.55 0.40 0.70 903 764

-2.00 -1.00 0.00 1.00 2.00

Favours No-Treatment Favours Self-Help

Anxiety (self-rated) at Post-Treatment compared to No-TreatmentAnxiety (Self-Rated Symptoms) at Post-Treatment

Group byTime

Study name Time (m) Risk ratio and 95% CI Statistics for each study

Relative Risk Lower Upper weight ratio limit limit

12-23 Dalby (2000) 14 1.20 0.35 0.01 8.41 Institutionalisation

12-23 Hogan (2001) 12 2.14 2.05 0.19 22.17 Institutionalisation

12-23 Newbury (2001) 12 3.21 0.94 0.13 6.55 Institutionalisation

12-23 Hall (1992) 12 7.41 0.38 0.11 1.37 Institutionalisation

12-23 Hebert (2001) 12 8.06 1.04 0.30 3.53 Institutionalisation

12-23 Kono (2004) 18 10.83 0.64 0.22 1.83 Institutionalisation

12-23 Yamada (2003) 18 17.26 1.30 0.56 3.01 Institutionalisation

12-23 Bernabei (1998) 12 21.52 0.67 0.32 1.43 Institutionalisation

12-23 Gill (2002) 12 28.37 0.72 0.38 1.39 Institutionalisation

12-23 0.78 0.55 1.10

24-35 Hall (1992) 24 42.30 0.16 0.04 0.67 Institutionalisation

24-35 Sorenson (1988) 30 57.70 1.02 0.81 1.28 Institutionalisation

24-35 0.46 0.08 2.80

36+ Hall (1992) 36 9.53 0.16 0.04 0.67 Institutionalisation

36+ Van Rossum (1993) 36 12.00 1.38 0.44 4.30 Institutionalisation

36+ Pathy (1992a) 36 12.86 1.29 0.46 3.66 Institutionalisation

36+ Byles (2004) 36 14.97 2.84 1.25 6.43 Institutionalisation

36+ Stuck (1995) 36 15.52 0.42 0.20 0.90 Institutionalisation

36+ Pathy (1992b) 36 16.48 0.56 0.29 1.09 Institutionalisation

36+ Stuck (2000) 36 18.64 1.51 0.99 2.30 Institutionalisation

36+ 0.90 0.49 1.67

0.1 0.2 0.5 1 2 5 10

Favours Home Visits Favours Controls

Institutionalisation (RR<1 favours home visits)

-2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0

0.0

0.5

1.0

1.5

2.0

Sta

nd

ard

Err

or

Log risk ratio

Funnel Plot of Standard Error by Log risk ratioMortality: Trim and Fill (missing studies shown, 1 trimmed)

-4 -3 -2 -1 0 1 2 3 4

0.0

0.2

0.4

0.6

0.8

1.0

Stan

dard

Err

or

Hedges's g

Funnel Plot of Standard Error by Hedges's gAnxiety (Self-Rated Symptoms) at Post-Treatment

Regression of Ave Age on Log risk ratio

Ave Age

Lo

g r

isk

rat

io

67.16 69.01 70.86 72.70 74.55 76.40 78.25 80.10 81.94 83.79 85.64

2.00

1.60

1.20

0.80

0.40

0.00

-0.40

-0.80

-1.20

-1.60

-2.00

Mortality by age: Meta-regression

Regression of Number of Contacts on Hedges's g

Number of Contacts

He

dg

es

's g

-1.35 1.47 4.29 7.11 9.93 12.75 15.57 18.39 21.21 24.03 26.85

2.00

1.72

1.44

1.16

0.88

0.60

0.32

0.04

-0.24

-0.52

-0.80

Anxiety (Self-Rated Symptoms) at Post-Treatment: Number of contacts with researchers and clinicians

Point Estimate SE P Q df P

Slope 0.03 <0.01 <0.001 Model 12.93 1 <0.001

Intercept 0.27 0.08 <0.01 Residual 30.34 27 0.30Total 43.27 28 0.03

Limitations

Junk-In, Junk-OutThe results of large trials

sometimes differChance Events: Aggregation and

Disaggregation

Conclusion

Systematic reviews seek to reduce bias and improve the reliability and accuracy of the conclusions.

Meta-analysis is a powerful research tool, but it should be conducted only in the context of a systematic review, and it has important limitations.

Documents

Systematic Reviews and Meta-Analysis