EPI-214: Lecture 1 Designing a Systematic Review (Meta-analysis) Dejana Braithwaite Assistant...

EPI-214: Lecture 1 Designing a Systematic Review (Meta-analysis)

Dejana Braithwaite

Assistant Professor

UCSF Department of Epidemiology and Biostatistics

April 11, 2013

Agenda

Introduction and definitions Steps of a systematic review Issues/Controversies Conclusions

What’s a Systematic Review?

“A review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyze data from the studies that are included in the review.”

Cochrane Collaboration

…and meta-analysis?

Statistical combination of >= 2 studies to produce single estimate of effect of exposure

*IPD= individual participant data

Systematic reviews

Meta-analyses

IPD*

The Cochrane Collaboration International systematic review initiative

• Archie Cochrane’s vision led to the opening of the first Cochrane centre (in Oxford, UK) in 1992 and the founding of the Cochrane Collaboration in 1993

Source: http://www.cochrane.org/cochrane/archieco.htm

Sattar N, et al. Statins and risk of incident diabetes: a collaborative meta-analysis of randomised statin trials. Lancet. 2010 Feb 27;375(9716):735-42.

Meta-analysis news coverage

Systematic review Driven by evidence-based medicine

movement and Cochrane collaboration

Advantages: Reduces bias Replicable Resolves controversy between conflicting findings Provides reliable basis for decision making

Use of meta-analysis as a prelude to clinical trials

Define pre-trial expected effect sizes sample size estimation Determine effect estimates in key

subgroups (e.g. based on gender, race/ethnicity or age)

Identify sources of heterogeneity in prior studies

Address these sources in design phase of new trial

Use of meta-analysis in study designs that are not clinical trials

Observational studies (e.g. case control, cohorts, cross-

sectional prevalence studies, etc.)

Studies evaluating diagnostic tests (sensitivity, specificity, predictive value)

“IPD” = individual patient data studies

Qualitative studies (meta-ethnography)

Resources required for systematic reviewing

Can be time consuming Team science (to reduce bias) Bibliographic software (e.g. Endnote) Statistical software (if appropriate)

*IE Allen & I Olkin: JAMA. 1999;282(7):634-635. doi:10-1001/pubs.JAMA-ISSN-0098-7484-282-7-jbk0818

Citations Retrieved for a Meta-analysis and Total Citations Retrieved for a Meta-analysis and Total Hours Required to Complete the Meta-analysis*Hours Required to Complete the Meta-analysis*

The mean total number of The mean total number of hours was 1139 (median, hours was 1139 (median, 1110), with a wide range from 1110), with a wide range from 216 to 2518 hours216 to 2518 hours..

(1) Pre-analysis search, retrieval, and database development: 588 (337) hours;

(2) statistical analysis & validation: 144 (106) hours;

(3) report and manuscript writing: 206 (125) hours;

(4) other (administrative): 201 (193) hours.

Total time=721 + 0.243Total time=721 + 0.243xx − 0.0000123 − 0.0000123xx22, where , where xx is the number  is the number of citations before exclusion criteria are applied. of citations before exclusion criteria are applied. 

1. Formulate research question Protocol

2. Identifying relevant work Apply inclusion /exclusion criteria

Analysis

3. Assessing the quality of studies

4. Summarizing evidence

5 steps of a systematic review

5. Interpreting the findings

Free-form question: Is it safe to provide population-wide drinking water fluoridation to prevent caries?

Structured question: The populations—Populations receiving drinking water sourced through a public water supply

The interventions or exposures—Fluoridation of drinking water (natural or artificial) compared with non-fluoridated water

The outcomes—Cancer is the main outcome of interest for the debate in your health authority

The study designs—Comparative studies of any design examining the harmful outcomes in at least two population groups, one with fluoridated drinking water and the other without.

Patient: Disease or condition Demographic characteristics

Intervention (or “Exposure”): Type of intervention Dose, duration, timing, etc.

Comparison: Absence of risk or treatment Placebo or alternative therapy

Components of a research question (PICOT)

Outcome: Risk or protective Dichotomous or continuous Type: mortality, quality of life, etc.

Type of Study: RCTs Cohort Case-control Cross-sectional All

Formulation of an etiology question

Is smoking a risk factor for breast cancer?

Are people who smoke regularly at a greater risk of developing breast cancer as compared to those who do not smoke?

Risk factor/Exposure Outcome

ExposurePatient

Outcome Comparison+ cohort & case-control studies

Protocol

Background Objectives Pre-determined selection criteria Planned search strategy Planned data abstraction Proposed method of synthesis of findings

1. Formulate research question

Protocol

2. Identifying relevant work Apply inclusion /exclusion criteria

Analysis

3. Assessing the quality of studies

4. Summarizing evidence

5 steps of a systematic review

5. Interpreting the findings

Where to locate studies

At least two of these: Pubmed Web of Science EMBASE Cochrane Central

Subject specific: PsychINFO CINAHL

Additional sources to identify studies for systematic reviews

Reference lists of retrieved articles Manual searching of relevant publications Experts in the field Corresponding or first authors of published

studies identified for the systematic review

Issues to consider

Publication bias Search bias

Inclusion/exclusion criteriaP - Population

I - Intervention

C - Comparison (if necessary)

O - Outcome

T - Type of study (if necessary)

Subject headings OR Textwords

To find studies using all of the PICO elements:

P and I and C and O (and T)

Exclusion criteria

Keep log of excluded studies Note reasons for exclusion Have eligibility checked by more than

one reviewer Develop strategy to resolve

disagreements

PRISMA-based search strategy flow-chart

Braithwaite et al Breast Ca Res Treatm 2012

EXAMPLE: SEARCH STRATEGY

You are interested in evaluating the benefits and harms associated with the use of screening mammography in women aged 70 and older. You need to determine what literature is available to conduct your review.

Quiz 1: You decide to conduct a systematic review (SR). What are two ways in which

SRs differ from narrative reviews?

They are the same thing

Systematic reviews use comprehensive searches and explicit methods

Systematic reviews always employ quantitative synthesis (meta-analysis)

Quiz 1 answers: SRs versus narrative reviews

They are the Same – Incorrect since narrative reviews do not employ explicit methods

Comprehensive Search and Explicit Methods – Correct!

Meta-analysis – Incorrect since SRs may or may not employ quantitative synthesis

You balanced precision and recall You have employed every possible

relevant citation regardless of the workload involved

That you limit the search so that only relevant citations are found

Quiz 2: When conducting a comprehensive search, what do

you need to assure?

Quiz 2 answers: When conducting a comprehensive search, what do you need

to assure? Balancing precision and recall - This is correct!

Precision - proportion of retrieved articles that are relevant.

Recall - proportion of potentially relevant articles retrieved by the search.

You’ve Gotten it All– incorrect since this is unnecessarily onerous

Only Have Relevant Citations in your Search – incorrect since overly restrictive strategy may miss citations

Quiz 4: You try to decide whether to do one search or two searches (one for benefits and another for harms). Which of the following would cause you to perform two searches?

Performing two searches is redundant and should be avoided

You decide a priori to only allow RCT data when evaluating benefits but allowing RCTs and observational trials for harms

If the available literature base on the topic is extensive

Never Do Multiple Searches - This is incorrect. Extensive literature may necessitate multiple searches.

Separate Benefits and Harms Search - This is correct! Benefits searches usually limited to RCTs due to greater internal validity. Harms searches usually broader.

When the available literature is extensive, two searches improves precision without appreciably impacting recall.

Quiz 4 answers: one versus multiple searches

Quiz 5: You decide to use hand searching of references from identified SRs, studies, and

abstract booklets from prominent meetings within the field of interest. Why would you do this?

Hand searching can capture citations that are not indexed or are improperly indexed.

Limiting hand searching to these sources can target this time intensive activity to areas where the yield will be the greatest.

Quiz 5: Value of hand searching

Both are Correct. Hand searching can yield up to 13-25% of citations that do not come up in database searches.

Since hand searching is the most time intensive search strategy, it should be targeted to maximize the yield.

Quiz 6: You decide to search the FDA website and ClinicalTrials.gov to identify studies that might be appropriate for your systematic review. These website are an

example of:

Grey literature Informally published written material

Quiz 6 answers: Grey Literature

Grey Literature or informally published written material – both are correct! This can be important for identifying

studies that were conducted but are, as of now, unpublished or incompletely published.

Can minimize publication bias.

Search strategy - summary

• A comprehensive understanding of the topic, the use of multiple searches, multiple databases, use of hand searching and grey literature, and development of a transparent reporting structure will give readers confidence in your systematic review.

1. Formulate research question

Protocol

2. Identifying relevant work Apply inclusion /exclusion criteria

Analysis

3. Assessing the quality of studies

4. Summarizing evidence

5 steps of a systematic review

5. Interpreting the findings

Principles of quality assessment

Quantitative studies Internal Validity

Allocation bias confounding Attrition Statistical analysis Intervention integrity Withdrawals and dropouts

External Validity generalizability or applicability

Quality Scoring of PublicationsQuality Scoring of Publications

Examines design and reporting

Jadad scoring technique (5 point Max)

Randomization (1 + 1 points)

Blinding (1 + 1 points)

Withdrawals (1 point)

http://en.wikipedia.org/wiki/Jadad_scale

(1996, Controlled Clinical Trials)

This followed the earlier 3 page quality scoring technique suggested by:Chalmers TC, Smith H, Blackburn B, Silverman B, Schroeder B, Reitman D, Ambroz A: A method for assessing the quality of a randomized control trial. Controlled Clinical Trials 2:31-49, 1981

Slide courtesy of I. Elaine Allen, PhD

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)

statement

Evidence-based minimum set of items for reporting in SRs and meta-analyses

The aim of the PRISMA Statement is to help authors improve the reporting of SRs and meta-analyses

Focused on randomized trials but can also be used as a basis for reporting SRs of other types of research

www.prisma-statement.org/

Recruit

participants

Allocate to intervention and

control groups Intervention group Control group

Implement intervention Implement intervention

Follow-up participants Follow-up participants

Measure outcomes Measure outcomes

Analyze outcomes Analyze outcomes

CONFOUNDING

INTEGRITY OF INTERVENTION

INTENTION-TO-TREAT

WITHDRAWALS/ DROP OUTS

BLINDING OUTCOME ASSESSORS

DATA COLLECTION METHODS

STATISTICAL ANALYSIS

SELECTION BIAS

ALLOCATION BIAS

1. Formulate research question

Protocol

2. Identifying relevant work Apply inclusion /exclusion criteria

Analysis

3. Assessing the quality of studies

4. Summarizing evidence

5 steps of a systematic review

5. Interpreting the findings

Data abstraction

Design and pilot data abstraction form

Consider >1 reviewer

Consider blinding of observers to authors, institutions and journals

Data abstraction elements

publication detailsstudy designpopulation details (n, characteristics)intervention detailssettingoutcomes and findings

Synthesizing the Evidence

NARRATIVE SYNTHESIS findings summarized and explained

qualitatively

META-ANALYSIS findings summarized and then combined

statistically

Is there heterogeneity?

No Yes

Meta-analysis Narrative synthesis

Deal with heterogeneity?

(e.g. subgroup analysis)

1. Formulate research question

Protocol

2. Identifying relevant work Apply inclusion /exclusion criteria

Analysis

3. Assessing the quality of studies

4. Summarizing evidence

5 steps of a systematic review

5. Interpreting the findings

Interpretation of results

Strength of the evidence Applicability (generalizability) of

results Trade offs between benefits, harms

and costs (if applicable) Implications for practice

Strength of the evidence

Rate how strong the overall case for the use or cessation of the intervention is: How good is the quality of included trials? How large and significant are the observed

effects? How consistent are the effects across trials? Is there a dose-response relationship?

External factors Biological plausibility Other evidence Concordance with related reviews

“Our meta-analysis data suggest that Chinese herbal

medicine in the treatment of chronic hepatitis B infection

may have potential therapeutic value; however, because

the studies we found were of generally poor quality, we

are unable to make firm conclusions.”

Strength of the evidence: exampleReview on Chinese herbal medicine for hepatitis B

McCulloch M et al. Chinese herbal medicine and interferon in the treatment of chronic hepatitis B:

a meta-analysis of randomized controlled trials. Am J Pub Health 2002;92:1619-28

To whom can the review results be applied?

Are there any compelling reasons why the

evidence should not be applied under certain

circumstances?

Biological issues

Cultural issues

Variation in baseline risk

Technology, skill, cost, etc.

Applicability (generalizability) of results

Trade offs between benefits, harms and costs

Discuss adverse effects (potential for harm)

E.g. compute NNH (number needed to harm that

indicates how many patients need to be exposed to a risk

factor over a specific time period to cause harm in a patient

that would not have otherwise been harmed.

If possible, discuss cost issues

No need for a formal economic analysis!

For patient care or public health:

Review found no evidence at all or weak

evidence

Review found evidence that clearly supports

intervention

Review found clear evidence of lack of

benefit

Review found clear evidence of potential for

harm

Review found evidence of important trade-offs

between known benefits and known adverse

effects

Implications of the review

“The currently available reliable evidence does not show a

survival benefit of mass screening for breast cancer (and the

evidence is inconclusive for breast cancer mortality), whereas

it has been shown that mass screening leads to increased

use of aggressive treatment. Women, clinicians and policy

makers should consider these findings carefully when they

decide whether or not to attend or support screening

programs.”

Olsen O et al. http://image.thelancet.com/lancet/extra/fullreport.pdf

Example: Cochrane Mammography Review

Issues & Controversies

Apples and oranges or ‘fruit salad’Apples and oranges or ‘fruit salad’

Publication Bias (the file drawer problem)Publication Bias (the file drawer problem)

Multiple OutcomesMultiple Outcomes

Data from non-RCTs Data from non-RCTs

Data from uncontrolled studiesData from uncontrolled studies

Data from observational studiesData from observational studies

Accumulating data and ‘stopping’ rulesAccumulating data and ‘stopping’ rules

Individual patient dataIndividual patient data

Systematic Reviews: Strengths and Limitations

Findings of a review of 300 studies Not all systematic reviews equally reliable Reporting could be improved by a standard set of guidelines

Out of 100 systematic reviews monitored: 7% needed updating at the time of publication Another 4% within a year Another 11% within 2 years**

Findings of a 2003 study: Extending searches beyond major databases, perhaps into grey

literature, would increase the effectiveness of reviews***

The aim of any Meta-Analysis:The aim of any Meta-Analysis:

Courtesy I. Elaine Allen PhD

Your own systematic review

• How could you use a systematic review to answer a specific clinical research question?

• What are the challenges and risks?