Upload
bathsheba-blake
View
218
Download
0
Tags:
Embed Size (px)
Citation preview
EPI-214: Lecture 1 Designing a Systematic Review (Meta-analysis)
Dejana Braithwaite
Assistant Professor
UCSF Department of Epidemiology and Biostatistics
April 11, 2013
Agenda
Introduction and definitions Steps of a systematic review Issues/Controversies Conclusions
What’s a Systematic Review?
“A review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyze data from the studies that are included in the review.”
Cochrane Collaboration
…and meta-analysis?
Statistical combination of >= 2 studies to produce single estimate of effect of exposure
*IPD= individual participant data
Systematic reviews
Meta-analyses
IPD*
The Cochrane Collaboration International systematic review initiative
• Archie Cochrane’s vision led to the opening of the first Cochrane centre (in Oxford, UK) in 1992 and the founding of the Cochrane Collaboration in 1993
Source: http://www.cochrane.org/cochrane/archieco.htm
Sattar N, et al. Statins and risk of incident diabetes: a collaborative meta-analysis of randomised statin trials. Lancet. 2010 Feb 27;375(9716):735-42.
Meta-analysis news coverage
Systematic review Driven by evidence-based medicine
movement and Cochrane collaboration
Advantages: Reduces bias Replicable Resolves controversy between conflicting findings Provides reliable basis for decision making
Use of meta-analysis as a prelude to clinical trials
Define pre-trial expected effect sizes sample size estimation Determine effect estimates in key
subgroups (e.g. based on gender, race/ethnicity or age)
Identify sources of heterogeneity in prior studies
Address these sources in design phase of new trial
Use of meta-analysis in study designs that are not clinical trials
Observational studies (e.g. case control, cohorts, cross-
sectional prevalence studies, etc.)
Studies evaluating diagnostic tests (sensitivity, specificity, predictive value)
“IPD” = individual patient data studies
Qualitative studies (meta-ethnography)
Resources required for systematic reviewing
Can be time consuming Team science (to reduce bias) Bibliographic software (e.g. Endnote) Statistical software (if appropriate)
*IE Allen & I Olkin: JAMA. 1999;282(7):634-635. doi:10-1001/pubs.JAMA-ISSN-0098-7484-282-7-jbk0818
Citations Retrieved for a Meta-analysis and Total Citations Retrieved for a Meta-analysis and Total Hours Required to Complete the Meta-analysis*Hours Required to Complete the Meta-analysis*
The mean total number of The mean total number of hours was 1139 (median, hours was 1139 (median, 1110), with a wide range from 1110), with a wide range from 216 to 2518 hours216 to 2518 hours..
(1) Pre-analysis search, retrieval, and database development: 588 (337) hours;
(2) statistical analysis & validation: 144 (106) hours;
(3) report and manuscript writing: 206 (125) hours;
(4) other (administrative): 201 (193) hours.
Total time=721 + 0.243Total time=721 + 0.243xx − 0.0000123 − 0.0000123xx22, where , where xx is the number is the number of citations before exclusion criteria are applied. of citations before exclusion criteria are applied.
1. Formulate research question Protocol
2. Identifying relevant work Apply inclusion /exclusion criteria
Analysis
3. Assessing the quality of studies
4. Summarizing evidence
5 steps of a systematic review
5. Interpreting the findings
Free-form question: Is it safe to provide population-wide drinking water fluoridation to prevent caries?
Structured question: The populations—Populations receiving drinking water sourced through a public water supply
The interventions or exposures—Fluoridation of drinking water (natural or artificial) compared with non-fluoridated water
The outcomes—Cancer is the main outcome of interest for the debate in your health authority
The study designs—Comparative studies of any design examining the harmful outcomes in at least two population groups, one with fluoridated drinking water and the other without.
Patient: Disease or condition Demographic characteristics
Intervention (or “Exposure”): Type of intervention Dose, duration, timing, etc.
Comparison: Absence of risk or treatment Placebo or alternative therapy
Components of a research question (PICOT)
Outcome: Risk or protective Dichotomous or continuous Type: mortality, quality of life, etc.
Type of Study: RCTs Cohort Case-control Cross-sectional All
Formulation of an etiology question
Is smoking a risk factor for breast cancer?
Are people who smoke regularly at a greater risk of developing breast cancer as compared to those who do not smoke?
Risk factor/Exposure Outcome
ExposurePatient
Outcome Comparison+ cohort & case-control studies
Protocol
Background Objectives Pre-determined selection criteria Planned search strategy Planned data abstraction Proposed method of synthesis of findings
1. Formulate research question
Protocol
2. Identifying relevant work Apply inclusion /exclusion criteria
Analysis
3. Assessing the quality of studies
4. Summarizing evidence
5 steps of a systematic review
5. Interpreting the findings
Where to locate studies
At least two of these: Pubmed Web of Science EMBASE Cochrane Central
Subject specific: PsychINFO CINAHL
Additional sources to identify studies for systematic reviews
Reference lists of retrieved articles Manual searching of relevant publications Experts in the field Corresponding or first authors of published
studies identified for the systematic review
Issues to consider
Publication bias Search bias
Inclusion/exclusion criteriaP - Population
I - Intervention
C - Comparison (if necessary)
O - Outcome
T - Type of study (if necessary)
Subject headings OR Textwords
To find studies using all of the PICO elements:
P and I and C and O (and T)
Exclusion criteria
Keep log of excluded studies Note reasons for exclusion Have eligibility checked by more than
one reviewer Develop strategy to resolve
disagreements
PRISMA-based search strategy flow-chart
Braithwaite et al Breast Ca Res Treatm 2012
EXAMPLE: SEARCH STRATEGY
You are interested in evaluating the benefits and harms associated with the use of screening mammography in women aged 70 and older. You need to determine what literature is available to conduct your review.
Quiz 1: You decide to conduct a systematic review (SR). What are two ways in which
SRs differ from narrative reviews?
They are the same thing
Systematic reviews use comprehensive searches and explicit methods
Systematic reviews always employ quantitative synthesis (meta-analysis)
Quiz 1 answers: SRs versus narrative reviews
They are the Same – Incorrect since narrative reviews do not employ explicit methods
Comprehensive Search and Explicit Methods – Correct!
Meta-analysis – Incorrect since SRs may or may not employ quantitative synthesis
You balanced precision and recall You have employed every possible
relevant citation regardless of the workload involved
That you limit the search so that only relevant citations are found
Quiz 2: When conducting a comprehensive search, what do
you need to assure?
Quiz 2 answers: When conducting a comprehensive search, what do you need
to assure? Balancing precision and recall - This is correct!
Precision - proportion of retrieved articles that are relevant.
Recall - proportion of potentially relevant articles retrieved by the search.
You’ve Gotten it All– incorrect since this is unnecessarily onerous
Only Have Relevant Citations in your Search – incorrect since overly restrictive strategy may miss citations
Quiz 4: You try to decide whether to do one search or two searches (one for benefits and another for harms). Which of the following would cause you to perform two searches?
Performing two searches is redundant and should be avoided
You decide a priori to only allow RCT data when evaluating benefits but allowing RCTs and observational trials for harms
If the available literature base on the topic is extensive
Never Do Multiple Searches - This is incorrect. Extensive literature may necessitate multiple searches.
Separate Benefits and Harms Search - This is correct! Benefits searches usually limited to RCTs due to greater internal validity. Harms searches usually broader.
When the available literature is extensive, two searches improves precision without appreciably impacting recall.
Quiz 4 answers: one versus multiple searches
Quiz 5: You decide to use hand searching of references from identified SRs, studies, and
abstract booklets from prominent meetings within the field of interest. Why would you do this?
Hand searching can capture citations that are not indexed or are improperly indexed.
Limiting hand searching to these sources can target this time intensive activity to areas where the yield will be the greatest.
Quiz 5: Value of hand searching
Both are Correct. Hand searching can yield up to 13-25% of citations that do not come up in database searches.
Since hand searching is the most time intensive search strategy, it should be targeted to maximize the yield.
Quiz 6: You decide to search the FDA website and ClinicalTrials.gov to identify studies that might be appropriate for your systematic review. These website are an
example of:
Grey literature Informally published written material
Quiz 6 answers: Grey Literature
Grey Literature or informally published written material – both are correct! This can be important for identifying
studies that were conducted but are, as of now, unpublished or incompletely published.
Can minimize publication bias.
Search strategy - summary
• A comprehensive understanding of the topic, the use of multiple searches, multiple databases, use of hand searching and grey literature, and development of a transparent reporting structure will give readers confidence in your systematic review.
1. Formulate research question
Protocol
2. Identifying relevant work Apply inclusion /exclusion criteria
Analysis
3. Assessing the quality of studies
4. Summarizing evidence
5 steps of a systematic review
5. Interpreting the findings
Principles of quality assessment
Quantitative studies Internal Validity
Allocation bias confounding Attrition Statistical analysis Intervention integrity Withdrawals and dropouts
External Validity generalizability or applicability
Quality Scoring of PublicationsQuality Scoring of Publications
Examines design and reporting
Jadad scoring technique (5 point Max)
Randomization (1 + 1 points)
Blinding (1 + 1 points)
Withdrawals (1 point)
http://en.wikipedia.org/wiki/Jadad_scale
(1996, Controlled Clinical Trials)
This followed the earlier 3 page quality scoring technique suggested by:Chalmers TC, Smith H, Blackburn B, Silverman B, Schroeder B, Reitman D, Ambroz A: A method for assessing the quality of a randomized control trial. Controlled Clinical Trials 2:31-49, 1981
Slide courtesy of I. Elaine Allen, PhD
PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)
statement
Evidence-based minimum set of items for reporting in SRs and meta-analyses
The aim of the PRISMA Statement is to help authors improve the reporting of SRs and meta-analyses
Focused on randomized trials but can also be used as a basis for reporting SRs of other types of research
www.prisma-statement.org/
Recruit
participants
Allocate to intervention and
control groups Intervention group Control group
Implement intervention Implement intervention
Follow-up participants Follow-up participants
Measure outcomes Measure outcomes
Analyze outcomes Analyze outcomes
CONFOUNDING
INTEGRITY OF INTERVENTION
INTENTION-TO-TREAT
WITHDRAWALS/ DROP OUTS
BLINDING OUTCOME ASSESSORS
DATA COLLECTION METHODS
STATISTICAL ANALYSIS
SELECTION BIAS
ALLOCATION BIAS
1. Formulate research question
Protocol
2. Identifying relevant work Apply inclusion /exclusion criteria
Analysis
3. Assessing the quality of studies
4. Summarizing evidence
5 steps of a systematic review
5. Interpreting the findings
Data abstraction
Design and pilot data abstraction form
Consider >1 reviewer
Consider blinding of observers to authors, institutions and journals
Data abstraction elements
publication detailsstudy designpopulation details (n, characteristics)intervention detailssettingoutcomes and findings
Synthesizing the Evidence
NARRATIVE SYNTHESIS findings summarized and explained
qualitatively
META-ANALYSIS findings summarized and then combined
statistically
Is there heterogeneity?
No Yes
Meta-analysis Narrative synthesis
Deal with heterogeneity?
(e.g. subgroup analysis)
1. Formulate research question
Protocol
2. Identifying relevant work Apply inclusion /exclusion criteria
Analysis
3. Assessing the quality of studies
4. Summarizing evidence
5 steps of a systematic review
5. Interpreting the findings
Interpretation of results
Strength of the evidence Applicability (generalizability) of
results Trade offs between benefits, harms
and costs (if applicable) Implications for practice
Strength of the evidence
Rate how strong the overall case for the use or cessation of the intervention is: How good is the quality of included trials? How large and significant are the observed
effects? How consistent are the effects across trials? Is there a dose-response relationship?
External factors Biological plausibility Other evidence Concordance with related reviews
“Our meta-analysis data suggest that Chinese herbal
medicine in the treatment of chronic hepatitis B infection
may have potential therapeutic value; however, because
the studies we found were of generally poor quality, we
are unable to make firm conclusions.”
Strength of the evidence: exampleReview on Chinese herbal medicine for hepatitis B
McCulloch M et al. Chinese herbal medicine and interferon in the treatment of chronic hepatitis B:
a meta-analysis of randomized controlled trials. Am J Pub Health 2002;92:1619-28
To whom can the review results be applied?
Are there any compelling reasons why the
evidence should not be applied under certain
circumstances?
Biological issues
Cultural issues
Variation in baseline risk
Technology, skill, cost, etc.
Applicability (generalizability) of results
Trade offs between benefits, harms and costs
Discuss adverse effects (potential for harm)
E.g. compute NNH (number needed to harm that
indicates how many patients need to be exposed to a risk
factor over a specific time period to cause harm in a patient
that would not have otherwise been harmed.
If possible, discuss cost issues
No need for a formal economic analysis!
For patient care or public health:
Review found no evidence at all or weak
evidence
Review found evidence that clearly supports
intervention
Review found clear evidence of lack of
benefit
Review found clear evidence of potential for
harm
Review found evidence of important trade-offs
between known benefits and known adverse
effects
Implications of the review
“The currently available reliable evidence does not show a
survival benefit of mass screening for breast cancer (and the
evidence is inconclusive for breast cancer mortality), whereas
it has been shown that mass screening leads to increased
use of aggressive treatment. Women, clinicians and policy
makers should consider these findings carefully when they
decide whether or not to attend or support screening
programs.”
Olsen O et al. http://image.thelancet.com/lancet/extra/fullreport.pdf
Example: Cochrane Mammography Review
Issues & Controversies
Apples and oranges or ‘fruit salad’Apples and oranges or ‘fruit salad’
Publication Bias (the file drawer problem)Publication Bias (the file drawer problem)
Multiple OutcomesMultiple Outcomes
Data from non-RCTs Data from non-RCTs
Data from uncontrolled studiesData from uncontrolled studies
Data from observational studiesData from observational studies
Accumulating data and ‘stopping’ rulesAccumulating data and ‘stopping’ rules
Individual patient dataIndividual patient data
Systematic Reviews: Strengths and Limitations
Findings of a review of 300 studies Not all systematic reviews equally reliable Reporting could be improved by a standard set of guidelines
Out of 100 systematic reviews monitored: 7% needed updating at the time of publication Another 4% within a year Another 11% within 2 years**
Findings of a 2003 study: Extending searches beyond major databases, perhaps into grey
literature, would increase the effectiveness of reviews***
The aim of any Meta-Analysis:The aim of any Meta-Analysis:
Courtesy I. Elaine Allen PhD
Your own systematic review
• How could you use a systematic review to answer a specific clinical research question?
• What are the challenges and risks?