Systematic Review Lecture 3: Advanced Topics Stephen Bent, MD Assistant Professor of Medicine, UCSF Course Director, Systematic Reviews (EPI 214)

Systematic ReviewSystematic ReviewLecture 3: Lecture 3:

Advanced TopicsAdvanced TopicsStephen Bent, MDStephen Bent, MD

Assistant Professor of Medicine, UCSFAssistant Professor of Medicine, UCSFCourse Director, Systematic Reviews Course Director, Systematic Reviews

(EPI 214)(EPI 214)

8 Steps to Systematic 8 Steps to Systematic ReviewReview

1. Research Question1. Research Question 2. Protocol2. Protocol 3. Search3. Search 4. Study selection (inclusion/exclusion)4. Study selection (inclusion/exclusion) 5. Quality assessment5. Quality assessment 6. Data abstraction6. Data abstraction 7. Analysis7. Analysis

A) Create summary measureA) Create summary measure B) Assess for heterogeneityB) Assess for heterogeneity C) Assess for publication biasC) Assess for publication bias D) Conduct sensitivity/subgroup analysesD) Conduct sensitivity/subgroup analyses E) Advanced issues/techniquesE) Advanced issues/techniques

8. Interpretation8. Interpretation

Bottom LineBottom Line

You can combine just about You can combine just about anything.anything.

Use judgment to decide whether it is Use judgment to decide whether it is appropriate and clinically appropriate and clinically meaningful to combine data on a meaningful to combine data on a particular topic.particular topic.

Can you conduct a systematic review when there are only a few studies?

Parachute use to prevent death and major trauma related

to gravitational challenge: systematic review of

randomised controlled trials Objectives To determine whether parachutes are effective in preventing major trauma related to gravitational challenge.

Design Systematic review of randomised controlled trials. Data sources: Medline,Web of Science, Embase, and the Cochrane

Library databases; appropriate internet sites and citation lists. Study selection: Studies showing the effects of using a parachute

during free fall. Main outcome measure Death or major trauma. Results We were unable to identify any randomised controlled trials

of parachute intervention. Conclusions As with many interventions intended to prevent ill

health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials. Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute.

Smith GCS and Pill JP. BMJ 2003;327:1459–61

OverviewOverview

Individual participant dataIndividual participant data Missing dataMissing data Different types of dataDifferent types of data Observational studiesObservational studies Generalized synthesis of evidenceGeneralized synthesis of evidence Meta-regressionMeta-regression Critique of a systematic reviewCritique of a systematic review

Individual Participant Individual Participant DataData

Identify all relevant studies and Identify all relevant studies and obtain full datasetsobtain full datasets

Potential advantages:Potential advantages: Detailed data checkingDetailed data checking Ensure appropriateness of analysesEnsure appropriateness of analyses Subgroup analysesSubgroup analyses Survival and other time-to-event Survival and other time-to-event

analysesanalyses Standardize inclusion/exclusion criteriaStandardize inclusion/exclusion criteria

Individual Participant Individual Participant DataData

Potential disadvantagesPotential disadvantages TimeTime CostCost No studies have yet documented No studies have yet documented

superioritysuperiority ““standard meta-analysis estimates of treatment standard meta-analysis estimates of treatment

contrasts are identical to the analysis of contrasts are identical to the analysis of individual participant data when homogeneity is individual participant data when homogeneity is assumed.”assumed.”

Conclusion: Conclusion: more evidence needed to justify IPDmore evidence needed to justify IPD Start with standard meta-analysisStart with standard meta-analysis

OverviewOverview

Individual participant dataIndividual participant data Missing dataMissing data Different types of dataDifferent types of data Observational studiesObservational studies Generalized synthesis of evidenceGeneralized synthesis of evidence Meta-regressionMeta-regression Survival dataSurvival data

Missing DataMissing Data

3 Types to consider3 Types to consider Whole studies (publication bias)Whole studies (publication bias) Data missing at the study level Data missing at the study level

Missing study effect sizes (the actual magnitude of Missing study effect sizes (the actual magnitude of effect may not be shown, especially when effect may not be shown, especially when negative).negative).

Missing measures of precision (no SD, for example)Missing measures of precision (no SD, for example) Missing information on covariatesMissing information on covariates

Data missing at the individual patient level Data missing at the individual patient level (when doing meta-analysis of individual (when doing meta-analysis of individual patient data)patient data)

Missing DataMissing Data

General approachesGeneral approaches Analyze only the complete casesAnalyze only the complete cases Single-value imputation (missing values Single-value imputation (missing values

replaced with “reasonable” estimates)replaced with “reasonable” estimates) Regression imputation (missing Regression imputation (missing

variables assumed to be linearly variables assumed to be linearly related, and regression used)related, and regression used)

Multiple imputation – multiple values Multiple imputation – multiple values imputed for each missing observation.imputed for each missing observation.

Example: Missing SD’sExample: Missing SD’s

Meta-analysis of SSRI’s compared to TCA’sMeta-analysis of SSRI’s compared to TCA’s 53 RCT’s identified, all used Hamilton Scale53 RCT’s identified, all used Hamilton Scale

Only 20 provided SD’sOnly 20 provided SD’s This would require exclusion of 33 studies!This would require exclusion of 33 studies!

The authors pooled the SD’s of the scale for The authors pooled the SD’s of the scale for the studies reporting this outcome, and the studies reporting this outcome, and used this as an estimate in studies where it used this as an estimate in studies where it was not reported.was not reported.

Song F et al. Selective serotonin reuptake inhibitors: a meta-analysis of efficacy and Acceptability. BMJ. 1993;306:683-7.

SSRI’s vs. TCA’sSSRI’s vs. TCA’s

Results:Results: Only studies reporting SD’s Only studies reporting SD’s

Mean difference = 0.13 (95% CI, -1.01 to Mean difference = 0.13 (95% CI, -1.01 to 1.28)1.28)

All studies using pooled SD’sAll studies using pooled SD’s Mean difference = 0.15 (95% CI, -0.60 to Mean difference = 0.15 (95% CI, -0.60 to

0.89)0.89)

Another common variation Another common variation of missing information on of missing information on

SD’sSD’s Studies may report mean responses Studies may report mean responses

and SD’s at baseline and follow-upand SD’s at baseline and follow-up Often do NOT report the SD of the Often do NOT report the SD of the

change (which is required to account change (which is required to account for correlation between individuals).for correlation between individuals).

A sensitivity analysis can be A sensitivity analysis can be conducted, imputing a range of conducted, imputing a range of values for this missing correlation.values for this missing correlation.

Weight loss study – no SD of Weight loss study – no SD of changechange

Treatment Group*

Baseline weight in kg

(SD)

Week 6 weight in kg

(SD)

Percent Change

p-value†

Citrus aurantium product

90.9 (17.5) 89.5 (16) -1.5% 0.05

Placebo

83.6 (17.5) 82.7 (18) -1.1% 0.10

No-placebo control

78.1 (11.5) 77.7 (10.5) -0.05%‡ Not reported

Colker et al. 1999. Current Therapeutic Research

OverviewOverview


Different types of dataDifferent types of data

Different scales (example)Different scales (example) Ordinal dataOrdinal data Binary dataBinary data Continuous outcomesContinuous outcomes Diagnostic tests (example)Diagnostic tests (example)

RCTs in Acute Bronchitis: RCTs in Acute Bronchitis: Different ScalesDifferent Scales

Study, Study, yryr

NN AbxAbx OutcomeOutcome ResultResult

Stott, 76Stott, 76 202077

DoxyDoxy Days of Yellow SpitDays of Yellow Spit 0.6 (-0.2 to 0.6 (-0.2 to 1.4)1.4)

Franks, 84Franks, 84 5454 TMP/TMP/SS

Cough Amount ScoreCough Amount Score 0.2 (-0.2 to 0.2 (-0.2 to 0.6)0.6)

Williamson, Williamson, 8484

6969 DoxyDoxy Days of Purulent Days of Purulent SputumSputum

-0.2 (-1.2 to -0.2 (-1.2 to 0.8)0.8)

Dunlay, 87Dunlay, 87 4545 ErythErythroro

Sputum production Sputum production scorescore

0.5 (0.1 to 0.5 (0.1 to 0.9)0.9)

Scherl, 87Scherl, 87 3131 DoxyDoxy Days of sputumDays of sputum 1.9 (-0.2 to 1.9 (-0.2 to 4.0)4.0)

Verheij, 94Verheij, 94 141400

DoxyDoxy Days of productive Days of productive coughcough

0.5 (-0.4 to 0.5 (-0.4 to 1.4)1.4)

Hueston, 94Hueston, 94 2323 ErythErythroro

Days of productive Days of productive coughcough

-0.4 (-2.4 to -0.4 (-2.4 to 1.6)1.6)

King, 96King, 96 9191 ErythErythroro

Days of sputum Days of sputum productionproduction

0.7 (-1.3 to 0.7 (-1.3 to 2.7)2.7)

ProblemProblem

How do you combine studies with How do you combine studies with slightly different outcomes?slightly different outcomes?

Option 1: - don’t do itOption 1: - don’t do it Option 2: Transform all outcomes to Option 2: Transform all outcomes to

an effect sizean effect size

What is an Effect Size?What is an Effect Size?

Effect size – a way of expressing Effect size – a way of expressing results in a common metricresults in a common metric

Units – standard deviationUnits – standard deviation

Effect SizeEffect Size

ES = ES = XX11 – X – X22

SDSDpooledpooled

1.1. ES increases as difference between ES increases as difference between means increasesmeans increases

2.2. ES increases as SD decreasesES increases as SD decreases

3.3. ES is expressed in units of SDES is expressed in units of SD

4.4. Summary ES combines the weighted ES Summary ES combines the weighted ES from each study.from each study.



Rough EstimatesRough Estimates SMALLSMALL 0.20.2 MEDIUMMEDIUM 0.50.5 LARGELARGE >0.7>0.7

ContextContext Mean Duration of Cough = 8 daysMean Duration of Cough = 8 days Standard Deviation = 3 daysStandard Deviation = 3 days

Main ResultMain Result

Summary ES = 0.21 (95% CI 0.05 to Summary ES = 0.21 (95% CI 0.05 to 0.36)0.36)

Summary Mean Summary Mean DifferencesDifferences

Outcome MeasureOutcome Measure Summary Mean Summary Mean Difference (95% CI)Difference (95% CI)

Days of Productive Days of Productive Cough (6 studies)Cough (6 studies)

0.4 days (-0.1 to 0.8)0.4 days (-0.1 to 0.8)

Days of cough (4 Days of cough (4 studies)studies)

0.5 days (-0.1 to 1.1)0.5 days (-0.1 to 1.1)

Time off work (6 Time off work (6 studies)studies)

0.3 days (-0.6 to 1.1)0.3 days (-0.6 to 1.1)

Different Types of Data: Different Types of Data: Diagnostic TestsDiagnostic Tests

Sensitivity and Sensitivity and SpecificitySpecificity

Sensitivity Sensitivity TP/(TP + FN)TP/(TP + FN)Positive in Positive in DiseaseDisease

SpecificitySpecificityTN/(TN + FP)TN/(TN + FP)Negative in Negative in HealthHealth

TNFNTest

-

FPTPTest

+

Disease

-

Disease

+

(+) Likelihood Ratio =SensitivitySensitivity

1-Specificity1-Specificity

(-) Likelihood Ratio =1-1-SensitivitySensitivity SpecificitySpecificity

Likelihood ratiosLikelihood ratios

LR+ = likelihood of + test in patient WITH dz.LR+ = likelihood of + test in patient WITH dz.

likelihood of a + test in patient WITHOUT likelihood of a + test in patient WITHOUT dz.dz.

LR- = LR- = likelihood of a negative test in patient likelihood of a negative test in patient WITH dz. WITH dz. likelihood of a negative likelihood of a negative test in a patient WITHOUT dz.test in a patient WITHOUT dz.

Does this patient have a Does this patient have a specific disease?specific disease?

What we thought before What we thought before (pre-test probability)(pre-test probability)

+ Clinical information + Clinical information (diagnostic test, LR)(diagnostic test, LR)

= What we think after = What we think after (post-test probability)(post-test probability)

Diagnostic OR = Diagnostic OR = +LR/-LR+LR/-LR

= TP x TN / FP x FN= TP x TN / FP x FNSensitivity Specificity Pos LR Neg LR Diag OR

0.5 0.5 1 1 10.6 0.6 1.5 0.67 2.30.7 0.7 2.3 0.43 5.40.8 0.8 4 0.25 160.9 0.9 9 0.11 810.95 0.95 19 0.05 3610.99 0.99 99 0.01 9801

SummarySummarySensitivity and Sensitivity and

SpecificitySpecificityTest Results With disease Without disease

Positive True positive False positiveTotal positive

Negative False negative True negativeTotal negative

Total w/disease Total w/o disease

Participants

TP(i)

FN(i)

FP(i)

TN(i)

n1 n2

Summary sensitivity =∑TP(i) [positives]

∑TP(i)+FN(i) [with disease]

∑TN(i) [negatives]

∑TN(i)+FP(i) [without disease]Summary specificity =

Quality of Studies of Quality of Studies of Diagnostic TestsDiagnostic Tests

A recently evaluated tool assesses the A recently evaluated tool assesses the quality of studies of diagnostic tests quality of studies of diagnostic tests ((QUADAS - Quality Assessment of Diagnostic Accuracy Studies)

In general, you need to assess all 3 parts of In general, you need to assess all 3 parts of the research question:the research question: E.g., in patients with symptom/presentation X, E.g., in patients with symptom/presentation X,

does the use of Y test predict disease Z?does the use of Y test predict disease Z? So, are patients well characterized and typical?So, are patients well characterized and typical? Is the test well described?Is the test well described? Is there an adequate gold standard?Is there an adequate gold standard? Do all patients get the gold standard? If not – Do all patients get the gold standard? If not –

potential for “verification bias.”potential for “verification bias.”

Example: US and CT for Example: US and CT for AppendicitisAppendicitis

Goal: to determine whether US or CT Goal: to determine whether US or CT is a “better” test for the evaluation of is a “better” test for the evaluation of suspected appendicitis.suspected appendicitis.

Diagnostic tests are complicated Diagnostic tests are complicated because there are 5 potential because there are 5 potential outcomes to summarizeoutcomes to summarize LR+, LR-LR+, LR- Sensitivity, SpecificitySensitivity, Specificity Diagnostic ORDiagnostic OR Assess heterogeneity, publication bias for Assess heterogeneity, publication bias for

EACH outcomeEACH outcome

Other Systematic Reviews of Other Systematic Reviews of Diagnostic Tests: The JAMA Diagnostic Tests: The JAMA

Rational Clinical Examination Rational Clinical Examination SeriesSeries

Very interesting type of review for those Very interesting type of review for those interested in the value of the history and interested in the value of the history and physical examination.physical examination. Both history and physical are diagnostic testsBoth history and physical are diagnostic tests Helps define the interview questions and exam Helps define the interview questions and exam

maneuversmaneuvers

Examples:Examples: Does this woman have a UTI?Does this woman have a UTI? Does this patient with a swollen joint have septic Does this patient with a swollen joint have septic

arthritis?arthritis?

Case 1Case 1

A 32 y.o. woman leaves a phone A 32 y.o. woman leaves a phone message at your office saying she message at your office saying she has dysuria and wants to know what has dysuria and wants to know what to do.to do.

11stst step: Formulate a differential step: Formulate a differential diagnosis and assign pre-test diagnosis and assign pre-test

probabilities.probabilities.

DiagnosisDiagnosis ProbabilityProbability

UTIUTI 50%50%

STDSTD 20%20%

Vaginal Yeast InfxnVaginal Yeast Infxn 20%20%

Bacterial VaginosisBacterial Vaginosis 10%10%

OtherOther < 1%< 1%

General rules about LR’sGeneral rules about LR’s

LR+LR++2 – increases post-test prob 15%+2 – increases post-test prob 15%+5 – increases post-test prob 30%+5 – increases post-test prob 30%+10 – increases post-test prob 45%+10 – increases post-test prob 45%

LR-LR-0.5 – decreases post-test prob 15%0.5 – decreases post-test prob 15%0.2 – decreases post-test prob 30%0.2 – decreases post-test prob 30%0.1 – decreases post-test prob 45%0.1 – decreases post-test prob 45%

Likelihood ratios for Likelihood ratios for suspected UTIsuspected UTI

TestTest LR+LR+ LR-LR-

DysuriaDysuria 1.51.5 0.50.5

FrequencyFrequency 1.81.8 (NS)(NS)

HematuriaHematuria 2.02.0 (NS)(NS)

Vag D/CVag D/C 0.30.3 3.13.1

DipstickDipstick 4.24.2 0.30.3

Self Self DiagnosisDiagnosis

4.04.0

Bent et al. JAMA. 2002;287:2701-2710.

Copyright restrictions may apply.

Bent, S. et al. JAMA 2002;287:2701-2710.

Proposed Algorithm for Evaluating a Women With Symptoms of Acute Urinary Tract Infection (UTI)

OverviewOverview


Meta-analysis of Meta-analysis of observational studiesobservational studies

Traditionally, most meta-analyses Traditionally, most meta-analyses were done with RCT’swere done with RCT’s

What are the unique concerns with What are the unique concerns with doing meta-analyses of observational doing meta-analyses of observational studies?studies?

Concerns with Meta-Concerns with Meta-analyses of Observational analyses of Observational

StudiesStudies Potential for greater publication biasPotential for greater publication bias In studies that are published, authors In studies that are published, authors

may only report outcomes with the may only report outcomes with the greatest statistical significance.greatest statistical significance.

Some studies may present RR’s Some studies may present RR’s (cohort), and others OR’s (case-control).(cohort), and others OR’s (case-control).

*Different studies adjust for different *Different studies adjust for different potential confounders (age, SES, DM, potential confounders (age, SES, DM, HTN, Medication, etc).HTN, Medication, etc).

Dealing with variability in Dealing with variability in observational studiesobservational studies

Many different methods are availableMany different methods are available For example, if one study adjusts for a For example, if one study adjusts for a

potential confounder – estimates of the effect potential confounder – estimates of the effect from this confounder can be applied to the from this confounder can be applied to the data in the unadjusted study (if raw data on data in the unadjusted study (if raw data on the prevalence of the confounder is present).the prevalence of the confounder is present).

If you are combining data from markedly If you are combining data from markedly different observational studies, consider different observational studies, consider using these adjustment techniques as using these adjustment techniques as sensitivity analyses.sensitivity analyses.

Author Study Design Sample Size Source of ALA RR 95% CI

Giovannucci, et al., 1993†

United States

Prospective cohort

47,855 Diet 1.25 0.82–1.92

Gann, et al., 1994

United States

Nested case/control

120 cases/120 controls Plasma 2.22 0.93–5.29

Godley, et al., 1996

United States

Case/control 89 cases/38 controls ErythrocytesAdipose tissue

1.692.73

0.54–5.26 0.70–10.61

Andersson, et al., 1996

Sweden

Case/control 526 cases/536 controls Diet 0.93 0.65-1.32

Harvei, et al., 1997

Norway

Nested case/control

141 cases/282 controls Serum 2.0 1.1-3.6

Meyer, et al., 1997

Canada

Case/control 215 cases/593 controls Diet 0.98 0.54-1.78

ALA and Prostate CancerALA and Prostate Cancer

Example: ALA and Prostate Example: ALA and Prostate Cancer RiskCancer Risk

RR=1.2 (1.01 to 1.43), Test for heterogeneity, p=0.00

ALA – Funnel PlotALA – Funnel Plot

Funnel Plot with Imputed Funnel Plot with Imputed Values for Publication BiasValues for Publication Bias

RR=0.94, 95% CI: 0.79-1.17

OverviewOverview


Generalized synthesis of Generalized synthesis of evidenceevidence

Methods have been developed to Methods have been developed to combine RCT’s, observational studies, combine RCT’s, observational studies, and even single-arm (no control) and even single-arm (no control) studies.studies.

In simple terms, a treatment effect is In simple terms, a treatment effect is calculated for each arm of each study – calculated for each arm of each study – treatment arms are then combined and treatment arms are then combined and these summary estimates are compared.these summary estimates are compared.

““the authors stress that caution should the authors stress that caution should be used at the interpretation stage be used at the interpretation stage when using this method”when using this method”

OverviewOverview

Individual participant dataIndividual participant data Missing dataMissing data Different types of dataDifferent types of data Observational studiesObservational studies Generalized synthesis of evidenceGeneralized synthesis of evidence Meta-regressionMeta-regression Critique of a systematic reviewCritique of a systematic review

Meta-regressionMeta-regression Examines whether the study effects Examines whether the study effects

(outcomes) are related to one or more (outcomes) are related to one or more of the study characteristics.of the study characteristics.

Can be used to understand/explain Can be used to understand/explain heterogeneity.heterogeneity.

Can be thought of as an Can be thought of as an epidemiological study of the trials or epidemiological study of the trials or studies.studies.

Clinical Questions: Clinical Questions: Meta-RegressionMeta-Regression

1.1. Are there certain situations in which BCG Are there certain situations in which BCG may be more effective for preventing TB?may be more effective for preventing TB?

2.2. What characteristics (clinical and study What characteristics (clinical and study design) might explain heterogeneity of design) might explain heterogeneity of the sensitivity and specificity of echo?the sensitivity and specificity of echo?

3.3. If fish oil lowers heart rate, what If fish oil lowers heart rate, what population and preparation population and preparation characteristics may relate to its greatest characteristics may relate to its greatest effect?effect?

Meta-regression: exampleMeta-regression: exampleStudyStudy OROR 95% CI95% CI

11 0.3910.391 0.121, 1.2620.121, 1.262

22 0.1890.189 0.077, 0.4620.077, 0.462

33 0.2500.250 0.069, 0.9090.069, 0.909

44 0.2330.233 0.176, 0.3080.176, 0.308

55 0.8030.803 0.514, 1.2560.514, 1.256

66 0.3840.384 0.316, 0.4660.316, 0.466

77 0.1950.195 0.077, 0.4970.077, 0.497

88 1.0121.012 0.894, 1.1460.894, 1.146

99 0.6240.624 0.391, 0.9960.391, 0.996

1010 0.2460.246 0.144, 0.4220.144, 0.422

1111 0.7110.711 0.571, 0.8860.571, 0.886

1212 1.5631.563 0.373, 6.5480.373, 6.548

1313 0.9830.983 0.582, 1.6610.582, 1.661

BCG vaccine: used BCG vaccine: used to prevent to prevent tuberculosistuberculosis

Odds ratio Odds ratio estimates from 13 estimates from 13 trials (right)trials (right)

Scientists have Scientists have suggested that suggested that effects may be effects may be related to related to geographic latitudegeographic latitude

Funnel PlotFunnel Plot

Funnel Plot – Organized by Funnel Plot – Organized by LatitudeLatitude

Meta-regression: Meta-regression: example, contexample, cont

Log odds ratio versus absolute Log odds ratio versus absolute latitude:latitude:


Same plot, showing precision:Same plot, showing precision:


Same plot, with fitted Same plot, with fitted (meta-)regression line:(meta-)regression line:


Is the slope of the line significantly Is the slope of the line significantly different from 0? different from 0?

If yes, we conclude that the study If yes, we conclude that the study effects are in fact related to to effects are in fact related to to latitudelatitude

Meta-regression: detailsMeta-regression: details

In a regression model for the data: In a regression model for the data: each study represents one each study represents one observationobservation

Weights equal to the study precisionWeights equal to the study precision Random effects model Random effects model

(recommended)(recommended) Built-in function in Stata: ‘metareg’Built-in function in Stata: ‘metareg’

Meta-regression: Meta-regression: warningswarnings

Relationships found via meta-regression Relationships found via meta-regression do do not not necessarily suggest causality. Note necessarily suggest causality. Note that relationships may be confounded by that relationships may be confounded by other factors (so latitude may be other factors (so latitude may be confounded by other health factors).confounded by other health factors).

Avoid using summaries of patient Avoid using summaries of patient characteristics as the covariates of characteristics as the covariates of interest. E.g., note that a relationship interest. E.g., note that a relationship between study effects and between study effects and averageaverage age age does notdoes not necessarily suggest that there is a necessarily suggest that there is a relationship between age and study effect relationship between age and study effect within each study.within each study.

Meta-regression Example 2: Meta-regression Example 2:

Echo, Sensitivity & SpecificityEcho, Sensitivity & Specificity

0.0 0.2 0.4 0.6 0.8 1.0

Specificity

0.0 0.2 0.4 0.6 0.8 1.0

Sensitivity

Echo ROC CurveEcho ROC Curve

0.0

0.2

0.4

0.6

0.8

1.0

0.0 0.2 0.4 0.6 0.8 1.0

1-Specificity

Sensitivity

Summary LR+

Risk ratio.01 1 100

Study % Weight

Risk ratio (95% CI)

1.94 (1.34,2.82) Betesin 9.0

1.67 (0.92,3.03) Bjornstad 3.1

2.46 (1.41,4.27) Cohen 3.9

5.88 (2.78,12.44) Dagianti 1.7

12.98 (3.41,49.37) Galanti 0.8

4.34 (1.83,10.34) Jun 2.0

1.94 (1.15,3.28) Luotolahti 5.0

6.53 (2.89,14.76) Marangelli 2.3

5.64 (3.25,9.78) Marwick1 4.0

2.59 (1.79,3.75) Marwick2 9.8

4.47 (2.90,6.88) Marwick3 5.1

3.93 (2.16,7.14) Roger1 4.5

1.28 (1.06,1.53) Roger2 25.5

1.42 (0.91,2.23) Roger3 7.1

4.69 (3.09,7.12) Ryan 10.1

5.64 (1.59,20.03) Tawa 1.2

7.25 (2.85,18.46) Williams 1.6

7.02 (3.09,15.93) crouse 3.4

3.06 (2.72,3.45) Overall (95% CI)

Summary LR-

Risk ratio.01 1 100

Study % Weight

Risk ratio (95% CI)

0.06 (0.02,0.18) Betesin 3.5

0.16 (0.04,0.74) Bjornstad 0.9

0.10 (0.03,0.42) Cohen 2.4

0.11 (0.03,0.42) Dagianti 3.6

0.04 (0.01,0.28) Galanti 3.8

0.04 (0.01,0.31) Jun 2.7

0.06 (0.02,0.20) Luotolahti 1.9

0.08 (0.03,0.23) Marangelli 5.3

0.33 (0.22,0.49) Marwick1 10.6

0.08 (0.03,0.19) Marwick2 6.5

0.19 (0.10,0.36) Marwick3 8.5

0.48 (0.36,0.63) Roger1 9.8

0.46 (0.29,0.75) Roger2 5.0

0.64 (0.39,1.03) Roger3 3.2

0.13 (0.08,0.19) Ryan 16.5

0.07 (0.02,0.29) Tawa 2.3

0.19 (0.09,0.40) Williams 4.8

0.12 (0.07,0.18) crouse 8.8

0.21 (0.18,0.24) Overall (95% CI)

Heterogeneity Tests, Heterogeneity Tests, EchoEcho

Sensitivity: p = .43Sensitivity: p = .43 Specificity: p = .059Specificity: p = .059 + Likelihood Ratio: p = .018+ Likelihood Ratio: p = .018 - Likelihood Ratio: p = .008- Likelihood Ratio: p = .008 ROC curve: p < .0001 ROC curve: p < .0001 DOR: p < .0000001DOR: p < .0000001

. metareg lnor pmi, wsse(selnor) eform

Meta-regression Number of studies = 18

------------------------------------------------------------------------------ lnor | exp(b) Std. Err. t P>|t| [95% Conf. Interval]-------------+---------------------------------------------------------------- pmi | 1.0103 .0131902 0.78 0.444 .9827211 1.038652------------------------------------------------------------------------------

Prior MI & Diagnostic OR Univariate Analysis

. metareg lnor men, wsse(selnor) eform


------------------------------------------------------------------------------ lnor | exp(b) Std. Err. t P>|t| [95% Conf. Interval]-------------+---------------------------------------------------------------- men | 1.008248 .0087052 0.95 0.356 .9899616 1.026872------------------------------------------------------------------------------

Gender and Diagnostic OR Univariate Analysis

. metareg lnor age, wsse(selnor) eform


------------------------------------------------------------------------------ lnor | exp(b) Std. Err. t P>|t| [95% Conf. Interval]-------------+---------------------------------------------------------------- age | .827714 .0432287 -3.62 0.002 .7409641 .9246203------------------------------------------------------------------------------

Age and Diagnostic OR Univariate Analysis

Mean age = 59

For a year over the mean, the DOR increases by e .8277 , (approximately 2-fold increase for 1 year)

Critique of a Systematic Critique of a Systematic ReviewReview 1. Research Question1. Research Question

2. Protocol2. Protocol 3. Search3. Search 4. Study selection (inclusion/exclusion)4. Study selection (inclusion/exclusion) 5. Quality assessment5. Quality assessment 6. Data abstraction6. Data abstraction 7. Analysis7. Analysis

A) Create summary measureA) Create summary measure B) Assess for heterogeneityB) Assess for heterogeneity C) Assess for publication biasC) Assess for publication bias D) Conduct sensitivity/subgroup analysesD) Conduct sensitivity/subgroup analyses E) Advanced issues/techniquesE) Advanced issues/techniques

8. 8. InterpretationInterpretation

Reviewing Journal Reviewing Journal ArticlesArticles

Very little formal teachingVery little formal teaching ““Because reviews are often highly Because reviews are often highly

negative, the new researcher negative, the new researcher implicitly learns from the negative implicitly learns from the negative reviews received on his or her own reviews received on his or her own submitted papers that reviews are submitted papers that reviews are supposed to be negative. It is as if supposed to be negative. It is as if the implicit message is: A reviewer’s the implicit message is: A reviewer’s job is to criticize the manuscript.”job is to criticize the manuscript.”

12 Tips on Reviewing 12 Tips on Reviewing ArticlesArticles

1. Know your mission1. Know your mission 2. Be speedy2. Be speedy 3. Read carefully3. Read carefully 4. Say positive things in your review4. Say positive things in your review 5. Don’t exhibit hostility5. Don’t exhibit hostility 6. Keep it brief6. Keep it brief 7. Don’t nitpick7. Don’t nitpick 8. Develop your own style8. Develop your own style 9. Be careful in recommending further 9. Be careful in recommending further

experimentsexperiments 10. Watch for egocentrism10. Watch for egocentrism 11. Make a recommendation11. Make a recommendation 12. Sign your review12. Sign your review

http://www.psychologicalscience.org/observer/getArticle.cfm?id=2157

ConclusionsConclusions You can combine almost anythingYou can combine almost anything

Use clinical judgment to guide you in Use clinical judgment to guide you in deciding how and whether to combine deciding how and whether to combine studies.studies.

Remember the main mission of a systematic Remember the main mission of a systematic review: to summarize a body of literature in review: to summarize a body of literature in a concise and clear way.a concise and clear way.

Get statistical input as needed.Get statistical input as needed.

Documents

Systematic Review Lecture 3: Advanced Topics Stephen Bent, MD Assistant Professor of Medicine, UCSF Course Director, Systematic Reviews (EPI 214)