Cochrane Diagnostic test accuracy reviews Introduction to meta-analysis Jon Deeks and Yemisi Takwoingi Public Health, Epidemiology and Biostatistics University

Cochrane Diagnostic test accuracy reviews

Introduction to meta-analysis

Jon Deeks and Yemisi TakwoingiPublic Health, Epidemiology and BiostatisticsUniversity of Birmingham, UK

Outline Analysis of a single study Approach to data synthesis Investigating heterogeneity Test comparisons RevMan 5

Test accuracy

What proportion of those with the disease does the test detect? (sensitivity)

What proportion of those without the disease get negative test results? (specificity)

Requires 2×2 table of index test vs reference standard

2x2 Table – sensitivity and specificity

Disease (Reference test)

Present Absent

Indextest

+ TP FP TP+FP

- FN TN FN+TN

TP+FN FP+TN TP+FP+FN+TN

sensitivityTP / (TP+FN)

specificityTN / (TN+FP)

Heterogeneity in threshold within a study

diseasednon-diseased

specificity=100% sensitivity=100%

0 40 80 120 160

test measurement

diagnostic threshold


TN FN FP TP



0 40 80 120 160

test measurement



TN FN FP TP



0 40 80 120 160

test measurement



TN FNFP TP



0 40 80 120 160

test measurement



TN FN FP TP



0 40 80 120 160

test measurement



TN FN FP TP



0 40 80 120 160

test measurement


Threshold effect

0.0

0.2

0.4

0.6

0.8

1.0

sen

sitiv

ity

0.00.20.40.60.81.0

specificity

Increasing threshold decreases sensitivity but increases specificity

Decreasing threshold decreases specificity but increases sensitivity

Threshold Sensitivity Specificity

65 0.99 0.69

70 0.98 0.84

75 0.93 0.93

80 0.84 0.98

85 0.69 0.99

Ex.1 Distributions of measurements and ROC plotno difference, same spread


0 40 80 120

test measurement

0.0

0.2

0.4

0.6

0.8

1.0

sen

sitiv

ity

0.00.20.40.60.81.0

specificity

Uninformative test

Ex.2 Distributions of measurements and ROC plotsmall difference, same spread


0 40 80 120

test measurement

0.0

0.2

0.4

0.6

0.8

1.0

sen

sitiv

ity

0.00.20.40.60.81.0

specificity

line of symmetry

Diagnostic odds ratios

FNFP

TNTPORDiagnostic

veLR

veLR

yspecificityspecificit

ysensitivitysensitivit

DOR

1

1

Ratio of the odds of positivity in the diseased to the odds of positivity in the non-diseased

Diagnostic odds ratiosSensitivity

Specificity 50% 60% 70% 80% 90% 95% 99%

50% 1 2 2 4 9 19 99

60% 2 2 4 6 14 29 149

70% 2 4 5 9 21 44 231

80% 4 6 9 16 36 76 396

90% 9 14 21 36 81 171 891

95% 19 29 44 76 171 361 1881

99% 99 149 231 396 891 1881 9801

Symmetrical ROC curves and diagnostic odds ratios

As DOR increases, the ROC curve moves closer to its ideal position near the upper-left corner.

(1)

line of symmetry

uninformative test

(2)

(5)

(16)

(81)(361)

0.0

0.2

0.4

0.6

0.8

1.0

sens

itivi

ty

0.00.20.4 0.6 0.81.0

specificity

Asymmetrical ROC curve and diagnostic odds ratios


0 40 80 120

test measurement

0.0

0.2

0.4

0.6

0.8

1.0

sen

sitiv

ity

0.00.20.40.60.81.0

specificity

ROC curve is asymmetric when test accuracy varies with threshold

LOW DOR

HIGH DOR

Challenges There are two summary statistics for each study –

sensitivity and specificity – each have different implications

Heterogeneity is the norm – substantial variation in sensitivity and specificity are noted in most reviews

Threshold effects induce correlations between sensitivity and specificity and often seem to be present Thresholds can vary between studies The same threshold can imply different sensitivities

and specificities in different groups

Approach for meta-analysis

Current statistical methods use a single estimate of sensitivity and specificity for each study

Estimate the underlying ROC curve based on studies analysing different thresholds

Analyses at specified threshold Estimate summary sensitivity and

summary specificity

Compare ROC curves between tests Allows comparison unrestricted to

a particular threshold

0.0

0.2

0.4

0

.60

.81

.0

sens

itivi

ty

0.00.20.40.60.81.0

specificity

ROC curve transformation to linear plot Calculate the logits of TPR and FPR Plot their difference against their sum

Moses-Littenberg statistical modelling of ROC curves

Moses-Littenberg SROC method Regression models used to fit straight lines to model

relationship between test accuracy and test threshold

D = a + bS Outcome variable D is the difference in the logits Explanatory variable S is the sum of the logits Ordinary or weighted regression – weighted by sample

size or by inverse variance of the log of the DOR

What do the axes mean? Difference in logits is the log of the DOR Sum of the logits is a marker of diagnostic threshold

Producing summary ROC curves

Transform back to the ROC dimensions

where ‘a’ is the intercept, ‘b’ is the slope when the ROC curve is symmetrical, b=0 and the

equation is simpler

Example: MRI for suspected deep vein thrombosis

Study

Fraser 2003Fraser 2002Sica 2001Jensen 2001Catalano 1997Larcom 1996Laissy 1996Evans 1996Spritzer 1993Evans 1993Carpenter 1993Vukov 1991Pope 1991Erdman 1990

TP

204940

344

1516269

2749

27

FP

14431205233000

FN

03360601000103

TN

34453

188

1916

43265271586

Sensitivity

1.00 [0.83, 1.00]0.94 [0.84, 0.99]0.57 [0.18, 0.90]0.00 [0.00, 0.46]1.00 [0.90, 1.00]0.40 [0.12, 0.74]1.00 [0.78, 1.00]0.94 [0.71, 1.00]1.00 [0.87, 1.00]1.00 [0.66, 1.00]1.00 [0.87, 1.00]0.80 [0.28, 0.99]1.00 [0.66, 1.00]0.90 [0.73, 0.98]

Specificity

0.97 [0.85, 1.00]0.92 [0.80, 0.98]0.43 [0.10, 0.82]0.86 [0.64, 0.97]0.89 [0.52, 1.00]0.99 [0.96, 1.00]1.00 [0.54, 1.00]0.90 [0.77, 0.97]0.93 [0.76, 0.99]0.95 [0.85, 0.99]0.96 [0.89, 0.99]1.00 [0.48, 1.00]1.00 [0.63, 1.00]1.00 [0.54, 1.00]

Sensitivity

0 0.2 0.4 0.6 0.8 1

Specificity

0 0.2 0.4 0.6 0.8 1

Sampson et al. Eur Radiol (2007) 17: 175–181

Transformation linearizes relationship between accuracy and threshold so that linear regression can be used

0.0

0.2

0.4

0

.60

.81

.0

sens

itivi

ty

0.00.20.40.60.81.0

specificity

weighted

unweighted

-10

12

34

56

78

D-5 -4 -3 -2 -1 0 1 2 3

S

Linear transformation

SROC regression: MRI for suspected deep vein thrombosis

The SROC curve is produced by using the estimates of a and b to compute the expected sensitivity (tpr) across a range of values for 1-specificity (fpr)

weighted

unweighted

-10

12

34

56

78

D

-5 -4 -3 -2 -1 0 1 2 3

S

Inverse transformation



weighted

unweighted

-10

12

34

56

78

D

-5 -4 -3 -2 -1 0 1 2 3

S0

.00

.20

.4

0.6

0.8

1.0

sens

itivi

ty0.00.20.40.60.81.0

specificity


697.01

697.01

697.01721.4 11

1

1

697.0,721.4

FPRFPR

e

TPR

ba



weighted

unweighted

-10

12

34

56

78

D

-5 -4 -3 -2 -1 0 1 2 3

S0

.00

.20

.4

0.6

0.8

1.0

sens

itivi

ty0.00.20.40.60.81.0

specificity



Poor estimation Tends to underestimate test accuracy due to zero-cell

corrections and bias in weights

Problems with the Moses-Littenberg SROC method

Problems with the Moses-Littenberg SROC method: effect of zero-cell correction

0.0

0.2

0.4

0

.60

.81

.0

sen

sitiv

ity

0.00.20.40.60.81.0

specificity

0.0

0.2

0.4

0

.60

.81

.0

sen

sitiv

ity

0.00.20.40.60.81.0

specificity

Problems with the Moses-Littenberg SROC method: effect of zero-cell correction

Problems with the Moses-Littenberg SROC method

Poor estimation Tends to underestimate test accuracy due to zero-cell

corrections and bias in weights Validity of significance tests

Sampling variability in individual studies not properly taken into account

P-values and confidence intervals erroneous Operating points

knowing average sensitivity/specificity is important but cannot be obtained

Sensitivity for a given specificity can be estimated

Mixed models Hierarchical / multi-level

allows for both within (sampling error) and between study variability (through inclusion of

random effects) Logistic

correctly models sampling uncertainty in the true positive proportion and the false positive proportion

no zero cell adjustments needed Regression models

used to investigate sources of heterogeneity

33

Investigating heterogeneity

CT for acute appendicitis

0.0

0.2

0.4

0

.60

.81

.0

sens

itivi

ty

0.00.20.40.60.81.0

specificityTerasawa et al 2004

(12 studies)

Sources of Variation

Why do results differ between studies?

Sources of Variation

I. Chance variationII. Differences in (implicit) thresholdIII. BiasIV. Clinical subgroupsV. Unexplained variation

Sources of variation: ChanceChance variability:

total sample size=100

Sen

sitiv

ity

0.0

0.2

0.4

0.6

0.8

1.0

Specificity1.0 0.8 0.6 0.4 0.2 0.0

Chance variability:total sample size=40

Sen

sitiv

ity

0.0

0.2

0.4

0.6

0.8

1.0

Specificity1.0 0.8 0.6 0.4 0.2 0.0

May be investigated by:– sensitivity analyses – subgroup analyses or – including covariates in the modelling

Investigating heterogeneity in test accuracy

Example: Anti-CCP for rheumatoid arthritis by CCP generation (37 studies)

(Nishimura et al. 2007)

Anti-CCP for rheumatoid arthritis by CCP generation: SROC plot

0.0

0.2

0.4

0

.60

.81

.0

sens

itivi

ty

0.00.20.40.60.81.0

specificity

Generation 1 Generation 2

00.10.20.30.40.50.60.70.80.910

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Example: Triple test for Down syndrome (24 studies, 89,047 women)

Sen

sitiv

ity

Specificity

00.10.20.30.40.50.60.70.80.910

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Studies of the triple test ( = all ages; =aged 35 and over)

Sen

sitiv

ity

Specificity

Verification bias

Down's Normal

Test +ve (high risk) 50 250

Test -ve (low risk) 50 4750

100 5000

Down's Normal

50 250

50 4750

100 5000

AMNIO

AMNIOSensitivity = 50%

Specificity = 95%

Follow-up = 100%

Down's Normal

Test +ve (high risk) 50 250

Test -ve (low risk) 50 4750

100 5000

AMNIO

BIRTH

Down's Normal

50 250

34 4513

84 4763

Sensitivity = 60%

Specificity = 95%

Follow-up = 95%

16 lost (33%)

237 lost (5%)

Participants recruited Participants analysed

Participants recruited Participants analysed

00.10.20.30.40.50.60.70.80.910

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1S

ensi

tivity

Specificity


00.10.20.30.40.50.60.70.80.910

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

= all verified by amniocentesis

Sen

sitiv

ity

Specificity


Limitations of meta-regression

Validity of covariate information poor reporting on design features

Population characteristics information missing or crudely available

Lack of power small number of contrasting studies

Which test is best?

The same approach used to investigate heterogeneity can be used to compare the accuracy of alternative tests

Comparison between HRP-2 and pLDH based RDT Types: all studies

75 HRP-2 studies and 19 pLDH studies

Comparison between HRP-2 and pLDH based RDT Types: paired data only

10 comparative studies

Issues in test comparisons Some systematic reviews pool all available studies that have assessed

the performance of one or more of the tests. Can lead to bias due to confounding arising from heterogeneity

among studies in terms of design, study quality, setting, etc

Adjusting for potential confounders is often not feasible

Restricting analysis to studies that evaluated both tests in the same patients, or randomized patients to receive each test, removes the need to adjust for confounders.

Covariates can be examined to assess whether the relative performance of the tests varies systematically (effect modification)

For truly paired studies, the cross classification of tests results within disease groups is generally not reported

Summary Different approach due to bivariate correlated data

Moses & Littenberg method is a simple technique useful for exploratory analysis included directly in RevMan should not be used for inference

Mixed models are recommended Bivariate random effects model Hierarchical summary ROC (HSROC) model

RevMan DTA tutorial included in version 5.1

Handbook chapters and other resources available at:

http://srdta.cochrane.org

http://srdta.cochrane.org/

Documents

Cochrane Diagnostic test accuracy reviews Introduction to meta-analysis Jon Deeks and Yemisi Takwoingi Public Health, Epidemiology and Biostatistics University