Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
1
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
METAANALYTIC INTEGRATION OF DIAGNOSTIC ACCURACY
STUDIES IN STATA
Ben A. Dwamena, MDDepartment of Radiology (Nuclear Medicine)
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
OUTLINE
Evaluation of Diagnostic TestsRationale For Meta-analysisExisting Methodology and LimitationsEmerging alternativeImplementation in Stata using midas
2
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
CLINICAL EVALUATION OF A
DIAGNOSTIC TEST
Consider a population to be made up of two groups: those with a disease and those without it. A test aims to identify people as belonging to one of these two groups.
It is usually assumed that a gold standard is available which can perfectly distinguish groups but cannot be used in routine practice due to problems such as invasiveness and/or cost.
However alternative, more practical, tests are available which are imperfect.
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
CLINICAL EVALUATION OF A DIAGNOSTIC TEST
Many test outcomes are measured on an explicit continuous scale (e.g. level of a chemical in the blood).
Changing the threshold test level which defines a positive and a negative test will also change the performance characteristics of a test.
3
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
THESHOLD EFFECT AND ROC
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
VARIATION IN THRESHOLD VS ACCURACY
4
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
2 X 2 TABLE FOR SINGLE STUDY
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
WHY META-ANALYSIS?
5
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
WHY META-ANALYSIS?
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
STATISTICAL ISSUES
Summary Estimates
Threshold Variability
Unobserved Heterogeneity
Covariate effects
Publication Bias
Clinical Interpretation
6
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
TRADITIONAL MODELSummary ROC Analysis and Regression
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
EXAMPLE DATAPeak Flow Velocity To Detect Renal Artery Stenosis
7
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
DISPLAY OF DATA IN ROC SPACE
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
SROC REGRESSION
8
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
SROC REGRESSION
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
LIMITATIONS OF SROC MODEL
9
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
BIVARIATE MIXED-EFFECTS BINOMIAL REGRESSION MODEL
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
BIVARIATE MIXED-EFFECTS BINOMIAL REGRESSION MODEL
10
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
BIVARIATE MIXED-EFFECTS BINOMIAL REGRESSION MODEL
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
BIVARIATE MIXED-EFFECTS BINOMIAL REGRESSION MODEL
11
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
BIVARIATE MIXED-EFFECTS BINOMIAL REGRESSION MODEL
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
BIVARIATE MIXED-EFFECTS BINOMIAL REGRESSION MODEL
12
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
BINOMIAL REGRESSION MODELXTMELOGIT CODE
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
BINOMIAL REGRESSION MODEL
GLAMM CODE
13
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
EXAMPLE DATASET:Axillary PET Scan in Primary Breast Cancer
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
SYNTAX FOR midasmidas varlist [if exp] [in range] [,
id(varname) year(varname) modeling_optionsquality_assessment_optionsreporting_optionsexploratory_graphics_optionspublication_bias_optionsforest_plot_optionsheterogeneity_optionsroc_optionsprobability_revision_optionsgeneral_graphing_options *]
14
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
MODELING Options
estimator(g|x)provides a choice between estimation with xtmelogit in release 10 versus gllamm in version 9 or earlier.
nip()specifies the number of integration points used for maximum likelihood estimation based on adaptive gaussian quadrature.
Default is set at 15 for midas (default in xtmelogit is 7).
Higher values improve accuracy at the expense of execution times.
model will be estimated by Laplacian approximation using nip(1)
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
QUALITY ASSESSMENT Options
qualitab
creates, using optional varlist of study quality items (presence=1, other=0) a table showing frequency of methodologic quality items.
qualibar
creates, combined with optional varlist of study quality items (presence=1, other=0) calculates study-specific quality scores and plots a bargraph of methodologic quality.
Qlab
may be combined with qualitab or qualibar to use variable labels for table and bargraph of methodologic items.
15
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
QUALITY ASSESSMENTmidas tp fp fn tn prodesign ssize30 fulverif testdescr /// refdescr subjdescr report brdspect blinded, qualib
0 20 40 60 80 100percent
blinded
brdspect
fulverif
prodesign
refdescr
report
ssize30
subjdescr
testdescr
BAR GRAPH OF QUALITY ASSESSMENT
Yes No
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
EXPLORATORY GRAPHICS Options
qqplot(dss|dlor|dlr)plots a normal quantile plot to (a) check the normality assumption (b) investigate whether all studies come from a single population (c) search for publication bias
cumproduces a cumulative meta-analysis plot using year of publication as measure for temporal evolution of evidence.
inf investigates the influence of each individual study on the overall meta-analysis summary estimate.
16
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
EXPLORATORY GRAPHICS Options
bivboximplements a two-dimensional analogue of the boxplot for univariate data It is used to assess distributional properties of sensitivity versus specificity and for indentifying possible outliers.
chiplot creates a chiplot for judging whether or not the paired performance indices are independent by augmenting the scatterplot with an auxiliary display.
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
CHI-PLOTmidas tp fp fn tn, id(author) year(year) ms(0.75) chip
17
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
BIVARIATE BOXPLOTmidas tp fp fn tn, id(author) year(year) ms(0.75) bivbox
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
REPORTING Options
table(dss|dlr|dlor)Creates a table of study specific performance estimates
Reports measure-specific summary estimates
Provides results of homogeneity (chi_squared) and inconsistency(I_squared) tests.
dss, dlr or dlor represent the paired performance measures:sensitivity/specificity, positive/negative likelihood ratios and diagnostic
score/odds ratios.
18
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
SUMMARY RESULTSmidas tp fp fn tn, id(author) year(year) es(x) ms(0.75) res(all)
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
REPORTING Options
results(all)
provides summary statistics, group-specific between-study variances, likelihood ratio test statistics and other global homogeneity tests.
results(het)
provides group-specific between-study variances, likelihood ratio test statistics and other global homogeneity tests.
results(sum)
provides summary statistics for all performance indices
19
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
SUMMARY RESULTSmidas tp fp fn tn, id(author) year(year) ms(0.75) es(x) res(sum)
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
PUBLICATION BIAS Optionspubbias
performs linear regression of log odds ratios on inverse root of effective sample sizes as a test for funnel plot asymmetry in diagnostic metanalyses. A non-zero slope coefficient is suggestive of significant small study bias (pvalue < 0.10).
maxbiasperforms Copas' worst-case sensitivity analysis for publication bias. calculates the upper limit of no of missing studies that will overturn statistical significance estimates the minimun likely publication probability (Copas and Jackson, 2004).
funnel plots a funnel plot, a two-dimensional graph with sample size on one axis and effect-size estimate on the other axis. The funnel plot capitalizes on the well-known statistical principle that sampling error decreases as sample size increases.
20
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
FUNNEL PLOTmidas tp fp fn tn, ms(0.75) fun
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
FUNNEL PLOT ASYMMETRY TESTmidas tp fp fn tn, pubb
21
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
FOREST PLOT Options
forest(dss|dlr|dlor)
Creates summary graphs with study-specific(box) and overall(diamond) point estimates and confidence intervals for each performance index pair
Confidence intervals lines are allowed to extend between 0 and 1000 beyond which they are truncated and marked by a leading arrow.
fordata
adds study-specific performance estimates and 95% CIs to right y-axis.
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
FOREST PLOTmidas tp fp fn tn, id(author) year(year) es(x) ms(0.75) ford for(dlr)
DLR POSITIVE (95% CI)
27.28[9.81 - 75.84]
1.80 [0.81 - 3.99]7.56 [1.68 - 34.04]2.39 [1.03 - 5.56]7.60 [1.69 - 34.19]5.89 [2.62 - 13.23]1.91 [0.90 - 4.05]15.55 [1.04 - 232.22]83.52 [5.26 - 1000.00]26.42 [1.69 - 414.06]9.78 [2.09 - 45.72]8.33 [1.84 - 37.69]14.25 [3.72 - 54.57]5.02 [2.52 - 10.00]7.18 [2.24 - 23.03]31.57 [2.04 - 488.19]3.34 [1.64 - 6.80]3.44 [1.90 - 6.24]18.05 [6.90 - 47.22]9.35 [4.53 - 19.32]4.00 [1.48 - 10.84]4.00 [1.48 - 10.84]
STUDY(YEAR)
Q = 67.63, df = 19.00, p = 0.00I2 = 59.30 [59.30 - 84.51]
COMBINED
Tse 1992Adler1 1993
Hoh 1993Crowe 1994
Avril 1996Bassa 1996
Scheidhauer 1996Utech 1996Adler2 1997
Palmedo 1997Noh 1998
Smith 1998Rostom 1999Yutani1 1999Hubner 2000
Ohta 2000Yutani2 2000
Greco 2001Schirrmeister 2001
Yang 2001
0.8 1000.0DLR POSITIVE
22
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
FOREST PLOTmidas tp fp fn tn, id(author) year(year) es(x) ms(0.45) for(dlr)
STUDY(YEAR)
Q = 40.28, df = 19.00, p = 0.00I2 = 52.84 [28.82 - 76.85]
COMBINEDTse 1992
Adler1 1993Hoh 1993
Crowe 1994Avril 1996
Bassa 1996Scheidhauer 1996
Utech 1996Adler2 1997
Palmedo 1997Noh 1998
Smith 1998Rostom 1999Yutani1 1999Hubner 2000
Ohta 2000Yutani2 2000
Greco 2001Schirrmeister 2001
Yang 2001
0 3DLR NEGATIVE
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
HETEROGENEITY Optionsgalb(dss|dlr|dlor)
The standardized effect measure (e.g. for lnDOR, lnDOR/precision) is plotted (y-axis) against the inverse of the precision(x-axis). A regression line that goes through the origin is calculated, together with 95% boundaries (starting at +2 and -2 on the y-axis). Studies outside these 95% boundaries may be considered as outliers.
hetfor creates composite forest plot of all performance indices to provide a general view of variability.Confidence intervals lines are allowed to extend between 0 and 1000 beyond which they are truncated and marked by a leading arrow.
covarscombined with an optional varlist permits univariable metaregression analysis of one or multiple covariables.
23
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
UNIVARIABLE META-REGRESSION
midas tp fp fn tn prodesign brdspect blinded, covars
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
COMPOSITE FOREST PLOTmidas tp fp fn tn, id(author) year(year) ms(0.75) hetfor scheme(s2manual)
T se 1 9 9 2
A d le r1 1 9 9 3
H o h 1 9 93
C ro w e 1 9 9 4
A vr i l 1 9 9 6
B a s sa 1 9 9 6
ch ei d h a ue r 19 96
U t e ch 1 99 6
A d le r2 1 9 9 7
P a lm e d o 1 9 9 7
N o h 1 9 98
S m i th 1 9 9 8
R o s to m 1 9 9 9
Y u t an i1 1 9 9 9
H u b n er 2 00 0
O h t a 2 0 0 0
Y u t an i2 2 0 0 0
G re c o 2 0 01
ch ir r m ei s t er 2 00 1
Y a n g 20 0 1
D a n f o r t h 2 0 0 2
G u l l e r 2 0 02
K e le m e n 2 0 0 2
N a k am o t o 1 2 0 0 2
N a k am o t o 2 2 0 0 2
R i e be r 2 0 0 2
a n _ H o e ve n 2 0 0 2
B a r ra n g er 2 00 3
F eh r 20 04
I n ou e 2 0 0 4
L ov r i c s 2 0 0 4
W a h l 20 0 4
Z or n o za 2 0 0 4
W e i r 2 0 0 5
G i l -R e n d o 2 0 06
K u m a r 20 06
S t a dn ik 2 0 0 6
C h u n g 20 06
V e ro n es i 2 0 0 6
S t u d yI d
0 . 0 1 .0
S e n siti vi ty
0 . 3 1 .0
S p e cifi cit y
0 . 3 7 31 . 3
P o s it iv e L i ke lih o o d R a tio
0 . 0 2 .8
N e ga t iv e L ike l ih o od R a t io
0 . 2 2 39 8 2 . 8
D ia g n os t ic O dd s R a ti o
24
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
ROC Optionssroc1
plots observed datapoints, summary operating sensitivity and specificity in SROC space.
sroc2
adds confidence and prediction contours.
rocplane
plots observed data in receiver operating characteristic space (ROC Plane) for visual assessment of threshold effect.
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
SUMMARY ROCmidas tp fp fn tn, es(x) ms(0.75) plot sroc1
25
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
SUMMARY ROCmidas tp fp fn tn, es(x) ms(0.75) plot sroc2
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
ROC PLANEmidas tp fp fn tn, id(author) year(year) es(x) rocp
26
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
PROBABILITY REVISION Options
pddam(p|r)produces a line graph of post-test probalities versus prior probabilities between 0 and 1 using summary likelihood ratios
lrmatrixcreates a scatter plot of positive and negative likelihood ratios with combined summary point. Plot is divided into quadrants based on strength-of-evidence thresholds to determine informativeness of measured test.
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
CONDITIONAL PROBABILITY PLOTmidas tp fp fn tn, es(x) pddam(p)
27
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
UNCONDITIONAL PREDICTIVE VALUESmidas tp fp fn tn, ms(0.75) es(x) pddam(r)
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
LIKELIHOOD RATIO SCATTERGRAMmidas tp fp fn tn, es(x) lrmat scheme(lean1)
28
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
PROBABILITY REVISION Options
fagan creates a plot showing the relationship between the prior probability, the likelihood ratio(combination of sensitivity andspecificity), and posterior test probability.
prior() combined with fagan allows user to specify a pretest probability overriding the default of using disease prevalence calculated from data when fagan is invoked alone.
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
FAGAN’S NOMOGRAM Imidas tp fp fn tn, es(x) fagan scheme(lean1)
29
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
FAGAN’S NOMOGRAM IImidas tp fp fn tn, es(x) fagan prior(.20) scheme(lean1)
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
STANDING ON SHOULDERS OF GIANTS
30
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
THANK YOU FOR YOUR ATTENTION
QUESTIONS?
SUGGESTIONS
COMMENTS
Thursday October 25, 2007 FIRST WEST COAST STATA USERS' GROUP MEETING
COMING SOON
New and Improved MIDAS package with routines for meta-analysis and meta-regression of roc curve area and continuous test result data.