Embed Size (px)
Citation preview
Computer Assisted Detection
Does it work for you and your practice?
Etta D. Pisano, MD Beth Israel Deaconess Medical Center
Harvard Medical School
Disclosures
Current contracts with Philips and Fuji
Founder and Board Chair NextRay, Inc.
Senior Director of Research Development at the ACR
Take Home Message CAD helps some readers.
CAD can hurt some readers. Unsurprisingly, it depends how well you read
mammograms and how you use it.
BIG TAKE HOME.
USE CAD TO IMPROVE SENSITIVITY. DO NOT USE CAD TO IMPROVE SPECIFICITY!
Topics
History of CAD for screening mammography.
What are the data?
Why does it help some readers and hurt others?
How can you tell if it hurts or helps you and should you be using it in your practice?
CAD – Developed primarily as a way to reduce missed cancers
Photo From Cedars Sinai Website
Development Work
Lab studies showed promise. Work at U of Chicago by Doi, Giger, and
Nishikawa showed improved reader performance for calcification and mass detection. Nishikawa et al. Med Phys 1993: 20(6): 1661-
1666. Jiang et al. Acad Radiol. 1999: 6(1): 22-33.
Commercialization and Dissemination
First FDA approval was in 1998.
Congress mandated Medicare coverage in 2001.
83% of BCSC screening mammograms were interpreted with CAD in 2012.
Evidence that CAD Works
Reader studies showed improved ROC performance.
Readers read selected cases with and without CAD.
Reader Studies
Cases were a mixture of benign and malignant, calcs and masses, dense and fatty.
Readers varied in practice settings and mammography experience.
Every system on the market performed such studies to receive FDA approval.
One Example FDA trial for VuComp
2012
21 radiologists read 280 cases – 140 cancers and 140 noncancers
Cases read without CAD and then with CAD
BIRADS was recorded for each case with laterality of lesions recorded and matched for truth.
One Example FDA trial for VuComp
2012
Primary goal of the analysis –
To determine if radiologists were more effective at finding cancer when using VuComp software than when NOT using the software.
WAS THERE IMPROVED SENSITIVITY???
AUC difference 0.016 (0.004-.0.029), p=0.013
“The preponderance of evidence supports computer-aided detection
for screening mammography.”
Robin Birdwell, MD Radiology 253:1, 9-16.
October 2009.
Table 1. Results of Six Prospective Sequential Read Clinical Studies of CADe with Mammography
Birdwell RL. Radiology. 2009:253(1):9-16.
Results of Historical Control Clinical Studies of Screening Mammograms Read with CADe
Assistance
Birdwell RL. Radiology. 2009:253(1):9-16.
BCSC Methods Lehman et al. JAMA IM 2015; 175(11): 1828-1837
Compared accuracy of screening mammography with and without CAD between 2003-2009.
495,818 digital mams with CAD
129,818 digital mams without CAD
271 radiologists
66 facilities
3159 cancers
BSCS Results
Sensitivity UNCHANGED with CAD – WITH: 85.3% (83.6-86.9) WITHOUT: 87.3% (84.5-89.7)
BSCS Results
Sensitivity UNCHANGED with CAD – WITH: 85.3% (83.6-86.9) WITHOUT: 87.3% (84.5-89.7)
Specificity UNCHANGED with CAD- WITH: 91.6% (91.0-92.2) WITHOUT: 91.4% (90.6-92.0)
BCSC Results
Cancer Detection Rate UNCHANGED With CAD With: 4.1/1000 Without: 4.1/1000
BCSC Results
107 Radiologists who read some cases WITH CAD and some cases WITHOUT CAD showed significantly WORSE SENSITIVITY. With CAD: 83.3% (81.0-85.6) Without CAD: 90.7% (89.8-91.7)
Is it time to Stop Paying for Computer-Aided
Mammography? (Guess what he answered?)
Joshua Fenton. JAMAIM November 2015.
Why CAD helps some radiologists and hurts
others, as extrapolated from DMIST CAD data.
DMIST and CAD
CAD was NOT allowed in DMIST interpretations (2001-2004)
So, through a reader study we studied the application of CAD (iCAD(14 readers) and R2 (15 readers)) to DMIST digital mammograms and determined how CAD affected diagnostic accuracy. All readers read case mixes with 150 cancers and 150 benign/normals, with and without CAD
Cole and Pisano, et al. AJR 2014. 203(4): 909-916
iCAD results Average of all 14 readers
AUC Sens. Spec
Without CAD
0.71 (0.66-0.76)
0.49 (0.40-0.57)
0.89 (0.83-0.93)
With CAD
0.72 (0.67-0.77)
0.51 (0.43-0.60)
0.87 (0.81-0.92) NO statistically significant differences!
BUT MOST DID NOT!
72.4%
R2 results Average of all 15 readers
AUC Sens. Spec
Without CAD
0.71 (0.66-0.76)
0.51 (0.46-0.56)
0.87 (0.83-0.91)
With CAD
0.72 (0.67-0.77)
0.53 (0.48-0.58)
0.86 (0.82-0.90) NO statistically significant differences!
But SOME readers (27.6%) did improve with CAD…(*p<0.05)
Reader AUC w/o
Δ AUC
Sens w/o
Δ Sens
Spec w/o
Δ Spec
Δread Cancer
Δread NoCA
i2 0.75 0.02* 55% 2% 95% 0 3 1
i5 0.77 0.02* 67% 5%* 76% -1% 8 2
i7 0.59 0.03* 23% 0% 91% +1% 1 0
i11 0.70 0.02* 47% 2% 86% 0 3 0
R1 0.70 0.04* 41% 4%* 93% 0 6 0
R9 0.75 0.02* 59% 2% 90% -1% 2 1
R11 0.72 0.02* 50% 4%* 87% 0 6 0
R12 0.73 0.02* 53% 3%* 91% 0 4 0
And SOME readers improved their sensitivity (*p<0.5) without achieving AUC (17.2%)
Reader AUC w/o
Δ AUC
Sens w/o
Δ Sens
Spec w/o
Δ Spec
Δread Cancer
Δread NoCA
i4 0.70 0.002 43% 6%* 93% -4% 9 4
R10 0.75 0.01 55% 5%* 83% -6%* 8 10
i9 0.68 0.03 44% 5%* 95% -2% 8 3
i12 0.70 0.01 48% 4%* 87% -2% 8 2
R5 0.72 0.01 49% 4%* 87% 0% 6 1
SOME INCREASED FPs without a big improvement in
TPs (24.1%)
Reader AUC w/o
Δ AUC
Sens w/o
Δ Sens
Spec w/o
Δ Spec
Δread Cancer
Δread NoCA
i8 0.78 0.002 63% 3% 79% -2% 3 3
R8 0.73 0.01 55% 1% 83% -2% 2 3
i3 0.75 0.001 51% 1% 90% -2% 2 3
R7 0.74 -0.01 67% 0% 74% -2% 1 3
R4 0.64 -0.01 50% 0.2% 80% -3% 2 5
i14 R6
0.76 0.80
-0.003 -0.001
79% 57%
3% 3%
59% 92%
-7%* -2%
4 4
10 3
So, 20/29 (69.0%) readers improved AUC, Sensitivity or BOTH.
Or used CAD properly, but did not improve sensitivity enough compared with their
detriment in specificity.
What went wrong with the other 9 readers?
Some readers (8/9 or 27.6%) COMPLETELY OR MOSTLY IGNORED
THE CAD PROMPTS
Reader AUC w/o
Δ AUC
Sens w/o
Δ Sens
Spec w/o
Δ Spec
Δread Cancer
Δread NoCA
R14 0.66 -0.002 32% 0% 96% 0% 0 0
i13 0.76 -0.002 39% 0% 98% -2% 1 0
i6 0.67 0.004 41% 1% 84% 0% 2 0
R2 0.70 -0.005 35% 0% 93% 0% 1 1
R15 0.71 -0.002 53% 0% 81% -1% 0 2
i1 0.72 0.01 40% 2% 95% 0% 2 1
R3 0.68 0.002 58% 1% 73% -1% 1 2
i10 0.65 0.02 39% 0% 87% -2% 0 3
One reader used CAD to try to improve SPECIFICITY
Reader AUC w/o
Δ AUC
Sens w/o
Δ Sens
Spec w/o
Δ Spec
Δread Cancer
Δread NoCA
R13 0.69 0.004 49% -1% 88% +1% -1 -1
CAD Computer Assisted Detection
NOT DIAGNOSIS
R2 CAD case.
Dense breast with slight difference in density on one view.
3 readers noted it and rated it suspicious, without CAD.
After CAD circled the area, none of the 12 readers who had missed it changed their minds.
Path= Invasive Cancer.
iCAD calcs Case
8/14 readers found the calcs and called them suspicious WITHOUT CAD.
5 more changed their ratings to suspicious AFTER CAD.
Diagnosis = DCIS.
What can YOU do to make sure you are using CAD correctly?
Don’t change your readings to BIRADS 1 or 2 based on CAD marks. CAD is intended to increase sensitivity, not improve specificity.
Consciously pay attention to the CAD marks and re-evaluate every marked area.
When you DO change your mind and decide to recall a patient based on CAD marks, find out what happens at work-up to those patients. Work as diligently to improve your CAD performance as you do to improve your recommendations for biopsy.
What can the CAD companies do to make CAD more likely to
improve our performance?
Develop easy-to-use tools to allow us to track our performance with CAD!
