White Paper - Parascript · Giotto in 2003 (IMS Giotto 3DL) and another FFDM (IMS Giotto SD) was acquired by the hospital in 2008. IMS DL and IMS SD systems are each set up with Acquisition

White Paper

Takeaway■ AccuDetect Galileo performs significantly better than Arcadia

Lab’s Galileo in terms of sensitivity and false positive rates, with over 60% reduction in false positive rate per image.

■ Hospitals should pay close attention to performance details when selecting CAD systems, especially false positive rates.

SummaryComputer-Aided Detection (CAD) tools are software programs that analyze digitized or digital mammography images to find features that are associated with breast cancer. The following retrospective study compares the performance of Arcadia Lab’s Galileo and Parascript AccuDetect Galileo CAD systems. AccuDetet Galileo is an adaptation of Parascript’s CAD software called AccuDetect for IMS Giotto Full Field Digital Mammography (FFDM) system.

What’s in this Paper?

parascript.com6273 Monarch Park Place, Longmont, CO 80503 USAT: 303.381.3100 | toll free: 888.225.0169F: 303.381.3101 | [email protected]

Retrospective Study Comparison of Arcadia Lab’s Galileo and Parascript® AccuDetect® Galileo CAD systems in screening environment

Investigators ............................................................................................................. 2Background .............................................................................................................. 2Study Design ............................................................................................................ 3Results ...................................................................................................................... 4Conclusions .............................................................................................................. 8References ............................................................................................................... 8About University of Udine ...................................................................................... 10

2 parascript.com

InvestigatorsDr. Rossano Girometti (Researcher), Prof. Chiara Zuiani (Associate Professor), Dr. Anna Linda (Staff Radiologist), Dr. Viviana Londero (Staff Radiologist), Dr. Anna Dal Col (Staff Radiologist), Dr. Silvia De Stefani (Resident in Radiology), Dr. Luisa Battigelli (Resident in Radiology), Eleonora Di Gaetano (MD, Resident in Radiology), Prof. Massimo Bazzocchi (Chief of the Department).

All the Investigators work for the Istituto di Radiologia Diagnostica – Azienda Ospedaliero Universitaria “S. Maria della Misericordia” di Udine, Via Colugna n.50-33100 Udine.

University of Udine and their department of radiology conducted the first retrospective clinical study of Parascript® AccuDetect® Galileo in November and December 2010. This was a nonprofit, no sponsorship study conducted at the Radiology Department of University of Udine, Italy.

The Udine Hospital has a long experience with digital images processed by Full Field Digital Mammography. The first mammography equipment was purchased from IMS Giotto in 2003 (IMS Giotto 3DL) and another FFDM (IMS Giotto SD) was acquired by the hospital in 2008. IMS DL and IMS SD systems are each set up with Acquisition Work Station (AWS) and there are also 2 Review Work Stations so each image from both FFDMs can be viewed on both workstations. For processing (RAW) and for presentation images are stored in hospital’s picture archiving system. Udine Hospital is doing both screening and diagnostic breast imaging, therefore occurrence of malignant cancers is about 10-12 cancer cases per 1,000 women. 20-22 mammogram cases are completed each day.

BackgroundComputer-Aided Detection (CAD) tools are software programs that analyze digitized or digital mammography images to find features that are associated with breast cancer. The purpose of CAD is to “mark” suspected findings on mammographic images, in order to help Radiologist in the detection of lesions1. CAD is more suited to detect than to characterize/interpret breast lesions1. In other words, CAD marks potential malignant lesions that Radiologist could have missed.

CAD is an intensive matter for research in the scenario of mammographic screening programs. Therefore, screening is a cost-effective strategy for earlier detection, and thus treatment of breast cancer1-2. However, prevalence of disease is lower in the screening scenario (no symptomatic patients) as compared to the clinical setting (different patient categories, see below), leading to subtler and less obvious mammographic features, more difficult to identify1,3. The rate of false-negative cases at screening (missed cancers) is up to 25%, 27-70% of lesions being visible in retrospect according to different experiences2. In order to overcome the problem of false-negatives, and to improve sensitivity in the detection of breast cancer, two methods have been adopted: (a) conventional double reading strategies, involving two radiologists; (b) a single

6273 Monarch Park Place, Longmont, CO 80503 USAT: 303.381.3100 | toll free: 888.225.0169F: 303.381.3101 | [email protected]

Computer-Aided Detection

(CAD) tools are software

programs that analyze

digitized or digital

mammography images

to find features that are

associated with breast

cancer.


3 parascript.com


6273 Monarch Park Place, Longmont, CO 80503 USAT: 303.381.3100 | toll free: 888.225.0169F: 303.381.3101 | [email protected]

reading by radiologist followed by the use of CAD, at lesser human cost1,4-5. Despite the large use of CAD, especially in the USA screening programs, analysis of these methods is still incomplete because no randomized controlled trials have been performed to assess changes in survival6. Since these studies are difficult to achieve, most experiments use surrogate end-points such as: (a) CAD sensitivity and the proportion of retrieved cancers (retrospective studies); (b) cancer detection rate and recall rate due to further investigations in patients with suspect findings (prospective studies)7. Ideally, a CAD-induced increase in cancer detection should be balanced by a small, clinically acceptable increase in recall rate due to further investigations on false-positive cases8.

Overall, available results make CAD a controversial matter. Some authors state that CAD provides no significant increase in cancer detection rate in clinical practice, at the expense of excessive recall rates7,9. Most evidence, on the contrary, supports its use as a second reader in the screening scenario, because of an adequate intrinsic sensitivity for cancer, especially for microcalcifications, and increased readers’ sensitivity up to 23.7%10. In any case, the reduction of specificity invariably associated with the CAD use may result in no overall clinical benefit1. CAD is still less sensitive to masses, especially in dense breasts. Moreover, readers who benefit from CAD are mainly inexperienced ones1. Better CAD performances have been obtained by using full-field digital mammography, which provides digital images, compared to screen-film mammography11.

One emerging issue is9 the occurrence of many false positive marks created by CAD systems during the screening process. The purpose of this study was to compare two different CAD systems and determine if false positive rates differ significantly from system to system.

Study DesignTwo different CAD systems were used during the study. Both Computer Aided Detec-tion (CAD) systems were designed for a computerized second read of mammography images in order to help radiologists confirm that a suspicious area requires further investigation. AccuDetect® Galileo is an adaptation of Parascript CAD solution called AccuDetect for IMS Giotto platform.

Galileo is manufactured by Arcadia Lab S.r.L and was specifically developed for IMS Giotto, a Bologna, Italy based manufacturer of FFDM systems.

The study was performed in collaboration with the Department of Radiology, University of Udine and focused on standalone results for malignant calcification and mass detec-tion.

The cases for the study were collected from two IMS Giotto Full Field Digital Mammog-raphy (FFDM) units and consequently retrieved from Udine Hospital’s picture archiving system . There were 118 cancer cases (86 masses, 6 calcifications, 26 mixed) and 209 negative (normal) cases. The negative cases had a 12-month follow up radiology report to mark them as normal. The truth data was obtained by a resident, experienced

4 parascript.com6273 Monarch Park Place, Longmont, CO 80503 USAT: 303.381.3100 | toll free: 888.225.0169F: 303.381.3101 | [email protected]

radiologist who read each case and accompanying radiology and pathology reports. The radiologist drew contours of mass lesions and outlined an area of malignant micro calcifications on the screen of a color monitor. The senior (senior) radiologist then reviewed the contours and confirmed their validity.

All retrospective normal and cancer cases were sent to Galileo and AccuDetect Galileo software and the following results were calculated:

a) Percent of mass cancer cases correctly marked by Galileo, b) Percent of calcification cancer cases correctly marked by Galileo, c) Percent of mass cancer cases correctly marked by AccuDetect Galileo, d) Percent of calcification cancer cases correctly marked by AccuDetect Galileo, e) Percent of mass cancer images correctly marked by Galileo, f) Percent of calcification cancer images correctly marked by Galileo, g) Percent of mass cancer images correctly marked by AccuDetect Galileo,h) Percent of calcification cancer images correctly marked by AccuDetect Galileo.

Figure 1. Parascript AccuDetect® Galileo CAD-system

ResultsDuring the comparison as depicted in Table 1 (next page), AccuDetect Galileo achieved higher sensitivity per case for all malignant lesions (p<0.012).



AccuDetect Galileo

achieved higher sensitivity

per case for all malignant

lesions (p<0.012).

Table 1: True Positive (Sensitivity) versus False Positive rate per case and per image for AccuDetect Galileo and Galileo CAD.

Figure 1: FROC curve for AccuDetect Galileo (ADG) and performance data for Galileo at 0.8 False Positive per case.



AccuDetect Galileo

achieved higher sensitivity

per image for all malignant

lesions (p<0.0001).

AccuDetect Galileo achieved higher sensitivity per image for all malignant lesions (p<0.0001).

Table 2:Sensitivity and False Positive rates per image.

Figure 2: AccuDetect Galileo (ADG) FROC curve for Masses when compared to Galileo.AccuDetect Galileo achieved more than 60% lower false positive rate per image for



AccuDetect Galileo

achieved more than 60%

lower false positive rate per

image for malignant lesion

hypothesis.

malignant lesion hypothesis (p<0.001) when the operating point for AccuDetect Galileo is set to match the sensitivity of Galileo.

Table 3: False positive rate for all cancers (Total FP) and separately for masses and calcifications.

Figure 3: AccuDetect Galileo FROC curve for calcifications when compared to Galileo.

The paired Student’s T-test and Wilcoxon matched-pairs signed-rank test were used to assess the statistical significance of AccuDetect Galileo results’ improvement for malig-nant lesions detection.



AccuDetect Galileo has

significantly better overall

performance than Galileo

in terms of both sensitivity

and false positive rates.

The paired Student’s T-test used to assess the statistical significance of false-positives reduction by 60% achieved by AccuDetect Galileo. The results of the tests are summa-rized in the table below:

Table 4: The paired Student T-test and Wilcoxon test results.

ConclusionsDuring the study AccuDetect Galileo was compared against state of the art product – Galileo CAD, which had shown better performance than widely used iCAD’s Second Look Digital according to a 2007 study15. The comparison proved that:

AccuDetect Galileo has significantly better overall performance than Galileo in terms of both sensitivity and false positive rates.

AccuDetect Galileo achieved more than 60% lower false positive rate per image for malignant lesion hypothesis (p<0.001) when operating point for AccuDetect Galileo is set to match the sensitivity of Galileo.

The study proved that not all CAD products are alike and in this comparison study, Para-script’s AccuDetect Galileo achieved a significant, 60% reduction in false positive rate per image and better overall performance in sensitivity. Therefore, hospitals should pay a close attention to performance details when selecting CAD systems, especially if they care about a significant false mark reduction on reviewed mammograms.

References1.Bazzocchi M et al. (2007) CAD systems for mammography: a real opportunity? A review of the literature. Radiol Med 112:329-353

2.Vyborny CJ et al. (2000) Computer-aided detection and diagnosis of breast cancer. Radiol Clin North Am 38(4): 725-740

3.Malich A. et al (2006) CAD for mammography: the technique, results, current role and further developments. Eur Radiol 16:1449-1460

4.Chan HP et al. (1999) Improvement of radiologists’ characterization of mammograph-ic masses by using computer-aided diagnosis: an ROC study. Radiology 212:817-827



5.Thurfjell EL et al. (1994) Benefit of independent double reading in a population-based mammography screening program. Radiology 191: 241-244

6.Helvie M (2007) Improving mammographic interpretation: double reading and com-puter aided diagnosis. Radiol Clin North Am 45:801-811

7.Taylor P, Potts HWW (2008) Computer aids and human second reading as interven-tions in screening mammography: Two systematic reviews to compare effects on can-cer detection and recall rate. EJC 44;798-807

8.Gur D et al. (2004) Changes in breast cancer detection and mammography recall rates after the introduction of a computer-aided detection system. J Natl Cancer Inst 96:185-190

9.Fenton JJ et al. (2007) Influence of computer-aided detection on performance of screening mammography. N Engl J Med; 356:1399-409

10.Ciatto S et al. (2004) Comparison of two commercial systems for computer-assist-ed detection (CAD) as an aid to interpreting screening mammograms. Radiol Med 107:480-488

11.Skaane P (2009) Studies comparing screen-film mammography and full-field digital mammography in breast cancer screening: updated review. Acta Radiol 1:3-14

12.Brancato B et al. (2008) Does computer-aided detection (CAD) contribute to the performance of digital mammography in a self-referred population? Breast Cancer Res Treat 111:373-376

13.The JS et al. (2009) Detection of breast cancer with full-field digital mammography and computeraided detection 192:337-340

14. Balleyguier C et al. (2007) BIRADS classification in mammography. Eur J Radiol 61:192-194

15. University of Udine, Department of Radiology (2007) CAD and mammography: state of the art and clinical impact. Materials of XXXVI Annual Meeting of the Radiolo-gists of the Alpe - Adria Region. Verona, 11.17.2007


10

About ParascriptThe Parascript image analysis suite extracts meaningful information from images. Employing patented digital image analysis and pattern recognition technologies, the Parascript image analysis suite improves decision quality in medical imaging, postal and payment automation, fraud detection and forms processing operations. Parascript software processes billions of digital images per year. Fortune 500 companies, postal operators, major government and financial institutions rely on Parascript products. Organizations include the U.S. Postal Service, Bell + Howell, Fiserv, Elsag, Lockheed Martin, NCR, Siemens and Burroughs. Parascript is online at http://www.parascript.com.

parascript.com6273 Monarch Park Place, Longmont, CO 80503 USAT: 303.381.3100 | toll free: 888.225.0169F: 303.381.3101 | [email protected]

©2012 Parascript, LLC. All rights reserved.

About University of UdineThe University of Udine was founded in 1978 as part of the reconstruction plan of Friuli after the 1976 earthquake. Its aim was to provide the Friulian community with an inde-pendent center for advanced training in cultural and scientific studies. The University currently has 10 faculties in: Agriculture, Economics, Engineering, Law, Foreign Lan-guages, Education, Humanities, Medicine and Surgery, Veterinary Science and Mathe-matical, Physical and Natural Sciences. The University is actively involved in student and staff exchange projects with universities within the EU and is currently engaged in close collaboration with several universities from Eastern Europe and other non-EU countries. Moreover the University participates in many research projects at national and interna-tional level. The present number of students enrolled at the University is approx. 17000.


Documents

White Paper - Parascript · Giotto in 2003 (IMS Giotto 3DL) and another FFDM (IMS Giotto SD) was acquired by the hospital in 2008. IMS DL and IMS SD systems are each set up with Acquisition