Grid Computing for Breast Cancer CAD. A Pilot

Experience in a Medical Environment.

Raul Ramos Pollán1, Jose Miguel Franco1, Jorge Sevilla1, Naimy González de Posada2, Noel Pérez Pérez2, Mario Augusto Pires Vaz2, Joana Loureiro3, Isabel

Ramos3, Miguel Ángel Guevara López2

1 CETA-CIEMAT, Centro Extremeño de Tecnologías Avanzadas, Trujillo, Spain

{raul.ramos, josemiguel.franco, jorge.sevilla}@ciemat.es

2 INEGI, Faculty of Engineering, University of Porto, Portugal {nposada, nperez, mguevaral}@inegi.up.pt

gmavaz@fe.up.pt

3 Faculty of Medicine, University of Porto, Portugal joanaploureiro@gmail.com

radiologia.hsj@mail.telepac.pt

Abstract. This paper presents a novel Grid based software platform to store and manage large mammography digital image repositories (MDIR) including associated patient information (clinical history, biopsies, etc.), a mammography image workstation for analysis and diagnosis (MIWAD) and a data training and analysis framework (DTAF). MDIR simplifies and reduces the cost of hosting digitalized content and metadata stored on Grid infrastructures, exploiting its features such as strong security contexts, data federation, and large storage and computing capacities. MIWAD allows interaction with repository content and also offers low and high level image processing implementing full lifecycle CAD tasks: enhancing, segmentation, feature extraction, training, and the semiautomatic classification of digital mammograms. DTAF allows using Grid computing power to explore the search space of possible configurations of Artificial Neural Networks (ANN) based classifiers to find the ones that best classify mammography data. This was validated successfully (0.85 average area under the ROC curve) in a dataset of 100 selected mammograms with representative pathological lesions and normal cases from the MIAS database and the 699 cases of the UCI Breast Cancer Wisconsin dataset. Now, a pilot experience is taking place at the Faculty of Medicine in the Porto University, where MDIR and MIWAD technologies are being evaluated in a real medical environment.

Keywords: Grid computing, breast cancer CAD, mammography, medical image analysis, machine learning classifiers.

1 Introduction

Breast cancer is a major concern and the second-most common and leading cause of cancer deaths among women [1]. According to published statistics, breast cancer

has become a major health problem in both developed and developing countries over the past 50 years, and its incidence has increased recently. In Portugal, each year, are estimated (diagnosed) 4500 new cases of breast cancer and 1600 women deaths from this disease [2]. An efficient diagnosis of breast cancer in its early stages can give a woman a better chance of full recovery, reducing the associated morbidity and mortality rates. Screening mammography is the primary imaging modality for early detection of breast cancer because it is the only method of breast imaging that consistently has been found to decrease breast cancer-related mortality, detecting it one and a half to four years before a cancer becomes clinically evident [3]. Double reading of mammograms (two radiologists read the same mammograms) [4] has been advocated to reduce the proportion of missed cancers. But the workload and cost associated with double reading are high. Instead with the introduction of Computer-Aided Diagnosis/Detection (CAD) systems only one radiologist is needed to read each mammogram rather than two. CAD systems, which use computer technologies to detect abnormalities in mammograms such as calcifications and masses, and the use of these results by radiologists for diagnosis [5], can play a key role in the early detection of breast cancer and help to reduce the death rate among women with breast cancer. For research scientists, there are several interesting research topics in cancer detection and diagnosis systems, such as high-efficiency, high-accuracy lesion detection algorithms, etc. Radiologists, on the other hand, are paying attention to the effectiveness of clinical applications of CAD systems.

This paper presents a novel Grid based software platform to store and manage large mammography digital image repositories (MDIR), a mammography image workstation for analysis and diagnosis (MIWAD) and a data training and analysis framework (DTAF). The MDIR is based on the Digital Repositories Infrastructure (DRI) developed at CETA-CIEMAT. DRI is a software platform aimed at simplifying and reducing the cost of hosting digital repositories over Grid infrastructures. DRI offers a high level API (Application Programming Interface) and user interfaces for applications and users to interact in a simple manner with hosted repositories. The MIWAD uses the DRI API to interact with the repository of mammograms, allowing low and high level image processing full CAD lifecycle tasks. DTAF allows using Grid computing power to explore the search space of possible configurations of Artificial Neural Networks (ANN) based classifiers to find the ones that best classify mammography data. This was validated successfully (0.85 average area under the ROC curve) in a dataset of 100 selected mammograms with representative pathological lesions and normal from the MIAS database [6]. Now a pilot experience is taking place at the Faculty of Medicine – Hospital São João in the Porto University (FMUP), where the developed Grid infrastructure (MDIR and MIWAD) are being evaluated in a real medical environment.

This paper is structured as follows. Section 2 describes the repository of mammography images. Section 3 shows the developed CAD system, based on MIWAD and classifiers found by DTAF. Section 4 presents and discusses the achieved results, and section 5 outlines the paper conclusions.

2 The Mammography Digital Image Repository (MDIR)

2.1 Digital Repositories Infrastructure (DRI) Overview

The MDIR presented here is hosted on the DRI platform developed at CETA-CIEMAT. Preliminary work using DRI to host a medical image repository can be found in [7]. Figure 1 shows the architecture of DRI and a sample deployment scenario. A repository is defined by a repository provider in a description file, which is an XML representation of the data model of the repository. Fields defined in this file (such as patient name, age or images) are marked up as either metadata or as large digital content. The DRI engine parses the repository description file and creates the appropriate storage structures so that (1) metadata is stored on a regular local database and (2) large digital content is stored on a Grid infrastructure. DRI also provides web and stand-alone applications for browsing the repository and managing its content (see Fig. 1). Repository providers also define navigation trees for users to browse the repository. In summary a repository provider needs only to define a repository description file and he can start using the underlying Grid storage and database to host his repository offering graphical tools to interact with the content.

The default DRI implementation uses gLite Grid middleware for authentication, storage of large digital content and computing power. A Grid infrastructure is typically made of a federation of sites (such as different data centers, hospitals, etc.), each one providing different amounts of Grid storage and computing power to the federation. Figure 1 shows a sample deployment scenario of several DRI instances over a shared Grid infrastructure. Note the following:

(1) Three sites participate in the federation: an hospital (deploying one DRI instance, a local database for metadata and a Grid site for storage), a small clinic (deploying one DRI instance and a local database) and one data centre (offering its Grid storage to the federation).

(2) When hospital users enter new repository items (patients and/or new mammograms), their DRI instance will store the metadata (such as patient information and/or diagnoses) on its local database, while storing the digitalized mammograms in the Grid storage space. Note that Grid policies established within the federation will determine where the digitalized mammogram will be physically stored, not necessarily on its local Grid storage.

(3) When clinic users enter new repository items, their DRI instance will store the metadata in their local database and mammograms in the Grid storage space according to the established Grid policies. Since they provide no Grid storage (maybe because they could not afford it) their mammograms will always be stored somewhere else.

(4) There might be many different Grid storage policies. For instance, only files

larger than 50Mb will be stored in the Data Centre Grid site. Or, the Hospital and the

Data Centre Grid Sites will maintain replicas of files of the last three months. Or, data generated in the Clinic will only be stored in the Data Centre Grid site, etc.

(5) Both local databases are isolated. Users logging into the hospital DRI instance will only access patient data held in its local database (although mammograms might be stored somewhere else), and analogously for clinic users. The

metadata spaces are not shared and therefore, no patient information is visible outside the institution scope.

(6) However, the clinic and the hospital might agree to share some data. For instance, since the clinic has few expert physicians for diagnoses, it might decide to synchronize regularly patient data from their local database to the hospital database, so that hospital expert physicians can occasionally diagnose patients. However, the clinic might decide not to expose patient sensitive data (name, etc.), but only anonymous records (patient studies). In any case, since the mammograms are stored in the federated Grid storage, hospital experts will be able to see and annotate them once they access to the patient data.

2.2 MDIR Data Model

The MDIR data model here presented (Fig. 2) is a subset of the DICOM medical file format [8] (customized by radiologists of the FMUP) for storing and managing specific patient information related to digital mammography images. This work complements recent results in managing DICOM objects within Grid environments (see the TRENCADIS middleware [9]) by applying the DICOM standard at FMUP and integrating it with the full CAD development lifecycle. In short, at a first level, MDIR is composed of many patients, each one might undergo one or more studies, each study is made of one or more series, each series contains one or more images and each image contains one or more annotations. An annotation corresponds to a mark made on the mammogram image, such as a circle, a text note or a segmentation of a certain region. In turn to support manual or automated classification, each annotation may have one or more feature sets associated. A feature represents a certain characteristic of the annotation, such as area, shape, elongation, etc. Each feature set is then assigned by an expert radiologist or an automatic classifier (for instance an artificial neural network, ANN) to a certain class belonging to a class family such as the BI-RADS [10] family of classes.

The model supports the same feature set (and thus the same annotation) to be given several classifications by different clinicians and automatic classifiers under different

DATA CENTRE

SMALL CLINIC HOSPITAL

DRI

GRID STORAGEGRID STORAGE

CUSTOM

exchange of

patients public data

between dbsWeb based

apps and

viewers

CAD Workstation

Web based apps,

Viewers

clinicians

other hospital users

clinic users

LOCAL DB

DRI

LOCAL DB

Fig. 1. DRI architecture in a sample deployment scenario

class families. This supports storage of a variety of sets of experiments of classification runs performed both by human experts and automatic classifiers, so that later become available for statistical analysis. Figure 2 and 3 below show the defined MDIR data model and their custom-made Java interface implementation being used and validate at FMUP.

3 Computer-Aided Detection/Diagnosis (CAD)

The term CAD gathers a wide range of algorithms, methods and technologies used in Computer-Aided Detection/Diagnosis under different areas of biomedical research and, particularly, within medical image analysis. Computer-Aided Detection (or

Fig. 3. MDIR custom-made java interface

Fig. 2. MDIR Data Model

Patient Study Series Image

ImageType

Annotation

GenericSeriesLayout

one to many

many to one

many to many

FeatureSetDefinition FeatureSetValues

ClassFamily ClassMember Classification

Classifier

ImageProcessorSet

ImageProcessor

SeriesLayout

ImageProcessorSetValues

CADe) deals with the use of computer systems to indicate the location of suspicious regions in a medical image. Computer-Aided Diagnosis (or CADx) aims at using computers to help radiologists in characterizing a region or a lesion, previously located by a human or a computer, so that they can take the appropriate diagnosis and patient therapy decisions, based on a suitable combination of CADe and Machine learning methods [11].

Machine learning approaches had been reported in the past few years for mammography images analysis and classification with different degrees of success [12]. Our work is focused in the usage of Artificial Neural Networks (ANN) based classifiers. There are different kinds of ANNs and, in addition to the choice of network structure (layers, neurons and activation functions) each one can be tuned by a number of parameters. An ANN configuration is a certain combination of ANN type, network structure and parameters and, in general, ANN design amounts to choosing well performing ANN configurations within the search space conformed by all possible configurations. This remains mostly a heuristic task, largely dependent on the experience of the ANN designer, the quality of the training sets used, the nature of the classification task in hand, etc.

Training a single ANN configuration is computationally expensive, being the required computing resources roughly dependant of the complexity and sizes of the network and the training set used. In turn, the exploration of the search space of possible ANN configurations for a given training set is far more computationally demanding. As part of this work, the Data Training and Analysis Framework (DTAF) was developed to allow using computer power harnessed by Grid infrastructure to explore regions of such search space with the aim of finding good ANN configurations and gaining better understanding of what kind of ANN configurations are better suited for a given problem.

3.1. Mammography Image Workstation for Analysis and Diagnosis

The process by which a mammography image is diagnosed with the help of the

CAD system goes through, but not limited to the following steps:

1. Region of Interest (ROI) selection: the specific image region where the lesion or abnormality is suspected to be (which can be manual, semiautomatic or automatically selected).

2. Image Preprocessing: the ROI pixels are enhanced so that, in general, noise is reduced and image details are enhanced.

3. Segmentation: the suspected lesion or abnormality is marked in any way and separated from the rest of the ROI by identifying its contour or a pixels region. Segmentation can be fully automatic (the CAD system determines the segmented region) or semi-automatic, where the user segments the region assisted by the computer through some interactive technique such as deformable models (snakes, active shape model, etc.) [13] or intelligent scissors (livewire) [14].

4. Feature Extraction and Selection: a vector of quantitative measures (features) of different nature is extracted out from the segmented region. These features typically describe the shape, the texture or the pixel values

and are used as input to the classifier in the following step. In [15] it was shown that, in certain cases, as little as four features may be sufficient to obtain correct classifications.

5. Classification: this last step is the one that finally offers a diagnostic to be used as a second opinion, by assigning the vector of extracted features to a certain class, corresponding to a lesion type and/or a benignancy/malignancy status [15].

The above process is supported by the MIWAD (see figure 4), which implement the TUDOR DICOM image viewer facilities [16] and it can be used in two modes:

1. Training set construction mode. The user performs steps 1 through 4,

and then he provides a classification based on his own knowledge and expertise. With this, we build datasets composed of extracted feature vectors that have been classified and use them to train classifiers.

2. CADx mode: The user performs steps 1 through 4, and then he uses a previously trained classifier to obtain a classification of the extracted feature vector and use it as second opinion in his diagnosis and patient management decisions.

3.2 Data Training and Analysis Framework

The Data Training and Analysis Framework (DTAF) is a Java based framework

that enables the systematic usage of Grid based computing resources to massively

1. ROI SELECTION

3. SEGMENTATION

4. FEATURE EXTRACTION

5. CLASSIFICATION

2. IMAGE ENHANCEMENT

image processingalgorithms

(pixel/bit based)

semiautomatic 9 features Initially

ANN Models – basedclassifiers

GUI based

1-2

3

4

5

Fig. 4. Mammography Image Workstation for Analysis and Diagnosis (MIWAD)

explores search spaces of ANN configurations. With this, one can define an “exploration” made of certain ANN configurations, distribute them into jobs and send them to a gLite infrastructure for training with specific training sets. DTAF is best aimed to produce binary classifiers (assigning each feature vectors one of two possible classes, such as benign-malign, or “microcalcification – no microcalcification”, although it also provides the capability of building multiclass ensemble classifiers [17] out of binary classifiers. Its basic elements are:

- Dataset or Training set: A dataset is made of a set of elements (or vectors), each one containing a set of features (used as input values for classifiers) and, optionally, an ideal class to which it belongs for supervised training (expected classifier output). Elements of a dataset are usually split into a training subset (to train classifiers) and a test subset (to measure their generalization capabilities when applied to unseen data).

- Binary Dataset: a dataset whose elements belong to only two classes, as opposed to a Multiclass Dataset, whose elements may belong to several (more than two) classes. DTAF provides you the tools to create multiple binary training sets for a given multi-class training set.

- Classifier: a machine learning algorithm able to assign one input vector to one class out of a set of available classes. A Binary Classifier is a classifier that assigns an input vector to one out of two possible classes. Binary classifiers use binary datasets for training.

- Engine: Engines encapsulate third party classifiers. Each engine defines a set of restrictions on the ANNs it can train (for example, the Radial Basis engine can only train ANNs with one hidden layer and a set of training parameters such as learning-rate and momentum).

- Engine configuration: An engine configuration specifies the parameters with which a particular engine is used to train a dataset with. For instance, a configuration of the FFBP engine might specify 3 layers with 10 neurons each, with 0.1 as learning rate and 0.5 as momentum over 10000 epochs.

- Exploration: an exploration over a dataset defines a set of engine configurations to train in batch mode in order to find those that best classify the training set or to later use them to build ensemble classifiers.

- Jobs: each exploration is split into number jobs. Each job will train a subset of the engine configurations defined in a given exploration. Jobs can be then executed in different manners: sequentially over the same computer or in parallel over a gLite infrastructure.

Currently, DTAF supports ANN engines from the Encog [18] and Weka toolkits [19]. The following table lists the currently supported ANN engines:

Table 1. Currently supported ANN engines in DTAF. Engine name Description encog.ffbp Feedforward with backpropagation training encog.ffga Feedforward with genetic algorithms based training encog.ffsa Feedforward with simulated annealing based training encog.ffsaroc FFSA with WEKA ROC based error evaluation encog.rb Radial basis encog.som Self-organizing map (unsupervised)

In addition, DTAF allows evaluation of binary classifiers through ROC (Receiver

Operating Characteristic) curves [20-21], whose area under the curve (Az) measures the capability of a classifier to distinguish correctly between the two classes of a binary training set. ROC curves are commonly used in biomedical research and provide a way to compare the performance of different binary classifiers. We use two methods to measure ROC curves: the bi-normal distribution as provided by JLABROC4 [22] and the method as provided by WEKA [19] which uses on the Mann-Whitney statistic. In both cases, DTAF generates both the ROC curve plots and calculates the area under the curve. Explorations are defined in exploration files, which are then read by DTAF when instructed to execute the exploration over a gLite infrastructure. The following shows an example exploration file.

Table 2. Sample exploration file explore.neurons.input = 9 explore.neurons.output = 2 explore.neurons.layer.01 = 18:36 explore.neurons.layer.02 = 9:18 explore.neurons.layer.03 = 5:9 explore.activation.input = tanh explore.activation.output = tanh:sigm explore.activation.layer.01 = tanh:gaus explore.activation.layer.02 = tanh explore.activation.layer.03 = tanh explore.nblayers.fixed = yes explore.trainingsets = BCW explore.stop.error = 0.1 explore.stop.epochs = 200 explore.trainengines = rb:ffrp:ffga:ffsaroc:ffsa explore.validation = asints

explore.encog.ffga.matepercent = 0.5 explore.encog.ffga.percentmate = 0.2 explore.encog.ffga.population = 100 explore.encog.ffsa.starttemp = 10:15 explore.encog.ffsa.endtemp = 2:5 explore.encog.ffsa.cycles = 100 explore.encog.ffsaroc.starttemp = 10 explore.encog.ffsaroc.endtemp = 2 explore.encog.ffsaroc.cycles = 100 explore.encog.rb.gaus.center = 0.5:0.7 explore.encog.rb.gaus.peak = 0.5 explore.encog.rb.gaus.width = 0.5 explore.encog.rb.gaus.spread = 0.5:1.0 explore.encog.ffbp.learnrate = 0.2:0.5 explore.encog.ffbp.momentum = 0.0 explore.numberofjobs = 50

4 Discussion and Results

We created a testing MDIR formed by 100 selected mammograms with representative pathological lesions and normal from the MIAS database [6], which are classified into seven classes: six types of lesions and a normal type (corresponding to the absence of lesion). It is therefore a multiclass data set that classifies data into the following classes: NORM (normal), CIRC (well circumscribed masses), SPIC (spiculated masses), ILLDEF (ill defined masses), MCALC (microcalcifications), ASIM (asymmetries) and ARCHD (architectural distortions). Then, using MIWAD features vectors were extracted (one for each mammogram) that combine shape and texture parameters. Based on these features vectors an experimental dataset was created with seven binary training sets (one for each class), distinguishing each class from the rest, without distinguishing further whether the lesion was benign or malign. We therefore had seven classification problems.

DTAF was validated with the binary MIAS datasets as just described and the UCI breast cancer Winconsin (BCW) dataset [23]. The BCW dataset is a binary dataset that assesses diagnoses from feature vectors extracting characteristics from breast cells, and not from breast images. Each vector contains nine features and is classified benign or malign. It contains 699 features vectors. Each binary training set (the seven derived from MIAS and the BCW one) was split into 50% test data and 50% train

data, and an exploration similar to the example exploration file shown above was defined for each of them. Exploration jobs were launched over the CETA-CIEMAT gLite production infrastructure (CETA-PROD). A total of 815 ANN configurations were trained using about 12 hours of CPU time over 10 cores. The following table summarizes the results obtained for each training set.

Table 3. Exploration results

Training set Best test Az Best test score Number of

configurations Computing

timeBCW 0.906 encof.rb 0.956 encof.ffrb 66 5.16 hrsMIAS-ASIM 0.871 encof.ffga 0.897 encog.ffsa 101 63.18 minMIAS-CIRC 0.859 encog.ffsa 0.897 encog.ffsa 104 54.14 minMIAS-ILLDEF 0.745 encog.ffrp 0.875 encog.ffsa 102 61.09 minMIAS-MCALC 1.000 encog.ffsaroc 0.923 encog.ffsaroc 104 46.67 minMIAS-ARCHD 0.824 encog.ffsa 0.875 encog.ffsa 101 63.16 minMIAS-SPIC 0.843 encof.ffga 0.878 encog.ffsa 132 62.21 minMIAS-NORM 0.766 encof.ffga 0.825 encof.ffsaroc 105 63.65 min

Also, for illustration purposes you can see here the ROC curves for the best BCW,

MIAS-ASIM and MIAS-MCALC binary classifiers. In the ROC curves you can also see the specific ANN configuration used to obtain

such result. Note that the ROC Az value shown in the figures is not exactly the same as the one shown in the table, since the plot uses the JLABROC value whereas the table shows the WEKA ROC value. Also, note that the test score shown in small in the figures is not necessarily the same as the one in the tables as best test Az and best test score is usually not obtained with the same ANN configuration.

As it can be seen the technologies described in this work (MDIR, MIWAD and DTAF) use Grid infrastructures to fully support the medical workflow to build and use mammography CAD systems, including data acquisition and archiving, image processing, construction of data sets and training classifiers. We can effectively achieve federation of resources (medical imaging) and well performing classifiers, even in the small scale shown in the results above.

What is most important, this work has been developed in close collaboration with radiologists, assessing its most important aspects, notably, the integration of the developed technologies within the IT systems of medical institutions, the alignment of the tools with the workflow of the specialists, minimizing its impact in its daily work and increasing the value of the obtained results.

5 Conclusions

This work showed three technologies developed under the ongoing collaboration among INEGI, FMUP and CETA-CIEMAT enabling effective exploitation of Grid resources in the area of medical imaging: (1) a mammography repository (MDIR) hosted on Grid storage by using the DRI platform, (2) an analysis workstation (MIWAD) that enables the full CAD lifecycle over content stored in the mammography repository and (3) a framework that exploits Grid computer power to explore configurations of machine learning classifiers that can be used as assistance for lesions diagnosis. Usage of these technologies was validated in a medical environment by building a pilot CAD system with satisfactory performance and usability results.

Future immediate work is focused on tuning the platforms to integrate them completely within medical work flows, enabling systematic construction of federated repositories of mammograms, building training sets and custom made classifiers and integrating them within easy to use workstations. This work, present and future, is always done in close collaboration with end-users and professionals from medical environments to ensure its acceptability and validation.

Acknowledgments

This work is part of the GRIDMED research collaboration project between INEGI (Portugal) and CETA-CIEMAT (Spain). Prof. Guevara acknowledges POPH - QREN-Tipologia 4.2 – Promotion of scientific employment funded by the ESF and MCTES, Portugal. CETA-CIEMAT acknowledges the support of the European Regional Development Fund

6 References

1. Althuis, M.D., et al., Global trends in breast cancer incidence and mortality

1973-1997. Int. J. Epidemiol., 2005. 34(2): p. 405-412. 2. Veloso, V., Cancro da mama mata 5 mulheres por dia em Portugal, in

CiênciaHoje. 2009: Lisboa, Portugal. 3. Tabár, L., et al., Beyond randomized controlled trials: organized

mammographic screening substantially reduces breast carcinoma mortality. Cancer, 2001. 91(9): p. 1724-1731.

4. Brown, J., S. Bryan, and R. Warren, Mammography screening: an

incremental cost effectiveness analysis of double versus single reading of

mammograms. BMJ (Clinical research ed.), 1996. 312(7034): p. 809-812. 5. Sampat, M.P., M.K. Markey, and A.C. Bovik, Computer-Aided Detection

and Diagnosis in Mammography, in Handbook of Image and Video

Processing (Second Edition), B. Al, Editor. 2005, Academic Press: Burlington. p. 1195-1217.

6. J. Suckling, et al., "The Mammographic Image Analysis Society Digital

Mammogram Database". . 1994, Exerpta Medica. International Congress Series p. 375-378.

7. Ramos, R., et al. Exploiting eInfrastructures for medical image storage and

analysis: A Grid application for mammography CAD. in The Seventh

IASTED International Conference on Biomedical Engineering. 2010. Innsbruck, Austria: ACTA Press.

8. NEMA. The DICOM Standard. 2008. 9. Blanquer Espert, I., et al., Content-based organisation of virtual repositories

of DICOM objects. Future Generation Computer Systems, 2009. 25(6): p. 627-637.

10. D'Orsi, C.J., et al., Breast Imaging Reporting and Data System: ACR BI-

RADS-Mammography. 4th Edition ed. 2003: American College of Radiology.

11. Giger, M., et al. PROGRESS IN BREAST CADx. in Biomedical Imaging:

From Nano to Macro, 2007. ISBI 2007. 4th IEEE International Symposium

on. 2007. 12. Jinshan, T., et al., Computer-Aided Detection and Diagnosis of Breast

Cancer With Mammography: Recent Advances. Information Technology in Biomedicine, IEEE Transactions on, 2009. 13(2): p. 236-251.

13. Xu, C. and J.L. Prince, Snakes, Shapes, and Gradient Vector Flow. IEEE TRANSACTIONS ON IMAGE PROCESSING, 1998. 7(3): p. 359-369.

14. Jianming, L., M. Tim, and T. Demetri, United Snakes. Medical image analysis, 2006. 10(2): p. 215-233.

15. López, Y., Novoa, Andra., Guevara, Miguel., Quintana, Nicolás., Silva, Augusto, Computer Aided Diagnosis System to Detect Breast Cancer

Pathological Lesions, in Progress in Pattern Recognition, Image Analysis

and Applications. 2008, Springer Berlin / Heidelberg. p. 453-460. 16. The TUDOR DICOM tools: Henri Tudor Research Center

[http://santec.tudor.lu/project/optimage/dicom/start]. 17. Lorena, A.C., A.C.P.L.F. de Carvalho, and J.M.P. Gama, A review on the

combination of binary classifiers in multiclass problems. Artificial Intelligence Review, 2008. 30(1-4): p. 19-37.

18. Heaton, J., Programming Neural Networks with Encog 2 in Java. 2010, Heaton Research, Inc.

19. Mark Hall, et al., The WEKA Data Mining Software: An Update. SIGKDD Explorations, 2009. 11(1).

20. Fawcett, T., An introduction to ROC analysis. Pattern Recognition Letters, 2006. 27(8): p. 861-874.

21. Yoon, H.J., et al., Evaluating computer-aided detection algorithms. Medical physics, 2007. 34(6): p. 2024-2038.

22. John Eng, M.D. ROC analysis: web-based calculator for ROC curves. 2006 May 17 [cited 2010 March 7]; Available from: http://www.jrocfit.org.

23. Asuncion, A. and D.J. Newman, UCI Machine Learning Repository

[http://www.ics.uci.edu/~mlearn/MLRepository.html]. 2007, Irvine, CA: University of California, School of Information and Computer Science.