EXPLORING PERCEPTUALLY SIMILAR CASES WITH MULTIDIMENSIONAL SCALINGEXPLORING PERCEPTUALLY SIMILAR CASES WITH MULTIDIMENSIONAL SCALING

Juan Wanga, Yongyi Yanga, Miles N. Wernicka, and Robert M. NishikawabJuan Wanga, Yongyi Yanga, Miles N. Wernicka, and Robert M. Nishikawab

aDepartment of Electrical and Computer Engineering, Illinois Institute of Technology, Chicago, IL 60616Department of Electrical and Computer Engineering, Illinois Institute of Technology, Chicago, IL 60616

bDepartment of Radiology, University of Pittsburgh, Pittsburgh, PA 15213

Abstract: Retrieving a set of known lesions similar to the one being evaluated might be of value for assisting radiologists to distinguish between benign andResults

Abstract: Retrieving a set of known lesions similar to the one being evaluated might be of value for assisting radiologists to distinguish between benign and

malignant clustered micro-calcifications (MCs) in mammograms. In this work, we investigate how perceptually similar cases with clustered MCs may relate to each

other in terms of their underlying characteristics (from disease condition to image features). We first conduct an observer study to collect similarity scores from a

☚ No apparent separation between cancer and benign

cases; however, there are more cancer cases (and other in terms of their underlying characteristics (from disease condition to image features). We first conduct an observer study to collect similarity scores from a

group of readers (five radiologists and five non-radiologists) on a set of 2,000 image pairs, which were selected from 222 cases based on their images features. We

then explore the potential relationship among the different cases as revealed by their similarity ratings. We apply the multidimensional scaling (MDS) technique to

cases; however, there are more cancer cases (and

fewer benign cases) in the right half of the plot than

in the left halfembed all the cases in a 2-D plot, in which perceptually similar cases are placed in close vicinity of each other based on their level of similarity. Our results show

that cases having different characteristics in their clustered MCs are accordingly placed in different regions in the plot. Moreover, cases of same pathology tend to

be clustered together locally, and neighboring cases (which are more similar) tend to be also similar in their clustered MCs (e.g., cluster size and shape). These

☚ Cases of same disease tend to be clustered together

locallybe clustered together locally, and neighboring cases (which are more similar) tend to be also similar in their clustered MCs (e.g., cluster size and shape). These

results indicate that subjective similarity ratings from the readers are well correlated with the image features of the underlying MCs in the cases, and that

perceptually similar cases could be of diagnostic value for discriminating between malignant and benign cases.

locally

☚ The neighboring cases tend to have MC clusters

similar in size and shape, and the MC clusters in the perceptually similar cases could be of diagnostic value for discriminating between malignant and benign cases. similar in size and shape, and the MC clusters in the

right half of the plot tend to be larger and irregular

Introduction

• Microcalcifications (MCs) in mammogramsFig. 2 MDS embedding of perceptually similar cases in the dataset, wherein cancer cases are denoted by “red dots” and benign cases • Microcalcifications (MCs) in mammograms

� Clustered MCs can be an important early sign of breast cancer in women

wherein cancer cases are denoted by “red dots” and benign cases are represented by “blue squares”. The spatial distribution patterns of clustered MCs are shown for some sample cases, where the spatial MC locations are indicated by the “green plus” signs.

� Content-based image retrieval (CBIR) might be of value in assisting radiologists in their diagnosis

� A major challenge facing the CBIR approach is how to retrieve images that are perceived to be

MC locations are indicated by the “green plus” signs.

� A major challenge facing the CBIR approach is how to retrieve images that are perceived to be

truly similar to the lesion under consideration

Purpose of this work• Purpose of this work

� To investigate how perceptually similar cases with clustered MCs may relate to each other in termsTo investigate how perceptually similar cases with clustered MCs may relate to each other in terms

of their underlying characteristics (from disease condition to image features)

Method☚ MCs are more

Fig. 1 A mammogram (left) and a magnified view (right)

• Mammogram Dataset

� 365 mammogram images from 222 cases were collected by the Department of Radiology at the University of Chicago

(a) Pair A (a) Pair A ☚ MCs are more

irregular and higher

in contrast in Fig. � 365 mammogram images from 222 cases were collected by the Department of Radiology at the University of Chicago

� Digitized with a spatial resolution of 0.1mm/pixel, all containing multiple MCs

in contrast in Fig.

3, and MCs are

more round and � The MCs were marked by a group of experienced radiologists, used as ground truth in the experiment

• Image Pair Selection

more round and

lower in contrast in

Fig. 4.• Image Pair Selection

� 2,000 image pairs were selected based on nine image features (features of individual MCs, spatial clustering features and texture-based features)

1,000 image pairs were selected from these 2,000 image pairs based on both the image features and similarity scores

Fig. 4.

� 1,000 image pairs were selected from these 2,000 image pairs based on both the image features and similarity scores

• Reader Study(b) Pair B (b) Pair B

� The readers scored the image pairs based on their perceptual similarity, using a discrete scale from 0 (most dissimilar) to 10 (most similar)

� To reduce the effect of reader fatigue, the 1,000 and 2,000 image pairs were divided into 4 and 10 reading sessions, respectively

Figure 3. Examples of neighboring image pairs selected from the upper-right (Pair A) and lower-right (Pair B) regions in the MDS plot.

Figure 4. Examples of neighboring image pairs selected from the middle-left (Pair A) and lower-left (Pair B) regions in the MDS plot. All

(b) Pair B (b) Pair B

� To reduce the effect of reader fatigue, the 1,000 and 2,000 image pairs were divided into 4 and 10 reading sessions, respectively

� The reading sessions were preceded by a “pre-calibration” session

Conclusion

upper-right (Pair A) and lower-right (Pair B) regions in the MDS plot. All cases are malignant.

middle-left (Pair A) and lower-left (Pair B) regions in the MDS plot. All cases are benign.

� The similarity scores of the 1,000 and 2,000 image pairs were obtained from a group of five radiologists and a group of five non-radiologists separately

• Weighted Multidimensional scaling (MDS) technique

Conclusion

We investigated how perceptually similar cases with clustered MCs may relate to each other in terms of their underlying characteristics. We• Weighted Multidimensional scaling (MDS) technique

� The average scores for the set of 1,000 image pairs (denoted by S1) which were scored by both radiologists and non-radiologists and the average scores for the

set of remaining 1,000 image pairs (denoted by S2) scored by only non-radiologists were used in MDS plot.

We investigated how perceptually similar cases with clustered MCs may relate to each other in terms of their underlying characteristics. We

conducted a reader study to collect similarity scores from a group of 10 readers on a set of 2,000 image pairs selected from 222 cases, and

applied the multidimensional scaling technique to embed all the cases in a 2-D plot according to their similarity ratings. The results showedset of remaining 1,000 image pairs (denoted by S2) scored by only non-radiologists were used in MDS plot.

� The z-score transformation was applied to both the similarity scores from individual readers before averaging

The proximity measure used in MDS is

applied the multidimensional scaling technique to embed all the cases in a 2-D plot according to their similarity ratings. The results showed

that the similarity ratings from the readers are well correlated with both the pathology and the image features of the underlying MCs.

� The proximity measure used in MDS is

Referenceretrieval and detection of masses in mammograms," Medical Physics, 34: 140-150 (2007).

1

3.75ijd

SC=

+

Average similarity scores for image pair consisting of cases i and j

� The weight is adjusted according to the level of similarity and the number of readers for scoring as following: for image pair p consisting of cases i and j,

[1] Kopans, D., "Breast imaging (3rd edition)," Williams & Wilkins (2007).

[2] Ei-Naqa, E., Yang, Y., et al, "A similarity learning approach to content-based image retrieval:

application to digital mammography," IEEE Transaction on Medical Imaging, 23: 1233-1244 (2004).

[3] Muramatsu, C., Li, Q, et al, "Experimental determination of subjective similarity for pairs of

retrieval and detection of masses in mammograms," Medical Physics, 34: 140-150 (2007).

[7] Aisen, A., Broderick, L., et al, "Automated storage and retrieval of thin-section CT images to assist

diagnosis: system description and preliminary assessment," Radiology, 228: 265-270 (2003).

[8] Borg, I., and Groenen, P., "Modern multidimensional scaling: theory and application," Springer

3.75ij

ij

dSC

=

+

consisting of cases i and j

� The weight is adjusted according to the level of similarity and the number of readers for scoring as following: for image pair p consisting of cases i and j, [3] Muramatsu, C., Li, Q, et al, "Experimental determination of subjective similarity for pairs of

clustered micro-calcifications on mammograms: observer study results," Medical Physics, 33: 3460-

3468 (2006).

[4] Nishikawa, R. M, Yang, et al, "Observer’s ability to judge the similarity of clustered calcifications

[8] Borg, I., and Groenen, P., "Modern multidimensional scaling: theory and application," Springer

(2005).

[9] Jiang, Y., Nishikawa, R. M., et al, "Malignant and benign clustered microcalcifications: automated

features analysis and classification," Radiology, 198: 671-678 (1996).21 if and 0ijp S SC∈ ≤

[4] Nishikawa, R. M, Yang, et al, "Observer’s ability to judge the similarity of clustered calcifications

on mammograms," Medical Imaging 2004: Image Perception, Observer Performance, and Technology

Assessment (2004).

[5] Zheng, B., Lu, A., et al, "A method to improve visual similarity of breast masses for an interactive

features analysis and classification," Radiology, 198: 671-678 (1996).

[10] Karahahiou, A. N., Boniatis, et al, "Breast cancer diagnosis: analyzing texture of tissue

surrounding microcalcifications," IEEE Transactions on Information Technology in Biomedicine, 12:

731-738 (2008).

[12] Wei, L., Yang, Y., et al, "Learning of perceptual similarity from expert readers for mammogram

2

1

1 if and 0

1.5 if and 0

2 if and 0

ij

ij

p

p S SC

p S SCw

p S SC

∈ ≤

∈ ≤=

∈ >[5] Zheng, B., Lu, A., et al, "A method to improve visual similarity of breast masses for an interactive

computer-aided diagnosis environment," Medical Physics, 33, 111-117 (2006).

[6] Tourassi, G. D., et al, "Evaluation of information-theoretic similarity measures for content-based

[12] Wei, L., Yang, Y., et al, "Learning of perceptual similarity from expert readers for mammogram

retrieval," IEEE Journal of Selected Topics in Signal Processing, 1:53-61 (2009).

Acknowledgement: This work was supported in part by NIH/NIBIB grant EB009905

2

1

2 if and 0

3 if and 0

p

ij

ij

wp S SC

p S SC

=∈ >

∈ >

www.mirc.iit.edu

Acknowledgement: This work was supported in part by NIH/NIBIB grant EB009905 13 if and 0ij

p S SC∈ >

www.mirc.iit.edu