A Robust Content-Based Image Retrieval System Based On: Color, Texture, and Color Layout

Abstract. Contbecoming an effecbased on their vibuilding CBIR representing the vtechniques that comparison has b(GCH) alone andWavelet transformfeature. Differentextracting color, tand used as the bimage and the dacombining GCH with color layout alone and they rinformation besid Index Terms—CoHistogram, 2-D H

T hete

last fewchniques

Retrieval (CBIRprocess of retrievbased on extractwithout resortingdirectly from thanalyzed automa[2]. Many commretrieval systemsexample: QBIC, searching image colors and texcomparing featusearch image witdatabase. In general, featfeatures accordinsemantics [1]. Wof features, therecombinations andbased retrieval inthe best matches systems. In fact, some CBcases of databasefor testing the acthe most importCBIR systems pcollection and obBased on the lowclassified into fou

ICENCO 2004

381

A Robust Content-Based Image Retrieval System Based On: Color, Texture, and Color layout

Mohamed A. Tahoun1, Khaled A. Nagaty2, Taha I. El-Arief2, Mohammed A-Megeed3

1. Computer Science Dep., Faculty of Computers and Informatics, Suez Canal University, Egypt

2. Computer Science Dep., Faculty of Computer and Information Science, Ain Shams University, Egypt. 3. Scientific Computing Dep., Faculty of Computer and Information Science, Ain Shams University, Egypt

E-mails: matahoun@yahoo.com, knagaty@asunet.shams.edu.eg, taha_elarif@yahoo.com , and mamegeed@hotmail.com

ent Based Image Retrieval (CBIR) is tive approach that used to retrieve images

sual features like color and texture. When systems, this requires choosing and isual features and finding combinations of

give the best matches. In this paper, a een done between Global Color Histogram the combination of GCH and 2-D Haar with and without adding the color layout

categories of images have been tested by exture and color layout features from them asis for a similarity test between a query

tabase images. The experiments show that and 2-D Haar wavelet transform together feature gives better results than using them eflect the importance of using the spatial e the color feature itself.

ntent-Based Image Retrieval, Global Color aar wavelet Transform, Color layout.

I. INTRODUCTION

years have witnessed many advanced evolving in Content-Based Image

) systems. CBIR is considered as the ing desired images from huge databases ed features from the image themselves to a keyword [1]. Features are derived e images and they are extracted and tically by means of computer processing ercial and research content-based image have been built and developed (For Netra, and Photobook [3]). CBIR aims at libraries for specific image features like

tures and querying is performed by re vectors (e.g., color histograms) of a h the feature vectors of all images in the

ures are classified into low and high level g to their complexity and the use of

hen retrieving images using combinations is a need for testing the accuracy of these comparing them with the single features order to find the combinations that give which enhance the performance of CBIR

IR systems give good results for special images as till now no standard data set curacy of CBIR Systems [4]. So one of ant challenges facing the evaluation of erformance is creating a common image taining relevance judgments [5].

-level features, CBIR strategies can be r categories:

A) Color Color is the first basic visual feature. It is relatively robust to background and independent of image size and orientation, thus it constitutes a very important attribute for image retrieval [3]. The color-based visual feature is captured by the image histogram which is constructed by computing the number of pixels in each color. A Global Color Histogram (GCH) is the most traditional way of describing the color property of an image [6]. Color representation techniques include: Color Histograms, Cumulated Color Histograms, Color Moments, and Color Sets [3]. There are two classes of techniques for color indexing: indexing by global color distribution which enables only whole images to be compared, and indexing by local or region color which enables matching between localized regions within images [7]. In image distance measures, the color content of one image is compared with second image or a query specification. Color histogram matching is an example of such measures where it looks for low histogram difference between the images.

B) Texture

Texture refers to the visual patterns that have properties of homogeneity that do not result from the presence of only a single color or intensity (for example: clouds, bricks, fingerprint, and rocks textures). It is important and useful in Pattern Recognition and Computer Vision research areas. The visual texture properties include: Coarseness, contrast, directionality, line- likeness, regularity, and roughness [3]. ( Fig. 1) Fig. 1 Textures Examples. From left to right: Bricks ,

Fingerprint, Clouds and Rocks textures. Texture representation techniques include: Co-occurrences Matrix, Wavelet Transform, Wavelet Transform together with KL expansion and Kohenon Maps, Wavelet Transform with Co-occurrence Matrix, and Gabor Wavelet Transform which was the best among many texture feature representations [3]. The most popular statistical representations of texture are Co-occurrences Matrix, Tamura Texture, and Wavelet Transform. Texture similarity techniques include texture description vector which summarizes the texture in a given image.

C) Shape

Shape is considered as the characteristic surface configuration of an object, an outline, or contour. Shape representation techniques can be divided into two main categories [6] [3]: the first is boundary-based where the former uses outer boundary of the shape and the second is region-based which uses the entire shape region and both aspects must be captured in qualitative sense [8]. The best way to enhance object-specific information in images is by segmenting the object in the image [9]. Netra, and Blobworld present a user with the segmented regions of an image and the user selects regions to be matched together with other attributes like color and texture [10]. Fourier Descriptor and Moment Invariants are successful representatives for these two categories while other methods of representation include: Polygonal models, boundary partitioning, curvature models, and implicit polynomials [3] [8]. Shape similarity techniques include shape histograms, boundary matching, and sketch matching.

Global Color Histogram (GCH) is the most traditional way of describing the color attribute of an image. It is constructed by computing the normalized percentage of color pixels in an arrange corresponding to each color elem nt [6].

D) Color Layout Color histogram does not include any spatial information about an image and we may find images that have the same color histogram although they have different color distributions. (Fig. 2) For this reason, many research results suggested that using color layout (both color feature and spatial relations) is a better solution in image retrieval [4]. Spatial relationship symbolizes the arrangements of objects within an image. Many methods for combining color information with spatial layout have developed while retaining the advantages of histograms. Color layout representations include: Color Tuple Histogram and Color Correlogram [11]. The rest of the paper is organized as follows: section II briefly covers the feature extraction and indexing processes using Global Color Histogram, 2-D Haar Wavelet transform, and the color layout algorithm, section III includes the experimental results, and finally with concluding remarks.

II. FEATURES EXTRACTION AND INDEXING

One of the most important challenges when building image based retrieval systems is the choice and representation of the visual features [6]. Color is the most intuitive and straight forward for the user while shape and texture are also important visual attributes but there is no standard way to use them compared to color for efficient image retrieval. Many content-based image retrieval systems use color and texture features [10]. In order to extract the selected features and index the database

images based on them, we used Global Color Histogram (GCH) and 2-D Haar Wavelet transform to extract the color and texture features respectively, and then constructing the color and texture features vectors. With respect to the color layout feature, the algorithm for the extraction and the indexing processes is explained. A) Global Color Histogram (GCH)

Fig. 2 Three images have the same color histogram but on the other hand have different color distributions

A histo In o256dataand combe simiquerwe com

wheis oof pimaof ccorrGretrandistadisp

DC

ICENCO 2004

382

e

Fig.3. (a) A colored image, (b) The three components Red, Green ,and Blue, and (c) The three corresponding histograms for Red, Green, and

Blue components in (b) respectively

true colored (RGB) image and the corresponding grams of each component are displayed in Fig. 3. rder to construct the color feature vector (its length is

×3) for both the query image and all images in the base, we identify the three color components (R, G, B)

compute the corresponding histograms of these ponents. The histograms for all database images will saved in the feature vector database. To test the larity between the two feature vectors one for the y image and the other for each image in the database used Manhattan Distance (1) as the histogram

parison distance measure :

∑=

−=G

j kk

k

ii

iki NM

jH

NM

jHD

1, *

)(*

)( (1)

re Hi(j) denote the histogram value for the ith image, j ne of the G possible gray levels, is the number ixels in an image i, M is the number of pixels in ge k, and M is the number of rows and N is the number olumns. We calculate the difference between each two esponding histograms of the three components Red, en, and Blue. Then we use the following sformation to convert the three distances into one nce that will be sorted and used as the basis for laying the results:

ii NM *

kk N*

(2)(B)D* 0.114 (G)D *0.587 (R) D* 0.299 CCC ++=

Where is the distance between the two color feature vectors, and D , , and are the distances between each two corresponding components for Red, Green, and Blue respectively [12]. The obtained distances after comparing the query image with all the database images will be sorted and the corresponding images are displayed during the retrieval process.

CD(R) C (G)DC (B)DC

B) 2-D Haar Wavelet Transform The wavelet transform is a tool that cuts up data or functions or operators into different frequency components and then studies each component with a resolution matched to its scale. Wavelet transform can be used to characterize textures using statistical properties of the gray levels of the points/pixels comprising a surface image. Formulas (3) and (4) illustrate the mother wavelets for the Haar wavelet:

Where φ is called the scale of the Haar wavelet and ψ is the actual wavelet (Fig. 4) [13]. The 2-D Haar wavelet analysis of an image gives four outputs at each level of analysis l ( l=3 in our experiments), one approximation and three details, the approximation Al, horizontal details Hl, vertical details Vl, and diagonal details Dl.(Fig. 5) The energy of each sub-band image is calculated using the following relation (5):

( )EMN

X i jj

n

i

m

===∑∑1

11,

Where M and N are the dimensions of the image, and X is the intensity of the pixel located at row i and column j. The texture feature vector will consist of the energies of horizontal, vertical and diagonal details for the three levels of analysis (the length of the texture feature vector is 9). Then, we apply the Euclidean distance (6) between each two sets of energies (feature vectors) one for the query image and the other for every image in the database and this process is repeated until comparing all images in the database with the query image [6].

( )2

1,∑

=

−=K

kikk

Ti yxD

(6)

Where K is the length of the texture feature vector, i represents the ith image in the database, and D is the Euclidean distance between the query image feature vector

and the i

Ti

x th database image feature vector . iy

C) Color Layout Extraction Because of the shortagedescribing the color featurspatial information beside th In traditional color layout divided into equal-sized blcomputed on the pixels iobtained values are storeEuclidean metric. In our experiments the stepfeature vector from an imag

1- Divide the image into 2- Extract the color fe

components for each s3- Calculate the avera

components in each color layout feature represent the color lay

In order to test the simillayout feature vectors one other for each image in the ddistance measure (7) [6]:

(3)

(4)

Fig. 4 The Haar Wavelet ψ and its scale function Φ

(5)

Fig. 5 (a) The original image,approximation and three detailDiagonal (D)), and (c) The threfeature vector consists of the energ

(b)

A3 H3 V3 D3

H2

V2 D2

Horizontal Details H1

Vertical Details V1

Diagonal Details D1

(a)

(c)

ICENCO 2004

383

(c)

Algorithm

of color histograms when e, there is a need to use the e color feature itself [4]. image indexing, the image is

ocks and the average color is n each block [14] [15]. The d for image matching using

s for creating the color layout e are: 16x16 sub-blocks. ature for each of the RGB ub-block. ge for each of the three sub-block and construct the vector (16x16x3) that will

out feature. arity between each two color for the query image and the atabase we used the Euclidean

(b) 3-level 2-D Haar wavelet (an s (Horizontal (H), Vertical(V), and e levels of analysis where the texture ies of Hl, Vl, and Dl and l =1 to 3.

( )2

1,∑

=

−=S

siss

CLi NMD

where S is the length of the color layout feature vector,

is the Euclidean distance between the query image

feature vector and the i

CLiD

iNM th database image feature vector

. After calculating the distances between each two color components we transformed them into one distance value [12] (8):

)(*0.114)(* 0.587)( * .2990 BDGDRDD CLCLCLCL ++=

where is the final Euclidean distance between the two color layout feature vectors and D , , and

are the Euclidean distances between each two corresponding components for Red, Green, and Blue respectively.

CLD

))(RCL )(GDCL

(BDCL

III. EXPERIMENTAL RESULTS

The general flow of the experiments starts with the features extraction process (based on GCH, 2-D Haar wavelet transform, and color layout feature) that is used to create the features vectors for each image in the database that are stored to be ready for the matching process (offline processing). When starting with a query image the same process will be done for each query image (online processing). The comparison between each two features vectors (one for the query image and the other for each database image) is performed using the Euclidean distance which is the most common form of the distance function and then these distances are normalized and sorted respectively and then they used as the basis for retrieving database images that are similar to the query [6] (Fig. 6). The database images contains 180 compressed colored (RGB) images downloaded from the internet [16]. The image collection includes four categories of images:

People, Roses, Animals, and Buses and all images are in size 384x256 pixels in jpg format. Fig. 7 displays a sample from each category in the database images. The experiments were run on Pentium IV, 2.4 GHz Processor and 256 MB RAM using Matlab version 6.

(7)

People

(8) Roses

Animals

Fig. 7 Samples of the four categories of the database images.

Buses

A) Accuracy Test In order to test the CBIR system performance using GCH only, the combination of GCH and 2-D Haar wavelet, and the combination of GCH and 2-D Haar wavelet in addition to color layout feature, the accuracy test were done to find the best results in each case. The retrieval accuracy is defined as the ratio between the number of relevant ( similar or fall in the same category) retrieved images and the total number of retrieved image (known as a single precision ) [5]. Query images are selected from one category of images and after applying the used techniques, the results are compared to the ones obtained using the collection containing all categories of images. It is remarked that the first relevant retrieved images are near to be the same in the two cases specially when using a combination of GCH and 2-D Haar Wavelet transform and this accuracy increases when adding the color layout feature.

Database

Retrieved images

Online processing Sorting Distances

Features Extraction

(GCH/Haar Wavelet/

Color layout)

Apply Metric between feature vectors Query

Images

Fig. 8 (a) The results obtained in the Buses category

Offline processing Features Extraction (GCH/Haar Wavelet/ Color layout)

Animals

Buses

People

Roses

Fig. 6 The general flow of the experiments applied on both query image and the stored images.

ICENCO 2004

384

Fig. 9 (b) The results based on GCH and 2-D Haar Wavelet

Fig. 8 (a) displays the results (the most similar 19 images beside the query image itself) obtained using the same category (Buses) based on the combination of GCH and 2-D Haar wavelet in addition to color layout feature while Fig. 8 (b) displays the results for the same query using the same techniques but in the collection containing all categories of images (People, Roses, Animals, and Buses) and it is shown that there are 15 similar images between the two cases. On the other hand, the number of similar images becomes less when using GCH only or the combination of GCH and 2-D Haar wavelet without adding the color layout feature. The next step in the accuracy test is to test the effect of GCH alone and the combination of GCH and 2-D Haar wavelet with and without the color layout feature using all database images.

In Fig. 9, the number of relevant retrieved images in (a3 images, in (b) is 12 images, and in (c) is 15 images, ait is clear that case number (c) gives the best results arrangement.

Fig. 8 (b) The results obtained in the collection of all database images

The same test was done for the other categories (PeopAnimals and Buses) and good results have been achievand confirmed that GCH and 2-D Haar wavelet transfoin addition to color layout gives the best retrieval accurcomparing with the retrieval accuracy when using Gonly or GCH and 2-D Haar wavelet transform alone.

TABLE I. The retrieval accuracy based on GCH only, and tcombination of GCH and 2-D Haar wavelet with and without a

the color layout feature.

The retrieval accuracy Percentage Techniques

People Roses Animals BusGlobal Color

Histogram (GCH) 84.35% 52.13% 80.95% 58.1

GCH + 2D Haar Wavelet 85.00% 87.75% 82.30% 83.7

GCH + 2D Haar Wavelet + Color

layout 85.32% 93.00% 88.65% 86.0

Table I illustrates the general retrieval accuracy percentobtained when using different query images ( 45 imageeach category) based on GCH only, the combinationGCH and 2-D Haar wavelet transform alone, and finawhen adding the color layout feature.

69%

85% 88%

0%

20%

40%

60%

80%

100%

GCH GCH+ 2-DHaar CGH+2-D Haa+ Color lyout

Ret

rieva

l Acc

urac

y

Fig. 9 (a) The results based on Global Color Histogram (GCH) only

Fig. 10 The general retrieval accuracy when using: GCH alone, GC

2-D Haar wavelet, and finally when adding to them the color layout Figure 10 displays the general retrieval accuracy obtainfrom testing all categories in the database where the bmatches obtained when combining GCH, 2-D Hwavelet together with color layout feature. These resureflect the importance of taking into account the spainformation beside the color feature itself. It is remarkthat the combination of the three techniques gives go

ICENCO 2004

385

Fig. 9 (c) The results based on the combination of GCH and 2-D Haar wavelet beside color layout feature

) is nd

and

le, ed rm

acy CH

he dding

es

3%

5%

9%

age s in of lly

r

H and feature.

ed est aar lts

tial ed od

results with both images that have low and high details as images in Roses and People categories respectively. On the other hand Global Color Histogram alone fails to give covenant results when the images have low details such as images in Roses and Buses categories. It is also noted that the details of images can not be completely described by color histograms and texture alone, so it is important to take into account the spatial information of colors in the image.

B) Noise Test In order to show the change in the retrieval accuracy as noise is added, a normal noise with different variances is added to the images. The noisy images are used as the query images and the rank of the original images is observed. We add six levels of noise ( from 10% to 60% - named N10 to N60) to each query image then we compare between the results obtained when using the noisy query images and the ones obtained using the original images based on: Global Color Histogram (GCH) alone, GCH and 2-D Haar wavelet transform with and without adding the color layout feature. In Fig 11, the retrieval accuracy (with respect to original images) when applying the six levels of noise based on GCH and the other two combinations which show that combining GCH and 2-D Haar wavelet in addition to color layout feature gives the best retrieval accuracy with respect to the original images.

IV. CONCLUSIONS The need for efficient content-based image retrieval systems becomes a must and the choice and representation of the visual features when building CBIR systems are important tasks. In this paper, a comparison has been done between GCH alone and the combination of GCH and 2-D Haar wavelet transform with and without adding the color layout feature. Different categories of images have been tested based on these techniques and the results showed that using GCH and 2-D Haar wavelet transform gives good results and when combining them with the spatial information, the retrieval accuracy increases and help enhancing the CBIR systems performance.

V. REFERENCES

[1] John Eakins and Margaret Graham, “Content-based

Image Retrieval”, JISC Technology Applications Programme. University of Northumbria at Newcastle. January 1999. http://www.unn.ac.uk/iidr/report.html

[2] Christopher C. Yang, “Content-based image retrieval: a comparison between query by example and image browsing map approaches “,Journal of Information Science, pp. 254–267, 30 (3) 2004.

[3] Rui Y. & Huang T. S., Chang S. F. “Image retrieval: current techniques, promising directions, and open issues”. Journal of Visual Communication and Image Representation, 10, 39-62, 1999.

[4] Ahmed M Ghanem, Emad M. Rasmy, and Yasser M. Kadah, “Content-Based Image Retrieval Strategies for Medical Image Libraries,” Proc. SPIE Med. Imag., San Diego, Feb. 2001.

[5] Henning Muller, Wolfgang Muller, David McG. Squire and Thierry Pun, “Performance Evaluation in Content-Based Image, Retrieval: Overview and Proposals”. Computing Science Center, University of Geneva, Switzerland, 2000.

[6] Vishal Chitkara “Color-Based image Retrieval Using Binary Signatures “Technical Report TR 01-08, University of Alberta, Canada, May 2001.

[7] John R. Smith and Shih-Fu Chang: Tools and Techniques for Color Image Retrieval, ACM Multimedia '95, San Francisco, Nov 5-9, 1995.

[8] Benjamin B. Kimia, “Symmetry-Based Shape Representations, ” Laboratory for Engineering Man/Machine Systems (LEMS), IBM, Watson ResearchCenter,October1999,http://www.lems.brown.edu/vision/Presentations/Kimia/IBM-Oct-99/talk.html

[9] Arnold W.M. Smeulders, Marcel Worring, Simone Santini, Amarnath Gupta, and Ramesh Jain, “ Content- Based Image Retrieval at the End Of the Early Years “,IEEE. Transactions On Pattern Analysis And Machine Intelligence, vol. 22, no. 12, December 2000.

[10] Qasim Iqbal and J. K. Aggarwal, “Combining Structure, Color, and Texture for Image Retrieval: A performance Evaluation”,16th International Conference on Pattern Recognition (ICPR), Quebec City, QC, Canada, August 11-15. 2002, vol. 2, pp. 438-443.

[11] Jing Huang and Ramin Zabih, “Combining Color and Spatial Information for Content-based Image Retrieval”, Computer Science Department, Cornell University,Ithaca,NY14853,http://www.cs.cornell.edu/rdz/Papers/ecdl2/spatial.htm.

[12] Web Sites: http://en.wikipedia.org/wiki/YIQ and http://v4l2spec.bytesex.org/spec/colorspaces.html

[13] M.G. Mostafa, M.F. Tolba, T.F. Gharib, M.A. Megeed,” Medical Image Segmentation Using Wavelet Based Multiresolution EM Algorithm”, IEEE International Conference on Industrial Electronics, Technology, & Automation., Cairo IETA’2001.

[14] Tat Seng Chua, Kian-Lee Tan, and Beng Chin Ooi, “Fast Signature-based color-spatial image retrieval “. In Proc. IEEE conf. On Multimedia Computing and Systems, 1997.

[15] C. Faloutsos, M. Flickner, W. Niblack, D. Petkovic, W. Equitz, and R. Barber., “ Efficient and effective quering by image content”, IBM Research report, Ayg. 1993.

[16] Web Site: http://wang.ist.psu.edu/~jwang/test1.tar

0%20%40%60%80%

100%

N10 N20 N30 N40 N50 N60

Six Levels of Noise (from 10% to 60%)

Ret

rieva

l Acc

urac

y

GCH

GCH+2D Haar

All Features

Fig. 11 The effect of noise on the retrieval accuracy with respect to

original images

ICENCO 2004

386