Long Term Learning for Content Extraction in Image Retrieval

Pradhee Tandon and C. V. JawaharCenter for Visual Information Technology

International Institute of Information TechnologyHyderabad-500 032, INDIA

{pradhee@research.,jawahar@}iiit.ac.in

Abstract

Learning in the form of relevance feedback is popularfor bridging the semantic gap in content based image re-trieval (CBIR). However, learning across users and ses-sions did not get its due attention in CBIR. In this pa-per, we propose a computationally simple yet effectivescheme for learning relevant features for a specific im-age. Learned concept is related to the spatial distributionof pixels to identify the pixels/regions which contributeto the semantic content of the image. This learning alsomakes the retrieval more accurate. Experimental studiesin this paper validate the applicability, both qualitativelyand quantitatively.

1 Introduction

Content based image retrieval (CBIR) has receivedconsiderable attention in the last decade [1]. In CBIR,typically, an image is represented as a feature vector,and retrieval is done by finding the most similar imagesin the database. However, human perception of similar-ity could be considerably different from what a machinecould directly compute, resulting in a semantic gap. Thisnecessitated learning schemes for finding the appropriatedistance metrics [3] as well as features [4, 9]. Relevancefeedback schemes [4, 9] learned the useful features withthe help of user interactions. User refines the retrievedresults, and indirectly contributes to the feature selec-tion in these approaches. However, this class of learningschemes lack in memory. They often start from the samestate (of all features being equally relevant) for the newquery and learn to improve the precision within a user-session. This class of learning schemes are referred to asshort term learning (STL) schemes, since the learning islimited to a user session.

A complementary paradigm, which has received someattention in the recent years is Long Term Learning(LTL) in CBIR. The focus has been primarily on extend-ing the knowledge learned from user interactions to un-

Figure 1: (a) A flower in the hedge and (b) the learnedcontent

seen images with the help of techniques like LSI [2, 7, 8].This class of algorithms have been motivated by the useof the factorization schemes in text retrieval literature.However, most of these schemes require huge memoryand computation to extend the knowledge across images.Another class of LTL approaches, which got motivatedby the web mining approaches, archive the user-logs andthereby achieve learning across sessions. This class ofalgorithms use the co-occurrence patterns of images tolearn. They typically need considerable user-logs to re-sult in useful learning.

In this work, we explore the possibility of a long termlearning scheme which can seamlessly merge with thetraditional CBIR with relevance feedback. We wouldlike to retain (or use) the valuable user inputs obtainedthrough user feedback, to learn across sessions. Such alearning scheme should be: (i) Incremental in nature.It should improve with every session, but with minimalcomputations. (ii) It should be content-specific. Learn-ing should result in enhancing the image representationsand thereby reducing the semantic gap. (iii) Transparentto the retrieval. Such a learning should be transparent tothe retrieval process. Retrieval should be possible evenwhen the content is not learned completely.

Our solution is motivated by some of the simple, butpowerful techniques in text retrieval. While indexing tex-tual documents, not all words are found to be equallyrelevant or useful for the retrieval tasks. There exist ef-fective techniques for capturing the keywords, given acollection of documents [6]. Borrowing this idea, we finda set of features which could be informative for the re-

NCC 2009, January 16-18, IIT Guwahati 262

w

wxi,ci

q

LTL Update

Rf

xi,q,ci

andLTL

DB

ComputerWeight

PreprocessingRetrieval

SubsystemResults

ci

Figure 2: Long Term Learning in Retrieval

trieval task. From the pattern classification point of view,these are more of discriminative features. However theyare calculated with computationally efficient techniqueswhich avoid factorizations and eigenvector computations.In addition, when the retrieval takes place (with or with-out relevance feedback), the relevance of the features to aspecific image is learned, across sessions, to further refinethe keywords (or discriminative features).

One of the key advantages of our work is that, it allowsto discover the content in the images over time. Given animage of a flower in a garden, our approach allows to learnthe pixels corresponding to the flower without any ex-plicit segmentation or user interaction at the pixel-level,provided that multiple users have retrieved this image,while searching for the flower. Figure 1 gives an exampleof the content learned over time. The brightness of thepixel is proportional to the utility of the content.

2 Discriminative Long Term Learning

In CBIR, an image is represented using a feature vec-tor of automatically extracted visual characteristics. Letxi = [xi1, . . . , xij , . . . xiN ]T be the feature vector corre-sponding to image i in the database. When an exampleimage (query) is given, its corresponding feature repre-sentation q is computed, and a set R of images withminimal distance (d(xi,q)) to q is treated as optimal forretrieval. It has been argued that not all feature dimen-sions are equally useful for the distance/similarity com-putation. Relevance feedback based approaches estimatethe importance of features to the query concept in termsof weights for each dimension. This relevance (w) is ob-tained through continued user interactions. This is thenused in the distance computation d(xi,q,w).

The success of relevance feedback comes from the factthat not all features are relevant for a given query. How-ever, a relatively unnoticed fact has been that not all fea-tures are helpful in characterizing the semantic contentof a given image. For example in Figure 1 (a), the yel-

low flower is the useful (or popular) content rather thanthe green leaves around it. In this paper, we argue thatsuch content can be automatically characterized from thehistory of interactions, and there after used in image re-trieval.

Let ci be the relative importance of features of imagei. Then a better semantic similarity can be computed as

di = f(xi, q, w, ci) (1)

There are two possible clues which could allow us to buildan estimate of ci: (a) the similarities and dissimilaritiesof images within a database (b) Past user preferences interms of acceptable and non-acceptable images to a givenquery.

Given a collection of images, there is some amount ofinherent information in it describing the content in theconstituent images. The features which are prominentin one image and those that are not prominent in otherimages would be more relevant and should be weightedhigher while computing the semantic similarity. Such anestimate of the relevance of features to images can beapriori computed and used for image retrieval. In thecase of relevance feedback, user feedback results in twosets of images: relevant and irrelevant images. The goalis then to emphasize features which selectively prefer rel-evant images and then remember and reinforce them forthe future sessions. This relative importance is capturedwell by the consistency of features across the relevantsamples, and discriminability of irrelevant examples fromrelevant examples. The consistency in features is charac-terized by their relative low variance. Therefore the ideais to emphasize features which show high consistency overthe relevant set and high variance over others.

There are two possible possible modes in which im-age retrieval typically takes place. In the first category,a query (text or image) is given and a set of relevantimages are retrieved. User accepts (selects) some of theimages and thus gives an indirect feedback. The secondpopular approach is to use relevance feedback and alongwith query-by-example. In this case, user gives explicitfeedback to identify positive and negative images. Boththese methods can be understood as a process of splittingthe retrieved images R into two subsets P and N .

We argue that a ratio of the inverse of the variance(or any other similar measure of dispersion) of a featureover the relevant set to the inverse of its variance overthe irrelevant set captures the utility of the feature forthe retrieval task. Given a set of relevant and irrelevantimages, such a measure could be computed for each in-dividual feature j as

sj =

[σjNσjP

](2)

where σ captures the measure of dispersion. Here sj isonly an instantaneous estimate of the importance of the

NCC 2009, January 16-18, IIT Guwahati 263

feature. It changes with user feedback. Note that this iscomputed for individual features.

σ2jN =

1|N |

∑

k

(xkj − µj)2

where xk ∈ N . One could also think of using other sim-ilar measures. A similar approach has been shown to bepromising in text retrieval research. There, the idea hasbeen to select key words [6], i.e., the terms which selec-tively or dicriminatively describe the current documentwith respect to others.

In addition to selecting such key words, there is an-other aspect which is very relevant to the text processingcommunity. It is the removal of stop words from text.Stop words are words which are common in a majorityof the documents and lack any descriptive capacity. Theycould come from the apriori information coming out oflanguages or the statistics/distribution of words in thedatabase. In images, these correspond to features whichshow similar variation over all images and thus should bede-emphasized by the formulation with there weights ide-ally set close to zero. This can be efficiently accomplishedby using a modified formulation where we take the loga-rithm of the ratio of inverse of variances discussed earlier.The logarithm ensures that the features which show sim-ilar variation over the relevant set and the other imagesare weighted close to zero. The modified formulation canbe presented as:-

sj = log

[σjRσjP

](3)

First we explain, how the relative of importance of fea-tures s, which gets accumulated over iterations, can beused for computing the weight w in a relevance feedbackframework, and then how to incrementally use them forcomputing ci = [ci1, ci2, . . . ciN ]T . The estimate of theimportance of the feature sj can be used for incremen-tally updating the weight, wj , for the corresponding fea-ture as in the equation below, which can then be usedfor tuning the comparison metric,

wtj = wt−1

j + sj (4)

where wj represent the weight for feature j after the iter-ations t and (t − 1). One could also add a rate to controlthe learning across iterations. Here we have shown howthe relative importance can be captured within a user ses-sion. In a non-iterative mode, this relative importancecan be directly computed as we explain below. However,our objective is to use these weights for computing thecontent vector ci for the image i.

At the end of the session the weight vector is used forupdating the content weights for the relevant images of

this session as long term learning from this query and isthen memorized for future use as in

cij = cij + ρ wj (5)

Here wj is the weight after the final user iteration, cij

represents the relative importance of feature j in imagei and reflects the relevant content learned by the systemover all previous queries. ρ slows the learning based onthe number of past sessions. When many users accessand accept an image for a specific feature or features, weindirectly conclude that these features are important forthat image.

However, in scenarios where iterative feedback fromthe user is missing, there is no incremental learning of thefeatures relevant to the query. Such is the case even withpopular web-based image search engines. Here, whenthe images are indexed by surrounding textual content,this method can be employed. This is, in a way, similarto the standard text processing scenario where given adatabase or a collection of documents the selective termsfor all documents are to be estimated. Here the ideais to emphasize those features which better discriminatethe relevant samples with respect others. On the linesof the method discussed above for the iterative feedbackbased approaches, we expect the consistency of the fea-tures over the relevant set and their variation over theirrelevant images makes them important for the images.

With little adaptation our earlier formulation for it-erative feedback in Equation 3 fits the requirements asin

wj = log

[σ′jRσ′jP

](6)

where σ′jR and σ′jP denote the dispersions of feature jover the R and P sets. The weights thus learned are thenused for incremental long term learning as in Equation 5.

Our incrementally improving approach to long termlearning allows flexibility in the learning, controlled inrate by factors like ρ. This ensures convergence of longterm learning to the generally acceptable content in theimage. Our incremental learning approach has a distinctadvantage in terms of its computational expense, allow-ing efficient long term learning. The incremental naturemakes it independent of available archived logs of feed-back etc. It is also independent of the query so it per-forms irrespective of the availability of iterative relevancefeedback. This allows it to perform independent of the re-trieval approach employed by the system thus making ita highly portable approach for long term learning. Suchunique characteristics of our approach make it an inex-pensive and effective approach for long term learning ofcontent.

NCC 2009, January 16-18, IIT Guwahati 264

3 Experiments and Discussion

We have conducted extensive experiments to validatethe applicability of the proposed long term learning ap-proach. In this section, we demonstrate the improvementin precision with our learning. As a baseline, we compareour performance with a system which has no long termlearning. Our CBIR implementation supports query byexample, and content representation using a subset ofMPEG-7 visual descriptors [5]. This implementation alsosupports relevance feedback for intra-session learning.

The MPEG-7 features we used, primarily capturethe color and texture of the image. We have incorpo-rated Color Moments, Color Layout Descriptor(CLD)and Color Structure Descriptor(CSD) from MPEG-7 intoour representation. We have used the top three colormoments namely, mean, variance and skew. CLD firstbreaks up the image into 8 × 8 blocks and extracts thedominant color in YCbCr space for each of these andforms a 8 × 8 pseudo-image. It then computes a DCTon this small image for each channel. After scaling thecoefficients with a standard matrix we choose the top fewcoefficients for each channel in zig-zag scan order. CSDslides a 8 × 8 window over the image and computes aoccurrence based frequency histogram of the image. Weuse a combination of the above descriptors for our featurevector.

We have used a diverse set of images for all our exper-iments. The set consists of around 1000 real images withabout 100 from each of the categories including trains,surfers, hills, cars, sunset images, flowers etc. and thuspresent complex visual content.

We have used the improvement in precision percent-age across sessions to demonstrate the effectiveness of ourproposed long term learning method. We have conductedexperiments to show the performance in both presenceand absence of iterative relevance feedback. For the itera-tive feedback based experiment we have randomly pickedset of 20 queries while ensuring equal representation fromall categories. We experimented for 20 sessions with 5 it-eration for each. In each iteration, we gave feedback onthe top 48 retrieved images. The system estimates thefeature weights and performs retrieval using them. After5 iterations, these weights update the long term learningfor the relevant images. This should result in a character-istic gain in precision in the first iteration of next session.We averaged the percentage precision over the queriesand have included some randomly sampled instances inTable 1. As expected, our approach shows improvementin performance as sessions progress, in comparison to theapproach without LTL.In the next experiment, we show how our approach im-proves retrieval even in absence of incremental featurelearning. We use the same random set of 20 queries andrun the system for 20 sessions each with only one round of

(a) Sunset

(b) Rocks

(c) Flower

Figure 3: Top 7 results for 3 queries for 3 different ses-sions.

Session 1 2 5 10 20noLTL 58.7 58.7 58.7 58.7 58.7Ours 58.7 63.7 69.5 71.6 73.8

Table 1: Percentage precision with iterative relevancefeedback.

feedback, the first one. As a result relevant and irrelevantsubsets of the retrieved set are formed. Using these, ourapproach computes the update to the content vector inlong term learning. The average percentage precision forsome randomly sampled sessions is compared to the LTLfree approach in Table 2. The improved performanceagain validates our approach.

We have also included some visual results for a fewqueries from the database as shown in Figure 3. We showthe top 7 results for 3 sessions for each query. The markedimage on the left is also the query. The improving resultsin the subsequent rows show the gain with our approach.

These results validate the ability of our proposed longterm learning method to improve retrieval accuracy.

NCC 2009, January 16-18, IIT Guwahati 265

Figure 5: Content extracted from sample images usingour proposed long term content learning algorithms.

Session 1 2 5 10 20noLTL 58.7 58.7 58.7 58.7 58.7Ours 58.7 64.5 70.8 74.1 76.6

Table 2: Percentage precision in absence of iterative rel-evance feedback.

4 Content Extraction

In most of the learning solutions in CBIR, validation ofthe idea is done using the performance improvement typ-ically measured as precision. There is no explicit methodfor validating whether the semantic gap is really gettingbridged. In case of user feedback based learning, rightcontent is in agreement with the feedback of a majorityof users.Long term learning provides us an estimate of the im-

Figure 4: Content regions emerge over sessions(l to r)

portance of specific features to the content in the imageusing ci. Every pixel in the image has contributed to thefeature vector description of the image. Now, in a class ofsituations, the relative importance of a feature cij couldbe mapped back to the image. i.e., the relevance of apixel (or region) to the semantic content of the imagecan be computed. To simplify let us assume that thefeature vector is primarily a color histogram. Then theestimated content Cmn = cij iff Imn is j, where Imn

is the color of the pixel (m, n). This allows distribution

of the estimate of the content to the constituent pixelswhich have contributed to the specific feature. All pixelsare similarly assigned a relevance based gray value. Thisimage shows the most relevant as the brightest regions,performing a naive region of interest extraction. Figure 4shows how the learning based content identification im-proves with sessions for a few sample images (left-most),from no information (in second image) towards region ofinterest extraction (in the right-most).

Next we present some images with their correspondingcontent images to validate our approach over some variedconcepts in Figure 5. Our approach for content presen-tation also allows visual evaluation of the performance oflong term learning.

5 Conclusion

In this paper we have derived inexpensive incremen-tally learning solutions for long term learning motivatedby pioneering ideas in text processing. We have pro-posed methods which work independent of the retrievalapproach, making them highly portable. Our experi-ments show the effectiveness of our proposed approachon real datasets. We also visually prove the correctnessof our learning based content extraction. In future wewould like to extend our approach to also handle multi-ple concepts in images.

References

[1] R. Datta, D. Joshi, J. Li, and J. Z. Wang. Image retrieval:Ideas, influences, and trends of the new age. ACM Com-put. Survey, 40(2):1–60, 2008.

[2] D. Heisterkamp. Building a latent semantic index of animage database from patterns of relevance feedback. Proc.of Intl Conf on Patt. Recog., 4:134–137, 2002.

[3] S. C. Hoi, W. Liu, and S.-F. Chang. Semi-superviseddistance metric learning for collaborative image retrieval.Conf. on Comp. Vision and Patt. Recog., pages 1–7, June2008.

[4] T. Huang and X. S. Zhou. Image retrieval with relevancefeedback: from heuristic weight adjustment to optimallearning methods. Proc. of the Intl Conf on Image Proc.,3:2–5, 2001.

[5] J. M. Mart́ınez. Mpeg-7: Overview of mpeg-7 descriptiontools, part 2. IEEE MultiMedia, 9(3):83–93, 2002.

[6] G. Salton and C. Buckley. Term-weighting approaches inautomatic text retrieval. In Information Processing andManagement, pages 513–523, 1988.

[7] A. Shah-Hosseini and G. M. Knapp. Learning image se-mantics from users relevance feedback. In Proc. of ACMMultimedia, pages 452–455, New York, USA, 2004.

[8] T. Yoshizawa and H. Schweitzer. Long-term learning ofsemantic grouping from relevance-feedback. In Proc of theIntl Workshop on MIR, pages 165–172, NY, USA, 2004.ACM.

[9] X. S. Zhou and T. S. Huang. Relevance feedback in imageretrieval: A comprehensive review. Multimedia Systems,8(6):536–544, April 2003.

NCC 2009, January 16-18, IIT Guwahati 266