PARAgrab: A Comprehensive Architecture for Web ImageManagement and Multimodal Querying

Dhiraj Joshi 1 Ritendra Datta 1 Ziming Zhuang 2

WP Weiss 2 Marc Friedenberg 2 Jia Li 3,1 James Z. Wang 2,1.

1Department of Computer Science and Engineering,2College of Information Sciences and Technology, and 3Department of Statistics

The Pennsylvania State University, University Park, PA, USA.

ABSTRACTWe demonstrate PARAgrab - a scalable Web image archival,retrieval, and annotation system that supports multiplequerying modalities. The underlying architecture of ourlarge-scale Web image database is described. Querying andvisualization techniques used in the system are explained.

1. INTRODUCTIONDigital image databases have attracted a large amountof commercial and academic interest in the recent years.Research has shown that image databases require differentialtreatment. The semantic content of pictures is significantlydifferent from their digitized representation, a problem oftenreferred to as semantic gap [10]. Moreover, only a fractionof images have manual labels and these labels often sufferfrom synonymy and polysemy [5]. As an alternative, peoplehave attempted to automate image interpretation by theirvisual content, a technology known as content-based imageretrieval (CBIR). Comprehensive surveys of CBIR can befound in [10, 1].

Design and management of very large multimedia databasesis a challenging task. An experimental system supportingmulti-paradigm querying for multimedia databases ispresented in [12]. Adaptation of similarity searches to userneeds in large multimedia databases has been addressedin [8]. Clustering in very large spatial image databases hasbeen studied in [9]. With the explosion of the Internet,a number of Web-based image retrieval systems have beenproposed. Among those that incorporate visual featuresinto the retrieval process are the iFind system [13] and theCortina system [7]. The SIMPLIcity system [11] has beenapplied to art, photography and other domain-specific imagedatabases.

Permission to copy without fee all or part of this material is granted providedthat the copies are not made or distributed for direct commercial advantage,the VLDB copyright notice and the title of the publication and its date appear,and notice is given that copying is by permission of the Very Large DataBase Endowment. To copy otherwise, or to republish, to post on serversor to redistribute to lists, requires a fee and/or special permission from thepublisher, ACM.VLDB’06 September 12–15, 2006, Seol, Korea.Copyright 2006 VLDB Endowment, ACM 1-59593-385-9/06/09

2. SYSTEM ARCHITECTUREPARAgrab has been designed in a modular fashion to ensure(1) efficient processing of large volumes of dynamicallychanging WWW image data, and (2) smoothness andparallelism of the work-flow. The efficiency in the work-flowis achieved primarily from the fact that the dynamic natureof the Web is reflected by the crawling/indexing modulesindependent and transparent to the query processingmodule. Fig. 1 shows the PARAgrab architecture.

2.1 Modular DesignThe PARAgrab system has the following main modules:

Crawler Module: A focused crawling strategy has beenadopted for our system. If a URL has successfully led to animage of interest, the URLs nearby and those at the samelevel are considered more promising. As a result, they areprioritized so that in the future, they will be explored first.Authoritative image Websites, such as PBase and Flickr, areused as seed URLs.

Indexing Module: Building indexes offline in orderto expedite online retrieval is critical to any large scaledatabase. Since the data being collected is highly multi-modal in nature, separate indexes need to be maintained tofacilitate simple or combined query modalities. We attackthis problem by using a MySQL database to store imagemeta-data and text attributes. We currently use MySQLto build text indexes. The method of building indexes onimage features involves running a batch job periodically toprocess newly added pictures. The job consists of multipleparallel processes run on a Linux cluster.

Query Processing Module: The query processingmodule interacts directly with the meta-data and imageindexes, a cache of ranked images, and a content filteringmodule to generate search results for the interface module.The query modalities supported are described in Sec. 3.

Interface Module: In PARAgrab, we aim at achieving aunified visualization of search results in order to maintainhigh user interest levels. Our interface, as shown in Fig. 2,presents search results in a simple and concise way so thatusers can search and browse with ease. Moreover, theinterface provides usage logs for potential improvements inretrieval performance.

CRAWLER

MODULE

INDEXING

MODULE

13323134134287

3254765

......................

...........

............................................

......................

...........

............................................

......................

...........

............................................

USER TAGS

INDEX

QUERY PROCESSING

MODULE

IS

CACHED

CACHE

........

BUFFER

WEB

car

MODULE

INTERFACE

USER LOG

CONTENT FILTER

LEARNING MODULE

YN

Figure 1: A schematic view of the PARAgrab architecture.

Figure 2: A screenshot of the search results obtainedfor the query dog. Note the unified visualization ofsearch results.

2.2 Image RankingIn PARAgrab, visual content based image retrieval isperformed using the Integrated Region Matching (IRM)distance [11]. A key advantage of using IRM is thatimage ranking, over large collections, can be performedin real time. This scalability is achieved by modularizingthe retrieval process into separate database indexing anddistance computation steps.

Among text-based methods, our system currently supportsuser tag and URL based image search. Tag matchingis performed in a way analogous to text based querying.URLs may not always be useful in determining imagesemantics. Yet, sometimes very high level information isembedded in the URL, e.g., name of the photographer,subject, organization, institute, etc. Our algorithm for URLbased matching is based on two intuitions: (1) An image ismore relevant to a query if it is closer to the right-mostsegment (2) An exact query match has greater relevancethan a match with its stemmed version.

3. QUERYING AND VISUALIZATIONThe ultimate verdict on the success of a Web-based systemis typically made by its end-users. In the PARAgrabsystem, we essentially walk the tight-rope between a feature-rich system and a user-friendly interface. The user iswelcomed with a random sample of 10 images from thepicture database. The database journey can be started by(1) keyword query, (2) clicking on an image, and (3) personalfile upload. For any picture, the user can add/remove tagsassociated with the picture. Fig. 2 depicts a typical keyword-based search result, consisting of three ranked lists puttogether in a compact representation. Clicking on a picturelaunches the visual similarity search. PARAgrab supportsthe following query refinement features:

Spelling Suggestions: For every query word, the systemcomputes its Levenshtein (edit) distance with each uniquetag in order to enlist closely spelled words (within athreshold), if any. We rank spelling suggestions in theincreasing order of their edit distances to the query.

Semantics Expansion: Query keywords yield WordNetbased synonyms, hypernyms, hyponyms, and sister termsordered by familiarity of use [5]. These words arerecommended to the user, hyper-linked in a way thatinitiates new keyword-based queries.

The primary advantage of content based image retrieval is todiscover images pertaining to a given concept in the absenceof reliable meta-data. In Fig. 3, we show how a combinationof text-search followed by visual-search achieves this goal.A two-level search beginning with the text query cat yieldsa number of pictures of cats which did not have any cat-related meta-data associated with them. Boxes are drawnaround the images used to initiate further visual searches,in Fig. 3 (a). The newly discovered cat images are shownenclosed within boxes in Fig. 3 (b) and (d). This strategyhas immense potential for human-guided tag propagationi.e., the newly discovered cat images can now be taggedby the user to assist in future keyword based queries.Interestingly, another second level visual search identifiesnear duplicates of the query image, as shown in Fig. 3 (c).Fig. 4 shows another instance where visual search retrievesnear-duplicates.

(a) (b)

(c) (d)

Figure 3: Depiction of a two level search usingthe PARAgrab system. The top-left figure showsthe results obtained for the text query cat. Theother three figures show the result of visual searchperformed on images from the first step. Boxes aredrawn around the specific pictures clicked.

Figure 4: A classic case where visual similarity helpsdiscover near-duplicates.

Typically, in a text search engine the only way a user canbrowse for relevance is through inspection of result pagesin rank order. However, in PARAgrab, given the facilityto search by image content, a user can explore picturesin sequential chains of successive similarity queries. Thevariety in such chains arises from the choice the user makesat each step in order to proceed with browsing. We refer tothese as:Strategy 1: Begin with a single query image and browseresult pages strictly in the order of visual similarity to query.Strategy 2: Begin with a single query image and thenhierarchically explore the next level, making each top levelresult as a subsequent query.Strategy 3: Begin with a single query, and keep performingvisual searches based on personal preference and relevanceto query. The sequence thus can potentially deviate furtherand further from the original query.

Fig. 5 presents a conceptual representation of the discussedsearch strategies. We next conduct a short study on threetext queries namely mountain, building, and people, usingthe results as seeds for employing different strategies. We

Strategy 1

Strategy 3

Strategy 2

Figure 5: Conceptual representation of threebrowsing strategies. Circles represent localneighborhoods in the image feature space. Arrowsrepresent jumps during browsing.

restrict this study to the strategies 1 and 2. In Fig. 6, thenumber of un-tagged relevant images obtained are recordedand plotted with the number of images browsed. Allnumbers are averaged over the first three seed imagesof mountain, building, and people respectively. It isevident that such compound browsing schemes are capableof revealing images otherwise impossible for simple textbased searches. The number of un-tagged relevant imagesdiscovered in the 110 image browsed (11 user clicks, ignoringbrowser buttons) for mountain, building, and people is about200%, 99%, and 62% more than the number obtained bytext-query alone. While we observe no significant differencesin the two strategies, differences are noticed among differentquery concepts in relevant image discovery. Given theflexibility of strategy 3, whether its employment makes anydifference would be an interesting new direction.

4. SIGNIFICANCE OF CONTRIBUTIONWhile there has been extensive research in the field of imageretrieval, we believe that our key contribution is that ofintegrating many successful ideas from multiple researchdomains into a large-scale working system. We feel that ourwork will promote image retrieval in the database researchcommunity and generate interest in database-related aspectsof the problem. Some other contributions are summarizedbelow.(1) In PARAgrab, we combine structured (meta-data)and unstructured (visual features) information into anintegrated image management framework.(2) Our modular architecture is capable of scaling up tohandle very large volumes of data. With a database ofabout half a million Web images and growing, PARAgrab ispublicly available for wide usage [6].(3) Multi-modal searching by keywords, visual examples,and local image file uploads are supported. To the bestof our knowledge, PARAgrab is the first large-scale workingsystem that incorporates all these modalities into a singleframework.(4) Besides the visual and text based ranking methods

10 20 30 40 50 60 70 80 90 100 1100

10

20

30

40

50

60

Number of images browsed

Num

ber

of r

elev

ant i

mag

es fo

und

mountain

building

people

10 20 30 40 50 60 70 80 90 100 1100

5

10

15

20

25

30

35

40

45

Number of images browsed

Num

ber

of r

elev

ant i

mag

es fo

und

mountain

building

people

Figure 6: Plots of the number of un-tagged relevant images discovered using strategies 1 and 2 respectively.The X axis represents the number of images browsed.

discussed before, PARAgrab also supports popularity-basedranking.(5) We incorporate natural language processing techniquesin image search to enable query refinement.(6) Offline caching of visual similarity ranking is performedto support real-time search. Caching is performed at regularintervals to reflect the dynamic nature of the database.(7) In a fashion similar to Flickr, PARAgrab supportsadding and removing picture tags. We collect the humanprovided annotations as meta-data to improve future searchby keywords.We believe that PARAgrab architecture will be a testbed fornew ideas and approaches to retrieval. In the near future,we plan to extend the system to support techniques such asclustering, relevance feedback, aesthetics based ranking [2],and story picturing [4]. We will also make the data collectedfrom user interactions available for research purposes.

5. DEMONSTRATION PLANOur demonstration will include a comprehensive walk-through of the PARAgrab system.We will present thevarious querying modalities namely (1) visual image search,(2) text based image search, and (3) search using localimage file uploads. Through this, we will demonstrate theadvantages of combining multiple modalities for enhancedimage search, contrasting it with Google Images. Thequery refinement features will also be elaborated during thedemonstration session. We will compare and contrast thebrowsing strategies (Fig. 6) and how they affect relevanceto query. The advantages of rank combination will beshown through various examples. In this process, feedbackthrough audience participation will help us understand end-user requirements better. We expect that this experiencewill familiarize the database community with the challengesinvolved in building useful image retrieval systems.

6. ACKNOWLEDGMENTSThis work is supported in part by the US National ScienceFoundation under Grant Nos. IIS-0219272, IIS-0347148,and ANI-0202007, The Pennsylvania State University, thePNC Foundation, and SUN Microsystems.

7. REFERENCES[1] R. Datta, J. Li, and J. Z. Wang, “Content-Based Image

Retrieval - A Survey on the Approaches and Trends of theNew Age,” Proc. MIR Workshop, 2005.

[2] R. Datta, D. Joshi, J. Li, and J. Z. Wang, “StudyingAesthetics in Photographic Images Using a ComputationalApproach,” Proc. ECCV, 2006.

[3] H. Feng, R. Shi, T. S. Chua, “A Bootstrapping Frameworkfor Annotating and Retrieving WWW Images,” Proc. ACMMultimedia, 2004.

[4] D. Joshi, J. Z. Wang, and J. Li, “The Story PicturingEngine - A System for Automatic Text Illustration,” ACMTrans. on Multimedia Computing and Applications,2(1):68-89, 2006.

[5] G. Miller, “WordNet: A Lexical Database for English,”Comm. of the ACM, 38(11):39-41, 1995.

[6] PARAgrab, http://paragrab.psu.edu.

[7] T. Quack, U. Monich, L. Thiele, and B. S. Manjunath,“Cortina: A System for Largescale, Content-based WebImage Retrieval,” Proc. ACM Multimedia, 2004.

[8] T. Seidl and H.-P. Kriegel, “Efficient User-AdaptableSimilarity Search in Large Multimedia Databases,” Proc.Int. Conf. Very Large Databases, 1997.

[9] G. Sheikholeslami, S. Chatterjee, and A. Zhang,“WaveCluster: A Multi-Resolution Clustering Approach forVery Large Spatial Databases,” Proc. Int. Conf. Very LargeDatabases, 1998.

[10] A. W. Smeulders, M. Worring, S. Santini, A. Gupta, andR. Jain, “Content-Based Image Retrieval at the End of theEarly Years,” IEEE Trans. Pattern Analysis and MachineIntelligence, 22(12):1349–1380, 2000.

[11] J.Z. Wang, J. Li, and G. Wiederhold, “SIMPLIcity:Semantics-Sensitive Integrated Matching for PictureLibraries,” IEEE Trans. Pattern Analysis and MachineIntelligence, 23(9), 947–963, 2001.

[12] J.-R. Wen, Q. Li, W.-Y. Ma, and H.-J. Zhang, “AMulti-paradigm Querying Approach for a GenericMultimedia Database Management System,” Proc.SIGMOD, 2003.

[13] H. J. Zhang, L. Wenyin, C. Hu, “iFind - A System forSemantics and Feature Based Image Retrieval overInternet,” Proc. ACM Multimedia, 2000.