Lecture 12Content-Based Image Retrieval

Slides by:Deepak Bote, Xiaoguang Feng,

David A. Forsyth, Clark F. Olson,Yossi Rubner, Linda G. Shapiro

2

Image databases

Large collections of image (and video) occur in many applications:

• Stock photos and footage• Military• World-wide web• Medical imaging• Space exploration• Surveillance• Many others…

3

Image databases

Most collections of images and videos are not image databases.

• No DBMS manages the data• No facility for complex queries is available

This is a rapidly expanding area of interest in computer vision:

• How can we find a particular image that we are interested in?• Can we locate images that meet some description?• How can we organize the collection meaningfully?• Can we extract new information by exploring the collection

automatically?

History

4

Image retrieval by annotation

5

Traditional text-based image search enginesManual annotation of images

Use text-based retrieval methods

E.g. Water lilies

Flowers in a pond

<Its biological name>

Limitations of annotation

Problem of image annotationLarge volumes of databases

Valid only for one language – with image retrieval this limitation should not exist

Problem of human perceptionSubjectivity of human perception

Too much responsibility on the end-user

Problem of deeper (abstract) needsQueries that cannot be described at all, but tap into the visual features of images

6

7

Image retrieval

Image have rich content that can be used for retrieval!

The problem that has been examined the most is: How can I find images that meet some description or are like an example?

This is often called “content-based image retrieval” (CBIR). Image content is used, rather than metadata.

Using example images:• Takes the responsibility of forming the query away from the user.• Allows each image to be described by its own features.

Mean color, histogram, etc.

Measures

No system is perfect. Usually measured using two criteria:• Recall: percentage of correct items found• Precision: percentage of items found that are correct

Which is more important?

8

Due to the imperfections in current methods, most systems have a user “in the loop”.

Retrieve-refine-retrieve cycle

User interaction

Y. Rubner, C. Tomasi, L.J. Guibas, “A metric for distributions with applications to image databases”, Intl. Conf. Computer Vision. © 1998, IEEE.

10

Features for retrieval

What features are used for retrieving images?

• Color

- Mean

- Overall distribution

- Relative locations• Texture

- Linear filters

- Textures of textures (!?)• Shape

- Sketches

- Segmented objects• Others

11

Color histograms

An early (and still popular) similarity measurement uses color histograms.• The RGB (or another) color

space is discretized into bins.• For each bin, a count is

maintained on the number of pixels that fall into the bin (since they have the right color)

Once constructed, the histograms can be compared using several metrics.

UC Berkeley Digital Library Project.http://elib.cs.berkeley.edu/

12

The QBIC system (IBM) was the first commercial system created.

It uses color, texture, shape, location, and keywords.

• h(I) is a K-bin histogram of a database image

• h(Q) is a K-bin histogram of the query image

• A is a K x K similarity matrix

Color histograms

The QBIC color histogram distance is:

dhist(I,Q) = (h(I) - h(Q)) A (h(I) - h(Q))T

QBIC – Search by color

13

** Images courtesy : Yong Rao

QBIC – Search by shape

14

** Images courtesy : Yong Rao

QBIC – Query by sketch

15

** Images courtesy : Yong Rao

Histogram distances

16

Histogram distances

17

Earth mover’s distance

18

Earth mover’s distance

19

20

Layout templates

Color histograms have no information about pixel locations. This yields false positives.

We can add spatial information by using a template of the image.

Example: Blue at top (sky), white below (snowy mountains), dark at bottom (mountains or lake)

Simplest implementation would divide image into regular blocks. P. Lipson, E. Grimson, P, Sinha, “Configuration

based scene classification and image indexing”, IEEE Conf. on Computer Vision and Pattern Recognition. © 1997 IEEE.

Color layout

21

** Image adapted from Smith and Chang : Single Color Extraction and Image Query

Color correlograms

22

Pick any pixel p1 of color Ci in the image I, at distance k away from p1 pick another pixel p2, what is the probability that p2 is also of color Ci?

P1

P2k

Red ?

Image: I

Jing Huang, S. Ravi Kumar, Mandar Mitra, Wei-Jing Zhu, and Ramin Zabih. Image indexing using color correlograms.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 762--768, 1997.

Color correlograms

23

The auto-correlogram of image I for color Ci , distance k:

Integrates both color and spatial information.

Efficient algorithms exist to compute this.

]||,|Pr[)( 2112)( kppIpIpI

iii CCk

C

Color correlograms

24

Jing Huang, S. Ravi Kumar, Mandar Mitra, Wei-Jing Zhu, and Ramin Zabih. Image indexing using color correlograms.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 762--768, 1997.

25

Retrieval with texture

Can compute local texture by:• Filtering with various kernels• Taking the signature at each pixel as the filter response at the pixel for each

kernel

How should texture signatures be used:• Histograms

- Image motion causes problems

- Can consider various motions separately, but more expensive

• Texture segmentation- Compare average texture signature within regions

Retrieval with texture

26

Yossi Rubner, Carlo Tomasi. Texture-Based Image Retrieval Without Segmentation. IEEE International Conference on Computer Vision, Kerkyra, Greece, September 1999, pages 1018-1024.

27

Retrieval with texture

Yossi Rubner, Carlo Tomasi. Texture-Based Image Retrieval Without Segmentation. IEEE International Conference on Computer Vision, Kerkyra, Greece, September 1999, pages 1018-1024.

28

K. Tieu, P. Viola, “Boosting image retrieval”, IEEE Conf. on Computer Vision and Pattern Recognition. © 2000 IEEE.

Positive examples (top) and retrieved results (bottom)

Texture of textures

Texture of textures

29

K. Tieu, P. Viola, “Boosting image retrieval”, IEEE Conf. on Computer Vision and Pattern Recognition. © 2000 IEEE.

Texture of textures

30

K. Tieu, P. Viola, “Boosting image retrieval”, IEEE Conf. on Computer Vision and Pattern Recognition. © 2000 IEEE.

The best filters to use for each class are learned using boosting.

Texture of textures

31

32

Retrieval using shape

We can do even better if we include shape in the query.

How do we know what shapes are in each image in the database?

• Segmentation

Blobworld (UC-Berkeley) uses color, texture, shape, and location to find good matches.

Background can be disregarded, if desired.

UC Berkeley Digital Library Project. http://elib.cs.berkeley.edu/

Shape measures

Global shape measures include:• Boundary length• Area enclosed• Boundary curvature

(overall or histogram)• Moments• Projections onto axes• Tangent angle histogram

However, global measures are not ideal in many situations.

33