Andrew Zisserman Talk - Part 1b

Demo

Example

Spatial re-ranking• improves precision

• but not recall …

Query images Prec.

Rec.

• high precision at low recall (like google)

• variation in performance over query

• none retrieve all instances

Obtaining visual words is like a sensor measuring the image

“noise” in the measurement process means that some visual words are missing or incorrect, e.g. due to

• Missed detections• Changes beyond built in invariance• Quantization effects

Consequence: Visual word in query is missing in target image

Why aren’t all objects retrieved?

Clustered and quantized to visual words

Clustered and quantized to visual words

sparse frequency vector

Set of SIFTdescriptorsquery image

Hessian-Affine regions + SIFT

descriptors

Hessian-Affine regions + SIFT

descriptors

1. Query expansion2. Better quantization

Query expansion

In text :• Reissue top N results as queries

• Pseudo/blind relevance feedback

• Danger of topic drift – this is a big problem for text

Query Expansion

Original query: Hubble Telescope Achievements

Example from: Jimmy Lin, University of Maryland

Query expansion: Select top 20 terms from top 20 documents according to tf-idf

Telescope, hubble, space, nasa, ultraviolet, shuttle, mirror, telescopes, earth, discovery, orbit, flaw, scientists, launch, stars, universe, mirrors, light, optical, species

Added terms:

Query Expansion: Text

In text :• Reissue top N results as queries

• Pseudo/blind relevance feedback

• Danger of topic drift – this is a big problem for text

In vision:• Reissue spatially verified image regions as queries

• Spatial verification like an oracle of truth

Query Expansion

Visual query expansion - overview

1. Original query

3. Spatial verification

4. New enhanced query

…

2. Initial retrieval set

5. Additional retrieved images

oracle

Query Image Originally retrieved image Originally not retrieved

What Query Expansion Adds

What Query Expansion Adds

What Query Expansion Adds

What Query Expansion Adds

Visual query expansion - overview

1. Original query

3. Spatial verification

4. New enhanced query

…

2. Initial retrieval set

5. Additional retrieved images

oracle

Bag of visual words particular object retrieval

Hessian-Affineregions + SIFT descriptors

visual words+tf-idf weighting

querying

sparse frequency vector

centroids(visual words)

Invertedfile

ranked imageshort-list

Set of SIFTdescriptorsquery image

[Lowe 04, Chum & al 2007]

Geometricverification

[Chum & al 2007]

Queryexpansion

Demo

Query image Originally retrieved Retrieved only after expansion

Query Expansion

Queryimage

Expanded results (better)

Original results (good)

Prec.

Prec.

Rec.

Rec.

Better Quantization

Problems arising from quantization

• Typically, quantization has a significant impact on the final performance of the system [Sivic03,Nister06,Philbin07]

• Quantization errors split features that should be grouped together and confuse features that should be separated

Voronoicells

And more …

i. Points 3 and 4 are close, but never matched

i. Points 1, 2 and 3 are matched equally

Overcoming quantization errors• Soft-assign each descriptor to multiple cluster centers• Assignment weight according to Gaussian on distance• Normalize weights to sum to one

[Philbin et al. CVPR 2008, Van Gemert et al. ECCV 2008]

A: 0.1B: 0.5C: 0.4

B: 1.0 Hard Assignment

Soft Assignment

Learning a vocabulary to overcome quantization errors[Mikulik et al. ECCV 2010, Philbin et al. ECCV 2010]

Several other solutions are possible …

e.g. Hamming embedding [Jegou&Schmid ECCV 2008]• Standard quantization using bag-of-visual-words• Additional localization in the Voronoi cell by a binary signature

• More on methods of soft assignment tomorrow

Soft Assignment: Implementation

Bag of words: score a match between two features by the scalar product of their weight vectors

Spatial re-ranking: also score the number of inliers using this measure

Soft Assignment: ResultsBenefit 1: Helping Query Expansion

Query

HardAssignment

Only one good initial result – QE doesn't significantly improve results

Soft Assignment: ResultsBenefit 1: Helping Query Expansion

Query

SoftAssignment

4 good results – allows query expansion to return these results

in addition to the ones above

Soft Assignment: ResultsBenefit 2: Better spatial localization

HardAssignment

SoftAssignment

Results: Baseline to State of the Art

1. Baseline Method K = 10K 0.389

2. Large Vocabulary K=1M 0.618

3. Spatial Re-ranking 0.653

4. Soft Assignment (SA) 0.731

5. Query Expansion (QE) 0.801

Mean Average Precision

6. SA & QE 0.825

Disadvantages of soft assignment?

Outline

1. Object recognition cast as nearest neighbour matching

2. Object recognition cast as text retrieval

3. Large scale search and improving performance

4. Applications

• accessing expert knowledge, data mining, inpainting, location search, large scale reconstruction, mobile apps, …

5. The future and challenges

Application

Accessing expert knowledge:

• Use an image query to access an annotated dataset

• Search with query image retrieve annotation

Visual Access to Classical Art Archives

Currently: 111 thousand Greek vase images

http://explore.clarosnet.org/XDB/ASP/clarosHome/

Application:

Object Mining in Large Datasets

Objective …

Automatically find and group images of same object/scene

Motivation

Applications:

• Dataset summarization

• Efficient retrieval

• Efficient pre-processing for automatic 3-D reconstruction (e.g. PhotoSynth)

Matching Graph

Build a ‘matching graph’ over all the images in the dataset

Each image is a node and a link represents two images having some object in common

Given this graph structure, apply various clustering algorithms to group the data

Finding Commonly Occurring Objects

Simple idea: strong spatial constraints gives a 'link' between two images

Edge strength = # inliers

Finding Commonly Occurring ObjectsUse these links to build up a graph over all images in the

dataset

Nodes = images, edges = spatially verified matches

Building the Matching Graph

• Use each image to query the dataset

• Each query gives a list of results scored by a measure of the spatial consistency to the query

• Threshold this consistency measure to determine the links in the matching graph

Connected Components

In a collection of images of multiple disjoint objects we expect the matching graph to also be disjoint

A simple first step is to take connected components of the matching graph and examine the clusters returned

Connected Components

Example: five connected components from the Oxford dataset

56 images 71 images 26 images 25 images 56 images

Connected ComponentsA problem with connected components is that ‘connecting images’ can sometimes join two disjoint objects

LinkingImages

Can overcome this problem by divide and merge strategy

Datasets

Statue of Liberty dataset (37,034 images)• Crawled from Flickr by querying for ‘statue of

liberty’• Lots of images of the Statue of Liberty but also of

New York and other sites

Rome dataset (1,021,986 images) [1]• Again, crawled from Flickr• Contains too much stuff to mention

[1] Photo tourism: Exploring photo collections in 3D, Noah Snavely, Steven M. Seitz, Richard Szeliski

Results: Statue of Liberty

Largest cluster – 8461 images of the Statue of Liberty

2nd largest – 276 aerial views of New York

Results: Statue of Liberty

3rd largest – 80 American flags

Results: Statue of Liberty

Results: Statue of Liberty

Smaller clusters

Lego Statue of Liberty 59 images

Staten Island 52 images

Results: Rome

18676 images

15818 images

9632 images

4869 images

Timings

21,339 high resolution images from Flickr tagged with 'statue of liberty'

Querying with every image in the database to build the graph takes ~2 hours

Finding connected components (v quick) using a threshold of 20 spatially verified inliers gives 11 clusters with more than 20 images

As an aside …Better matching with fewer features[Turcot & Lowe, ICCV Workshop 2009].

• Build matching graph • Augment image bag-of-word histograms using neighbours

• Like query expansion, but done in advance on the `server side’

Application: Internet-based inpaintingPhoto-editing using images of the same place[Whyte, Sivic and Zisserman, 2009], but see also [Hays and Efros, 2007].

Application: place recognition (retrieval in a structured (on a map) database)

[Knopp, Sivic, Pajdla, ECCV 2010] http://www.di.ens.fr/willow/research/confusers/

Query

Query Expansion(Panoramio, Flickr, … )

Best match

Image indexingwith spatial verification

Optimized image database

ConfuserSuppressionOnly negative training data

(from geotags)

Image database

17

Correctly recognized examples

More correctly recognized examples

Application: Matching and 3D reconstruction in large unstructured datasets.

Building Rome in a Day, SameerAgarwal, Noah Snavely, Ian Simon, Steven M. Seitz and Richard Szeliski,International Conference on Computer Vision, 2009http://grail.cs.washington.edu/rome/

See also [Havlena, Torrii, Knopp and Pajdla, CVPR 2009].Figure: N. Snavely

Example of the final 3D point cloud and cameras57,845 downloaded images, 11,868 registered images. This video: 4,619 images.

The Old City of Dubrovnik

Bing visual scan

Application: Mobile visual search apps

and others… Snaptell.com, Moodstocks.com

ExampleExample

Slide credit: I. Laptev

Sivic, J. and Zisserman, A.Video Google: A Text Retrieval Approach to Object Matching in VideosProceedings of the International Conference on Computer Vision (2003)http://www.robots.ox.ac.uk/~vgg/publications/papers/sivic03.pdf

Demo: http://www.robots.ox.ac.uk/~vgg/research/vgoogle/

Chum, O., Philbin, J., Isard, M., Sivic, J. and Zisserman, A.Total Recall: Automatic Query Expansion with a Generative Feature Model for Object RetrievalProceedings of the International Conference on Computer Vision (2007)http://www.robots.ox.ac.uk/~vgg/publications/papers/chum07b.pdf

Demo: http://www.robots.ox.ac.uk/~vgg/research/oxbuildings/

Philbin, J. and Zisserman, A.Object Mining using a Matching Graph on Very Large Image Collections Proc. of the Indian Conference on Vision, Graphics and Image Processing (2008)http://www.robots.ox.ac.uk/~vgg/publications/papers/philbin08b.pdf

Papers and Demos