Common Visual Pattern Discovery via Spatially Coherent Correspondences

Hairong Liu, Shuicheng Yan

Learning & Vision Research Group, ECE, National University of Singapore

CVPR10

Hairong Liu

http://www.lv-nus.org/

Speaker:

Authors :

What is Common Visual Pattern?

Common Visual Pattern:A set of feature points share similar local features as well assimilar spatial layout

General Correspondence Problem

Schematic Illustration of Our ApproachSchematic Illustration of Our Approach

First, find all candidate correspondences by local featuresSecond, construct correspondence graphThird, obtain all dense subgraphsFourth, recover all correspondence configurations

(b)

T

G

(a)

T

x*

(c) (d)

Spatial coherence of two

correspondences

Candidate Correspondences

Two images with common visual

pattern

A large number of candidate

correspondences

Local feature is not enough, need spatial information

Dynamic Correspondence Graph

T

G

Spatial coherence of two

correspondences

l2

Each vertex represents a candidate correspondence

The weight of edge represents spatial coherence, that is:

Weight is a function of s, and common visual patterns correspond to dense subgraphs at correct scales

Spatial Coherency

Weights is the scale factor between two images

Graph DensityGraph Density

Graph density f(x) represents average affinity of a subgraph

m is unknown!

A is weighted adjacency

matrix

Graph Mode (Dense Subgraph)Graph Mode (Dense Subgraph)

f(x)Graph

Graph Modes

Graph Modes correspond to the peaks of graph density f(x)

Probabilistic CoordinateProbabilistic Coordinatex

xi represents the probability of cluster x contains vertex iri(x)=(Ax)i, the reward at vertex i, represents the affinity between vertex i and the cluster x

Graph Mode and Feature ModeGraph Mode and Feature Mode

Feature Mode Graph Mode

Similar Point: both reflect dense regions in dataDifferences:•In graph, no self-contribution•Graph mode is only the composite effect of points within the cluster, thus is inherently robust to noises and outliers

Properties of Graph ModeProperties of Graph Mode

x*

r = f(x*)

r > f(x*)r < f(x*)

x* is a graph mode

Lagrangian Function

KKT Conditions

SolutionSolution

Drop vertices to form very dense subgraph

Replicator Equation

Relation Space AnalysisRelation Space Analysis

Tessellate Relation space Run the procedure in parallel

Two Sampling StrategiesTwo Sampling Strategies

Each vertex is a sample (ICML10)

The sufficiently large neighborhood of each vertex forms a sample (CVPR10)

No sampling strategy can guarantee to find all

modes; but in applications, we can find all significant modes with

very high probability

Recover Correspondences

From each maximizer x*, we can recover a correspondence configuration

Large components have high priorities

Experiments & Results

TasksPoint Set Matching. (Randomly generated point

sets)Image Matching. (ETHZ toy images)Near-duplicate Image Retrieval. (Columbia

dataset)Goal

Robustness to noisesRobustness to large amount of outliersDifferent correspondence configurations (one to

one, one to many, many to many)

Exp-A: Point Set MatchingExp-A: Point Set Matching Our method is more robust to

noises

Our method is nearly not affected by

outliers

Test the robustness of all three methods to noises and outliers

Q

PAdd

noises

Add outlies to both point

sets

Rotation

Exp-A: Point Set Matching (cont)Exp-A: Point Set Matching (cont)

Detect similar point sets at different scales

Large peaks indicate correct

scales

Exp-B: Image MatchingExp-B: Image Matching

Different correspondence configurations, different scales

Our method can find all

correspondence configurations

Exp-C: Near-duplicate Image RetrievalExp-C: Near-duplicate Image Retrieval

Correspondences between near duplicate images

Exp-C: Near-duplicate Image Retrieval (cont)Exp-C: Near-duplicate Image Retrieval (cont)

150 near-duplicate pairs and 300 non-duplicate imagesAlthough simple, our method still get best result

Summary & Future work

Contributions in this workDynamic correspondence graphObtain all significant dense subgraphs by

systematically samplingFuture work

Better approximation methodBetter sampling strategiesAutomatically select scales

CodeCode

We extended our CVPR work (a very preliminary study) and ICML work, and proposed an efficient, non-parametric tool–Graph Shift, which can be thought as the counterpart of mean shift algorithm on graph . The code is available at:

http://www.lv-nus.org/GraphShiftCode.zipIf you can formulate your problem into the

problem of detecting dense subgraphs, it will be very helpful, especially for HUGE graph.

You can freely test and use it for academic purpose.