Image Similarity and the Earth Mover’s Distance Empirical Evaluation of Dissimilarity Measures for Color and Texture Y. Rubner, J. Puzicha, C. Tomasi and

Image Similarity and the Earth Mover’s Distance

Empirical Evaluation of Dissimilarity Measures for Color and TextureY. Rubner, J. Puzicha, C. Tomasi and T.M. Buhmann

The Earth Mover’s Distance as a Metric for Image Retrieval

Y. Rubner, C. Tomasi and J.J. Guibas

The Earth Mover’s Distance is the Mallows Distance: Some Insights from Statistics

E. Levina and P.J. Bickel

Learning-Based Methods in Vision - Spring 2007

Frederik Heger(with graphics from last year’s slides)

1 February 2007

2 LBMV Spring 2007 - Frederik Heger [email protected]

How Similar Are They?Images from Caltech 256


Similarity is Important for …

• Image classification• Is there a penguin in this picture?• This is a picture of a penguin.

• Image retrieval• Find pictures with a penguin in them.• Image as search query

• Find more images like this one.

• Image segmentation• Something that looked like this was called penguin before.


Space Shuttle Cargo Bay

Image Representations: Histograms

Normal histogram Cumulative histogram

•Generalize to arbitrary dimensions•Represent distribution of features

• Color, texture, depth, …

Images from Dave Kauchak



Joint histogram• Requires lots of data• Loss of resolution to

avoid empty bins


Marginal histogram• Requires independent features• More data/bin than

joint histogram




Adaptive binning• Better data/bin distribution, fewer empty bins• Can adapt available resolution to relative feature importance



EASE Truss Assembly



Clusters / Signatures• “super-adaptive” binning• Does not require discretization along any fixed axis



Distance Metrics

-

-

-

= Euclidian distance of 5 units

= Grayvalue distance of 50 values

= ?

x

y

x

y


Issue: How to Compare Histograms?

Bin-by-bin comparisonSensitive to bin size. Could use wider bins …

… but at a loss of resolution

Cross-bin comparisonHow much cross-bin influence is necessary/sufficient?


Overview: Similarity Measures

Heuristic Histogram Distance:

Minkowski-form distance (Lp)

Special Cases:

L1 Mahattan distance

L2 Euclidian Distance

L Maximum value distance



Heuristic Histogram Distance:Weighted-Mean-Variance (WMV)

Info:• Per-feature similarity measure• Based on Gabor filter image representation• Shown to outperform several parametric models

for texture-based image retrieval



Nonparametric Test Statistic:Kolmogorov-Smirnov distance (KS)

Info:• Defined for only one dimension• Maximum discrepancy between cumulative

distributions• Invariant to arbitrary monotonic feature

transformations



Nonparametric Test Statistic:Cramer/von Mises type statistic (CvM)

Info:• Squared Euclidian distance between distributions• Defined for single dimension



Nonparametric Test Statistic:

2

Info:• Very commonly used



Information-theory Divergence:Kullback-Leibler divergence (KL)

Info:• Code one histogram using the other as true

distribution• How inefficient would it be?• Also widely used.



Information-theory Divergence:Jeffrey-divergence (JD)

Info:• Similar to KL divergence• But symmetric and numerically stable



Ground Distance Measure:Quadratic Form (QF)

Info:• Heuristic approach• Matrix A incorporates cross-bin information



Ground Distance MeasureEarth Mover’s Distance (EMD)

Info:• Based on solution of linear optimization problem

(transportation problem)• Minimal cost to transform one distribution to the

other• Total cost = sum of costs for individual features


Summary: Similarity Measures


Earth Mover’s Distance

≠



≠



=



=

(amount moved) * (distance moved)


How EMD Works

All movements

(distance moved) * (amount moved)

(distance moved) * (amount moved)

* (amount moved)

n clusters

Q

P

m clusters


How EMD Works

Move earth only from P to Q

P’

Q’n clusters

Q

P

m clusters


How EMD Works

n clusters

Q

P

m clusters

P cannot send more earth than there is

P’

Q’


How EMD Works

n clusters

Q

P

m clusters

Q cannot receive more earth than it can hold

P’

Q’


How EMD Works

n clusters

Q

P

m clusters

As much earth as possiblemust be moved

P’

Q’


Color-based Image Retrieval

Jeffrey divergence

Quadratic form distance

Earth Mover Distance

χ2 statistics

L1 distance


Red Car Retrievals (Color-based)


Zebra Retrieval (Texture-based)


EMD with Position Encoding

without position

with position


Issues with EMD

• High computational complexity• Prohibitive for texture segmentation

• Features ordering needs to be known • Open eyes / closed eyes example

• Distance can be set by very few features.• E.g. with partial match of uneven distribution weight

EMD = 0, no matter how many features follow


Help From Statisticians

• For even-mass distributions, EMD is equivalent to Mallows distance• (for uneven mass distributions,

the two distances behave differently)• Trick to compute Mallows distance

• 1-D marginals give better classification results than joint distributions (experimental results)

• Get marginals from empirical distribution by sorting feature vectors


EMD Summary / Conclusions

• Ground distance metric for image similarity

• Uses signatures for best adaptive binning and to lessen impact of prohibitive complexity

• Can deal with partial matches

• Good performance for color/texture classification

• Statistical grounding


Last Slide

Comments?

Questions?

Documents

Image Similarity and the Earth Mover’s Distance Empirical Evaluation of Dissimilarity Measures for Color and Texture Y. Rubner, J. Puzicha, C. Tomasi and