Properties of Histograms and their Use for Recognition

Stathis Hadjidemetriou, Michael Grossberg,

Shree Nayar

Department of Computer Science

Columbia University

New York, NY 10027

Motivation

• Histogramming is a simple operation:

• Histograms have been used for:– Object recognition [Swain & Ballard 91, Stricker & Orengo 95]

– Indexing from visual databases [Bach et al, 96, Niblack et al 93, Zhang et al 95]

• Histogram advantages:– Efficient– Robust [Chatterjee, 96]

• Histogram limitation: – Do not represent spatial information

Motivation

Overview

•Image transformations that preserve the histogram

•Image structure through the multiresolution histogram

•Multiresolution histogram compared with other features

Invariance of Histogram with Discontinuous Transformations

Cut and rearrange regions

Shuffle pixels

Invariance of Histogram with Continuous Transformations

Rotation

Shear

What is the complete class of continuous transformations that

preserves the histogram?

Model for Image

Continuous domain Image: Map from continuous domain to intensities

Model for Histogram

U

Histogram count for bin U

≡Area bounded by level sets U

U

•Vector fields, X, morph images [Spivak, 65]:

Continuous Image Transformations

X

Gradient Transformations

FX

Original

5.122 )( yxF

40

)(sin

22 yxF

xyF

yxF 2

20

)(sin.

.20

)(sin

yx

yxF

Histograms of Gradient Transformations

Condition 1: Histogram Preservation and Local Area

Histogram preservedLocal area preserved

[Hadjidemetriou et al, 01]

T

……

……

T

Condition 2: Local Area Preservation and Divergence

•Divergence is rate of area change per unit area

Local area preserveddivergence is zero [Arnold, 89]

Small region

•Fields along isovalue contours of an energy function F

Isovalue contours

)sin(xyF

Hamiltonian Fields

•Flow of incompressible fluids [Arnold, 89]

Hamiltonian flow

Computing Hamiltonian Fields

22 yxF Gradient of

1. Compute gradient of F 2. Rotate gradient pointwise 900

jx

Fi

y

FFR

)(90

22 yxF Hamiltonian of

090R

Transformations preserve histogram of all images corresponding field is Hamiltonian

[Hadjidemetriou et al, CVPR, 00, Hadjidemetriou et al, IJCV, 01]

Theorem

Condition 3: Divergence and Hamiltonian Fields

Divergence of field is zero Hamiltonian field [Arnold, 89]

0

jx

Fi

y

Fdiv

Examples of Hamiltonian TransformationsLinear: Translations, rotations, shears

Original 3xF

7.022 )( yxF 5.122 )( yxF 10

)(sin

yxF

yxF 2

Examples of Hamiltonian Transformations

4

)(sin

4

)(sin

yxyxF

40

)(sin

22 yxF

65

coshyx

F

)(3 2244 yxyxF

Border Preserving Hamiltonian Transformations

0border

F

~

FFF win0|

,0|

borderwin

borderwin

F

F

2)2/(

,

2

1wx

xwin eF

0 w

h

Examples of Windowed Hamiltonian Transformations

yxF ~

2~

yF 3~

xF 24~

3xxF )3(

324

24~

yy

xxF

xyF ~

10

)(sin

~ yxF

22

~

yxF 5.122~

yxF

20

sin

.20

sin~

yx

yxF

Examples of Windowed Hamiltonian Transformations

Identical histograms:

4

sin

.4

sin~

yx

yxF

65

cosh~ yxF

yxF 2

~

Weak Perspective Projection

•Depth (z) causes scaling

cos2

2

z

fmw

[Hadjidemetriou et al, 01]

•Planar object tilt causes shearing and scaling

The Hamiltonian transformations is the complete class of continuous image transformations that preserves the

histogram

How can spatial information be embedded into the histogram?

Previous work on Features combining the Histogram with Spatial Information

•Local statistics:−Local histograms [Hsu et al, 95, Smith & Chang, 96, Koenderink and Doorn, 99, Griffin, 97]

−Intensity patterns [Haralick,79, Huang et al, 97]

•One histogram: −Derivative filters [Schiele and Crowley, 00, Mel 97]

−Gaussian filter [Lee and Dickinson, 94]

•Many techniques are ad-hoc or not complete

Multiresolution HistogramG(l2)

Limitations of HistogramsDatabase of synthetic images with identical histograms

[Hadjidemetriou et al, 01]

Matching with Multiresolution Histograms

Match under Gaussian noise of st.dev. 15 graylevels:

Matching with Multiresolution Histograms Match under Gaussian noise of st.dev. 15 graylevels:

How is Image Structure Encoded in the Multiresolution Histogram?

?

Image structure

Differences of histogramsdl

l))(*(d GLh

LImage

h(L*G(l))Multiresolution histogram

Histogram Change with Resolution and Spatial Information

•Bin j: dl

lhdj

))(*( GL

Spatial information

•Averages of bins:

where Pj are proportionality factorsdl

ldh jm

jj

))(*(

1

GLP

ill-conditioned

well-conditioned

Histogram Change with Resolution and Fisher Information Measures

qJ= Generalized Fisher

information measures of order q [Stam, 59, Plastino et al, 97]

≡

)(LqJ L is the image

xdq

D

22

LLL

=

D is the image domain

dl

ldh jm

jqj

))(*(

1

GLP

Averages

Image Structure Through Fisher Information Measures

?

Image structure

Differences of histogramsdl

l))(*(d GLh

LImage

h(L*G(l))Multiresolution histogram

Fisher information measures (Analysis)

P

Jq

Shape Boundary and Multiresolution Histogram15.0)( yxR LSuperquadrics:

=0.56

Histogram change with l is higher for complex boundary

=1.00 =1.48 =2.00

=6.67

Texel Repetition and Multiresolution Histogram

,, pTiling

Histogram change with l is proportional to number of texels (analytically)

Texel Placement and Multiresolution Histogram

Std. dev. of perturbation

Histogram change with l decreases with randomness

Matching Algorithm for Multiresolution Histograms

Burt-Adelson image pyramid

Cumulative histograms

L1 norm

Differences of histograms betweenconsecutive image resolutions

Concatenate to form feature vector

Histogram Parameters

•Bin width

•Smoothing to avoid aliasing

•Normalization:−Image size−Histogram size

179x179

89x89

44x44

5x5

……

Database of Synthetic Images

108 images with identical histograms [Hadjidemetriou et al, 01]

Sensitivity of Matching for Synthetic Images

Database of Brodatz Textures91 images with identical equalized histograms: 13 textures

different rotations

Match Results for Brodatz Textures

Match under Gaussian noise of st.dev. 15 graylevels:

Sensitivity of Class Matching for Brodatz Textures

Database of CUReT Textures 8,046 images with identical equalized histograms : 61

materials under different illuminations [Dana et al, 99]

Match Results for CUReT Textures

Match under Gaussian noise of st.dev. 15 graylevels:

Match Results for CUReT Textures

Match under Gaussian noise of st.dev. 15 graylevels:

Sensitivity of Class Matching for CUReT Textures

100 randomly selected images per noise level

Embed spatial information into the histogram with the multiresolution histogram

How well does the multiresolution histogram perform compared to other image features?

Comparison of Multiresolution Histogram with Other Features

•Multiresolution histogram:−Variable bin width−Histogram smoothing

•Fourier power spectrum annuli [Bajsky, 73]

•Gabor features [Farrokhnia & Jain, 91]

•Daubechies wavelet packets energies [Laine & Fan, 93]

•Auto-cooccurrence matrix [Haralick, 92]

•Markov random field parameters [Lee & Lee, 96]

Comparison of Effects of Transformations on the Features

Feature Translation RotationUniform Scaling

1Fourier power

spectrum annuliinvariant robust equivariant

2 Gabor features invariant sensitive equivariant

3Daubechies wavelet

energiessensitive sensitive sensitive

4Multiresolution

histogramsinvariant invariant equivariant

5Auto-cooccurrence

matrixinvariant robust equivariant

6Markov random field

parametersinvariant sensitive sensitive

Comparison of Class Matching Sensitivity of Features

Database of Brodatz textures

Comparison of Class Matching Sensitivity of Features

•Database ofCUReT textures•100 randomly selected images per noise level

Sensitivity of Features to Matching

FeatureGaussian

NoiseDatabase

size,# classesIlluminati-

on Parameter selection

Fourier power spectrum annuli

sensitive sensitive robust very sensitive

Gabor features robust robust robust sensitive

Daubechies wavelet energies

sensitive robust robust robust

Multiresolution histogram

robust robust robust robust

Auto-cooccurrence matrix

very sensitive very sensitive very sensitive very sensitive

Markov random field parameters

very sensitive very sensitive sensitive N/A

Comparison of Computation Costs of Features

1Markov random field

parametersO(n(2-1)2-(2-1)3/3)

2 Gabor features ( (logn+1)nlogn)

3Fourier power spectrum

annuliO(n3/2)

4 Auto-cooccurrence matrix O(n)

5 Wavelet packets energies O(nl)

6 Multiresolution histograms nn- number of pixels- window widthl- resolution levels

Decreasing cost

The multiresolution histogram compared to other image features is robust and efficient

Summary and Discussion

•Hamiltonian transformations preserve features based on:−Histogram−Image topology

•Multiresolution histograms:−Embed spatial information

•Comparison of multiresolution histograms with other features:−Efficient and robust

Recognition of 3D Matte Polyhedral Objects

•Face histograms:–Magnitude scaled by tilt angle ()

–Intensity scaled by illumination (ai )

•In an object database find [Hadjidemetriou et al, 00]:–Object identity

–Pose ()–Illumination (ai )

•Total histogram: Sum of h(i) of visible faces

A Simple Experiment

Object 1: Object 2:

Object 3: Object 4:

Object Tests Rank=1 Rank=2

Total 40 38 2

Shape Elongation and Multiresolution Histogram

Elongation:

y

x

St. dev. along axes: x, y.

•Gaussian:Sides of base : rx, ry.

y

x

r

r

•Pyramid:

Elongation:

1

(analytically)

Histogram change with l

Are all image resolutions equally significant?

Resolution Selection with Entropy of Multiresolution Histograms

•Entropy-resolution plot [Hadjidemetriou

et al, ECCV, 02] :–Global–Non-monotonic

….

l

…………………………………

Examples of Entropy-Resolution Plots

The entropy of the multiresolution histogram can be used to detect significant image resolutions

Future Work

•Histogram preserving fields:−Transformations over limited regions−Sensitivity of features to image transformation

•Multiresolution histograms:−Color images−Rotational variance with elliptic Gaussians

•Resolution selection:−Preprocessing step−Non-monotonic features