Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure

ter Haar Romeny, ICPR 2010

Introduction toScale-Space

and Deep Structure


Importance of Scale

Painting by Dali

• Objects exist at certain ranges of scale.

• It is not known a priory at what scale to look.


At the original scale of a dithered image we cannot calculate a derivative.We need to observe the image at a certain scale.

BLUR


Solution?

Look at all scales simultaneously

Scale

x

y

Scale Space


Scale Space in Human Vision

The human visual system is a

multi-scale sampling device

The retina contains receptive

fields; groups of receptors

assembled in such a way that they

form a set of apertures of widely

varying size.


Practical Implementation

Convolve the image with a Gaussian Kernel

2

221

2/22

)2(1),(

D

D

xx

exG


We can calculate derivatives and combinations of them at all scales

22yx LL

yyxx LL

Gradient Magnitude

Laplacian

Original Image


Main Topic

In this presentation we will show how

we can exploit the deep structure of

images to define invariant interest

points and features which can be used

for matching problems in computer

vision.

We consider only grey-value images.


Interest Points

The locations of particularly characteristic points

are called the interest points or key points.

These interest points have to be as invariant as

possible, but at the same time they have to carry

a lot of distinctive information.


Why interest points in scale-space?

Information in interest points is defined

by their neighborhood. But how big

should we choose this neighborhood?

• Let’s take the corners of the mouth as interest points.

• The red circles are the areas in which the information is gathered.

• If we make the picture bigger, the size of the neighborhood is too small.

• The neighborhood should scale with the image


When the interest points are detected in

scale space they do not only have spatial

coordinates x and y, but also a scale .

This scale tells us how big the

neighborhood should be.

Why interest points in scale-space?


Which interest points to use?

Our interest points have to be detected in scale space.They also have to…

…contain a lot of information…be reproducible…be stable…be well understood


We suggest Top-Points

The points we introduce have these

desired properties.


Critical Points, Paths and Top-Points

Maxima

Minimum

SaddlesL=0Critical Points


Critical Points, Paths and Top-Points

Maxima

Minimum

SaddlesL=0Critical Points

Det(H)=0Top-Points


Possible to calculate them for every function of the image L(x,y,)

Original Gradient Magnitude

Laplacian Det(H)


Detecting critical paths

Since for a critical path

L=0, intersection of

level surfaces Lx=0 with

Ly=0

will give the critical

paths.


Detecting Top-Points

Since for a top-point both L=0 and det[H]=Lxx Lyy-Lxy

2=0, we can find them by intersecting the paths with the level surface Det[H]=0



Original image

Top-points and features

Reconstruction


Metameric class

OriginalBy adjusting boundary and smoothness constraints we can improve the visual performance.

For this 300x300 picture 1000 top-points with 6 features were used.


Localization of Top-Points

For points close to top-points it is

possible to calculate a vector pointing

towards the position of the top-point.

x

y

Approximated Top-PointsDisplacement VectorsReal Locations


Stability of Top-Points

We can calculate the variance

of the displacement of top-

points under noise.

We need 4th order derivatives

in the top-points for that.



Thresholding on stability

Stable Paths Unstable Paths


Invariance of top-points

Top-points are invariant to certain transformations.

By invariant we mean that they move according to the

transformation.

Allo

wed

Tra

ns.


Differential invariants

We use the complete set

of irreducible 3rd order

differential invariants.

These features are

rotation and scaling

invariant.


The task

We have a scene and from

that scene we want to

retrieve the location of the

query object.


The top-points and differential invariants are

calculated for the query object and the scene.


Distance between feature vectors

A sensible distance between feature vectors is essential.

We have used Euclidean distance on ‘normalized’

differential invariants.

We tried Mahalanobis distance obtained from a training

set.


Similarity measure

We can calculate the propagation of noise in scale

space*.

This enables us to calculate a covariance matrix for

each feature vector.

The dissimilarity (“distance”) measure is expressed

as:

*Topological and Geometrical Aspects of Image Structure, Johan Blom


We now compare the

differential invariant features.

compare

distance = 0.5distance = 0.2distance = 0.3


The vectors with

the smallest

distance are paired.

smallest distance

distance = 0.2


A set of coordinates is

formed from the

differences in scale

(Log(o1)- Log(s2)) and in

angles (o1- s2).

(1, 1)


Dq

Important Clusters

For these clusters we calculate the mean and

Clustering (,)

If these coordinates are plotted

in a scatter plot clusters can be

identified.

• In this scatter plot we find two dense clusters


The stability criterion removes much of the scatter:


Rotate and scale according to the cluster means.


The translations we find correspond to the location of the objects in the scene.


In this example we have two clusters of correctly matched points.

C1

C2


The transformation of each object in the scene

matching to the query object is known from the

clustering.


We can transform the outline of the query object and project it on the scene image.


VideoGoogle?

Documents

Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure