Corners and Blobs - University of Ioanninacnikou/Courses/Image_Analysis/2010-2011/04_Corn… · • Corners are repeatable and distinctive. C.Harris and M.Stephens. "A Combined Corner

1

Image Analysis

Feature extraction: corners and blobs

University of Ioannina - Department of Computer Science

Christophoros [email protected]

Images taken from:Computer Vision course by Svetlana Lazebnik, University of North Carolina at Chapel Hill(http://www.cs.unc.edu/~lazebnik/spring10/).D. Forsyth and J. Ponce. Computer Vision: A Modern Approach, Prentice Hall, 2003.M. Nixon and A. Aguado. Feature Extraction and Image Processing. Academic Press 2010.

2 Corners and Blobs

C. Nikou – Image Analysis (T-14)

3 Motivation: Panorama Stitching


4Motivation: Panorama Stitching

(cont.)


Step 1: feature extractionStep 2: feature matching

5Motivation: Panorama Stitching

(cont.)


Step 1: feature extractionStep 2: feature matchingStep 3: image alignment

6 Characteristics of good features

Repeatability


• Repeatability– The same feature can be found in several images despite geometric

and photometric transformations.

• Saliency– Each feature has a distinctive description.

• Compactness and efficiency– Many fewer features than image pixels.

• Locality– A feature occupies a relatively small area of the image; robust to

clutter and occlusion.

2

7 Applications

• Feature points are used for:– Motion tracking.– Image alignment. – 3D reconstruction.

Obj t iti


– Object recognition.– Image indexing and retrieval.– Robot navigation.

8 Image Curvature

• Extends the notion of edges.• Rate of change in edge direction.• Points where the edge direction changes rapidly are

characterized as corners.• We need some elements from differential geometry.


g y• Parametric form of a planar curve:

( ) [ ( ), ( )]C t x t y t=

• Describes the points in a continuous curve as the endpoints of a position vector.

9 Image Curvature (cont.)

• The tangent vector describes changes in the position vector:

( ) ( ) [ ( ), ( )] ,dx dyT t C t x t y tdt dt

⎡ ⎤= = = ⎢ ⎥⎣ ⎦


dt dt⎣ ⎦

• Intuitive meaning– think of the trace of the curve as the motion

of a point at time t. – The tangent vector describes the

instantaneous motion.


• At any time moment, the point moves with velocity magnitude:

2 2( ) ( ) ( )C t x t y t= +


1 ( )( ) tan( )

y ttx t

ϕ − ⎛ ⎞= ⎜ ⎟

⎝ ⎠

in the direction:


• The curvature at a point C(t) describes the changes in the direction of the tangent with respect to changes in arc length:

( )d


( )( ) d ttdsϕκ =

• The curvature is given with respect to the arc length because a curve parameterized by the arc length maintains a constant velocity.


φ(t)

Gradient direction N(t)


Tangent T(t)Edge curve C(t)

( )( ) d ttdsϕκ =

3


• Parameterization of a planar curve by the arc length is the length of the curve from 0to t: ( )( )

t dC ts t dt= ∫


0( )

dt∫

• This implies:

2 2

0

( ) ( ) ( ) ( ) ( )tds t d dC t dC tdt x t y t

dt dt dt dt⎛ ⎞

= = = +⎜ ⎟⎝ ⎠∫


• Parameterization by the arc length is not unique but it has the property:

( ) ( )( )( ) ( )

dC t dC tdC td d d


( ) ( ) 1( )

dC t dC t dt dt dtdtds ds dC tds dt dsdt dt dt

= = = = =


• Back to the definition of curvature:

( ) ( )( ) d t d t dttds dt dsϕ ϕκ = =


1

2 2

1tan( ) ( )

d ydt x x t y t

−⎛ ⎞⎛ ⎞= ⎜ ⎟⎜ ⎟⎝ ⎠⎝ ⎠ +

3/22 2

( ) ( ) ( ) ( )( )( ) ( )

x t y t y t x ttx t y t

κ −⇔ =

⎡ ⎤+⎣ ⎦


• Some useful relations from differential geometry of planar curves (easily deduced) are:

( ) ( )dC t T tds

=2

2

( ) ( ) ( )dC t t N tds

κ=


ds ds

• The curvature is the magnitude of the second derivative of the curve with respect to the arc length:

2

2

( )( ) dC ttds

κ =


• It is also useful to express the normal vector at a point to a curve in Cartesian coordinates:

( ) ( )( ) ( ) [ ( ) ( )]y t x tN t T t x t y t⎡ ⎤−

= ⊥ =⎢ ⎥


2 2 2 2( ) , ( ) [ ( ), ( )]

( ) ( ) ( ) ( )N t T t x t y t

x t y t x t y t= ⊥ =⎢ ⎥⎢ + + ⎥⎣ ⎦


• There are three main approaches to compute curvature

– Direct computation on an edge image.– Derive the measure of curvature from image


intensities.– Correlation.

3/22 2

( ) ( ) ( ) ( )( )( ) ( )

x t y t y t x ttx t y t

κ −=

⎡ ⎤+⎣ ⎦

4


• Direct computation from edges.– Difference in edge direction.

( ) ( 1) ( 1)t t tκ ϕ ϕ= + − −


• Connected edges are needed (hysteresis).

with1 ( 1) ( 1)( ) tan

( 1) ( 1)y t y ttx t x t

ϕ − ⎛ ⎞+ − −= ⎜ ⎟+ − −⎝ ⎠


• Direct computation from edges.– Smoothing is generally required by considering more

than two pixels:

1( ) ( 1 ) ( 1)n

i i∑


1( ) ( 1 ) ( 1)

it t i t i

nκ ϕ ϕ

=

= + + − + −∑– Not very reliable results.

– Reformulation of a first order edge detection scheme.

– Quantization errors in angle measurement.– Threshold to detect corners.


• Direct computation from edges.


Object silhouette Thresholded curvature

• The result depends strongly on the threshold.


• Computation from image intensity.– It should be computed along the curve (normal to the

image gradient) for each pixel.– Cartesian coordinates for the angle of the tangent φ(x,y).


– Measure of angular changes in the image with respect to location φ’(x,y).

– The curve at an image point may be approximated by:

( )( )

( ) cos ( , )

( ) sin ( , )

x t x t x y

y t y t x y

ϕ

ϕ

′= +

′= +


• Computation from image intensity.– The curvature is given by:

( , ) ( , ) ( ) ( , ) ( )( , ) x y x y x t x y y tx yt x t y tϕ

ϕ ϕ ϕκ ′

′ ′ ′∂ ∂ ∂ ∂ ∂= = +

∂ ∂ ∂ ∂ ∂


t x t y t∂ ∂ ∂ ∂ ∂with

( ) ( )

1

( ) ( )cos ( , ) , sin ( , ) ,

( , ) tan x

y

x t y tx y x yt t

Mx yM

ϕ ϕ

ϕ −

∂ ∂′ ′= =∂ ∂

⎛ ⎞′ = −⎜ ⎟⎜ ⎟

⎝ ⎠

Normal to the curve. Recall that this is associated with the Css(t).


• Computation from image intensity.– Substituting the all the terms:

( )2 2

3/22 2

1( , ) y yx xy x y x x y

x y

M MM Mx y M M M M M Mx x y yM M

ϕκ ′

∂ ∂⎧ ⎫∂ ∂= − + −⎨ ⎬∂ ∂ ∂ ∂⎩ ⎭+


( )x y

– Alternatively, by differentiating backwards:

( )2 2

3/22 2

1( , ) y yx xy x y x x y

x y


ϕκ ′−

∂ ∂⎧ ⎫∂ ∂= − − +⎨ ⎬∂ ∂ ∂ ∂⎩ ⎭+

5


• Computation from image intensity.– Other measures differentiate along the normal to the

curve.– The idea is that curves may be thicker than one pixel

id


wide.– Differentiating along the normal measures the difference

between internal and external gradient angles.– Theoretically, these are equal. However, in practice they

differ due to image discretization.– The more the edge is bent, the larger the difference

(Kass et al. IJCV 1988).


• Computation from image intensity.– The measures are:

( )2

3/22 2

1( , ) y y y xx x y x y x y

x y

M M M Mx y M M M M M M Mx x y yM M

ϕκ ′⊥

∂ ∂ ∂⎧ ⎫∂= − − +⎨ ⎬∂ ∂ ∂ ∂⎩ ⎭+


( )x y

( )2 2

3/22 2

1( , ) y yx xx x y x y y

x y


ϕκ ′−⊥

∂ ∂⎧ ⎫∂ ∂= − + − +⎨ ⎬∂ ∂ ∂ ∂⎩ ⎭+

27 Image Curvature (cont.)• Computation from image intensity.

ϕκ ′ ϕκ ′−


Better than direct computation but the results are not consistent.

ϕκ ′⊥ ϕκ ′−⊥

28Finding Corners – Image Curvature

by Correlation


• Key property: in the region around a corner, image gradient has two or more dominant directions.

• Corners are repeatable and distinctive.

C.Harris and M.Stephens. "A Combined Corner and Edge Detector.“Proceedings of the 4th Alvey Vision Conference: pages 147--151.

29

• We should easily recognize the point by looking through a small window.

• Shifting a window in any direction should give a large change in intensity.

The Basic Idea


“edge”:no change along the edge direction

“corner”:significant change in all directions

“flat” region:no change in all directions

Source: A. Efros

30 Harris Corner Detector

[ ]2

,( , ) ( , ) ( , ) ( , )

x yE u v w x y I x u y v I x y= + + −∑

Change in appearance for the shift [u,v]:

h f dd


IntensityShifted intensity

Window function

orWindow function w(x,y)

Gaussian1 in window, 0 outside

Source: R. Szeliski

6

31 Harris Detector (cont.)

[ ]2

,

( , ) ( , ) ( , ) ( , )x y

E u v w x y I x u y v I x y= + + −∑

Change in appearance for the displacement [u,v]:


Second-order Taylor expansion of E(u,v) around (0,0):

⎥⎦

⎤⎢⎣

⎡⎥⎦

⎤⎢⎣

⎡+⎥

⎦

⎤⎢⎣

⎡+≈

vu

EEEE

vuEE

vuEvuEvvuv

uvuu

v

u

)0,0()0,0()0,0()0,0(

][21

)0,0()0,0(

][)0,0(),(

32 Harris Detector (cont.)The bilinear approximation simplifies to

where M is a 2×2 matrix computed from image derivatives:

⎥⎦

⎤⎢⎣

⎡≈

vu

MvuvuE ][),(


2

2,

( , ) x x y

x y x y y

I I IM w x y

I I I⎡ ⎤

= ⎢ ⎥⎢ ⎥⎣ ⎦

∑

where M is a 2×2 matrix computed from image derivatives:

M

33

The surface E(u,v) is locally approximated by a quadratic form. Let’s try to understand its shape.

Harris Detector (cont.)

⎥⎤

⎢⎡

≈u

MvuvuE ][),(


⎥⎦

⎢⎣v

MvuvuE ][),(

⎥⎥⎦

⎤

⎢⎢⎣

⎡=∑ 2

2

yyx

yxx

IIIIII

M

34

⎥⎤

⎢⎡

⎥⎤

⎢⎡

∑ 12 0λyxx III

M

First, consider the axis-aligned case where gradients are either horizontal or vertical.

Harris Detector (cont.)


⎥⎦

⎢⎣

=⎥⎥⎦⎢

⎢⎣

=∑2

12 0 λyyx

yxx

IIIM

If either λ is close to 0, then this is not a corner, so look for locations where both are large.

35 Harris Detector (cont.)

Since M is symmetric, it can be written as: RRM ⎥⎦

⎤⎢⎣

⎡= −

2

11

00λ

λ

We can visualize M as an ellipse with axes lengths determined by the eigenvalues and orientation determined by the rotation matrix R


by the rotation matrix R.

direction of the slowest change

direction of the fastest change

(λmax)-1/2

(λmin)-1/2const][ =⎥

⎦

⎤⎢⎣

⎡vu

Mvu

Ellipse equation:

36Visualization of second moment

matrices


7

37Visualization of second moment

matrices (cont.)


38 Window size matters!


39 Interpreting the eigenvalues

λ2

“Corner”λ1 and λ2 are large,λ λ

“Edge” λ2 >> λ1

Classification of image points using eigenvalues of M :

C. Nikou – Image Analysis (T-14)λ1

λ1 ~ λ2;E increases in all directions

λ1 and λ2 are small;E is almost constant in all directions

“Edge” λ1 >> λ2

“Flat” region

40 Corner response function

“Corner”R > 0

“Edge” R < 0

22121

2 )()(trace)det( λλαλλα +−=−= MMR

α: constant (0.04 to 0.06)


“Edge” R < 0

“Flat” region

|R| small

41 Harris detector: Steps

1. Compute Gaussian derivatives at each pixel.

2. Compute second moment matrix M in a Gaussian window around each pixel.


3. Compute corner response function R.4. Threshold R.5. Find local maxima of response function

(nonmaximum suppression).

42 Harris Detector: Steps (cont.)


8


Compute corner response R



Find points with large corner response: R>threshold



Take only the points of local maxima of R




47 Invariance

Features should be detected despite geometric or photometric changes in the image: if we have two transformed versions of the same image, features should be detected in corresponding locations.


48 Models of Image Transformation

• Geometric– Rotation

– Scale


– Affinevalid for: orthographic camera, locally planar object

• Photometric– Affine intensity change (I → a I + b)

9

49Harris Detector: Invariance

Properties• Rotation


The ellipse rotates but its shape (i.e. eigenvalues) remains the same.

Corner response R is invariant to image rotation


Properties (cont.)• Affine intensity change

• Only derivatives are used => invariance to intensity shift I → I + b• Intensity scale: I → a I


R

x (image coordinate)

threshold

R

x (image coordinate)

Partially invariant to affine intensity change


Properties (cont.)• Scaling


All points will be classified as edges

Corner

Not invariant to scaling


Properties (cont.)• Harris corners are not invariant to scaling.• This is due to the Gaussian derivatives

computed at a specific scale.• If the image differs in scale the corners


• If the image differs in scale the corners will be different.

• For scale invariance, it is necessary to detect features that can be reliably extracted under scale changes.

53 Scale-invariant feature detection

• Goal: independently detect corresponding regions in scaled versions of the same image.

• Need scale selection mechanism for finding characteristic region size that is covariant with the image transformation


the image transformation.• Idea: Given a key point in two images

determine if the surrounding neighborhoods contain the same structure up to scale.

• We could do this by sampling each image at a range of scales and perform comparisons at each pixel to find a match but it is impractical.

54Scale-invariant feature detection

(cont.)• Evaluate a signature function and plot the

result as a function of the scale.– The shape should be similar in different scales.


10


(cont.)• The only operator fulfilling these requirements

is a scale-normalized Gaussian.


T. Lindeberg. Scale space theory: a basic tool for analyzing structures at different scales. Journal of Applied Statistics, 21(2), pp. 224—270, 1994.


(cont.)• Based on the above idea, Lindeberg (1998)

proposed a detector for blob-like features that searches for scale space extrema of a scale-normalized LoG.


T. Lindeberg. Feature detection with automatic scale selection.International Journal of Computer Vision, 21(2), pp. 224—270, 1998.


(cont.)


58

f

d

Edge

Derivative

Recall: Edge Detection


gdxdf ∗

gdxd

Source: S. Seitz

Derivativeof Gaussian

Edge = maximumof derivative

59 Recall: Edge Detection (cont.)

f Edge

Second derivative


gdxdf 2

2

∗

gdxd

2

2 Second derivativeof Gaussian (Laplacian)

Edge = zero crossingof second derivative

Source: S. Seitz

60 From edges to blobs

• Edge = ripple.• Blob = superposition of two edges (two ripples).


Spatial selection: the magnitude of the Laplacian response will achieve a maximum at the center of the blob, provided the scale of the Laplacian is “matched” to the scale of the blob.

maximum

11

61 Scale selection• We want to find the characteristic scale of the blob by

convolving it with Laplacians at several scales and looking for the maximum response.

• However, Laplacian response decays as scale increases:


Why does this happen?

increasing σoriginal signal(radius=8)

62 Scale normalization

• The response of a derivative of Gaussian filter to a perfect step edge decreases as σincreases.

πσ 21


63 Scale normalization (cont.)

• The response of a derivative of Gaussian filter to a perfect step edge decreases as σincreases.

• To keep the response the same (scale-


invariant), we must multiply the Gaussian derivative by σ.

• The Laplacian is the second derivative of the Gaussian, so it must be multiplied by σ2.

64 Effect of scale normalizationUnnormalized Laplacian responseOriginal signal


Scale-normalized Laplacian response

maximum

65 Blob detection in 2D

• Laplacian: Circularly symmetric operator for blob detection in 2D.


2

2

2

22

yg

xgg

∂∂

+∂∂

=∇

66 Blob detection in 2D (cont.)

• Laplacian: Circularly symmetric operator for blob detection in 2D.


⎟⎟⎠

⎞⎜⎜⎝

⎛∂∂

+∂∂

=∇ 2

2

2

222

norm yg

xgg σScale-normalized:

12

67 Scale selection

• At what scale does the Laplacian achieve a maximum response for a binary circle of radius r?


r

image Laplacian

68 Scale selection (cont.)

• The 2D LoG is given (up to scale) by

2/

222 2/)(222 )2( σσ yxeyx +−−+

For a binary circle of radius r, the LoG achieves a maximum at


2/r=σ

r

2/rimage

Lapl

acia

n re

spon

se

scale (σ)

69 Characteristic scale

• We define the characteristic scale as the scale that produces a peak of the LoG response.


characteristic scaleT. Lindeberg (1998). "Feature detection with automatic scale selection."International Journal of Computer Vision, 30 (2): pp 79--116.

70 Scale-space blob detector

1. Convolve the image with scale-normalized LoG at several scales.

2. Find the maxima of squared LoG response in scale-space.


71 Scale-space blob detector: Example




13



74

• Approximating the LoG with a difference of Gaussians:

( )2 ( , , ) ( , , )xx yyL G x y G x yσ σ σ= +(Laplacian)

Efficient implementation


( , , ) ( , , )DoG G x y k G x yσ σ= −

( p )

(Difference of Gaussians)

• We have studied this topic in edge detection.

75 Efficient implementation

Divide each octave into an equal number K of intervals such that:

1/2 K nk kσ σ= =


D. G. Lowe. "Distinctive image features from scale-invariant keypoints.”International Journal of Computer Vision 60 (2), pp. 91-110, 2004.

02 ,1,..., .

nk kn K

σ σ= ==

Implementation by a Gaussian pyramid.

76 The Harris-Laplace detector

• It combines the Harris operator for corner-like structures with the scale space selection mechanism of DoG.

• Two scale spaces are built: one for the Harris corners and one for the blob detector (DoG)


corners and one for the blob detector (DoG).• A key point is a Harris corner with a

simultaneously maximun DoG at the same scale.• It provides fewer key points with respect to DoG

due to the constraint.

K. Mikolajczyk and C. Schmid, Scale and Affine invariant interest point detectors, International Journal of Computer Vision, 60(1):63-86, 2004.

77From scale invariance to affine

invariance


• For many problems, it is important to find features that are invariant under large viewpoint changes.

• The projective distortion may not be corrected locally due to the small number of pixels.

• A local affine approximation is usually sufficient.

78From scale invariance to affine

invariance (cont.)


• Affine adaptation of scale invariant detectors.• Find local regions where an ellipse can be reliably

and repeatedly extracted purely from local image properties.

14

79

•Recall: RRIIIIII

yxwMyyx

yxx

yx⎥⎦

⎤⎢⎣

⎡=

⎥⎥⎦

⎤

⎢⎢⎣

⎡= −∑

2

112

2

, 00

),(λ

λ

We can visualize M as an ellipse with axis lengths determined by the eigenvalues and orientation determined by R.

Affine Adaptation


direction of the slowest

change

direction of the fastest change

(λmax)-1/2

(λmin)-1/2

by the eigenvalues and orientation determined by R.

const][ =⎥⎦

⎤⎢⎣

⎡vu

Mvu

Ellipse equation:

80 Affine adaptation example


Scale-invariant regions (blobs)

81 Affine adaptation example


Affine-adapted blobs

82 Affine adaptation

• The “covarying” of Harris corner detector ellipse may be viewed as the “characteristic shape” of a region.

• We can normalize the region by transforming the ellipse into a unit circle


ellipse into a unit circle.

• The normalized regions may be detected under any affine transformation.

83 Affine adaptation (cont.)

• Problem: the second moment “window” determined by weights w(x,y) must match the characteristic shape of the region.

• Solution: iterative approach– Use a circular window to compute the second moment matrix.


– Based on the eigenvalues, perform affine adaptation to find an ellipse-shaped window.

– Recompute second moment matrix in the ellipse and iterate.

84 Iterative affine adaptation


K. Mikolajczyk and C. Schmid, Scale and Affine invariant interest point detectors, International Journal of Computer Vision, 60(1):63-86, 2004.

http://www.robots.ox.ac.uk/~vgg/research/affine/

15

85 Orientation ambiguity

• There is no unique transformation from an ellipse to a unit circle

– We can rotate or flip a unit circle, and it still stays a unit circle.


86 Orientation ambiguity (cont.)

• We have to assign a unique orientation to the keypoints in the circle:

– Create the histogram of local gradient directions in the patch.

– Assign tho the patch the orientation of the peak of the


g p psmoothed histogram.

0 2 π

87 Summary: Feature extraction

Extract affine regions Normalize regionsEliminate rotational

ambiguityCompute appearance

descriptors


SIFT (Lowe ’04)

88 Invariance vs. covariance

• Invariance: features(transform(image)) = features(image)

• Covariance: features(transform(image)) = transform(features(image))


Covariant detection => invariant description

Documents

Corners and Blobs - University of Ioanninacnikou/Courses/Image_Analysis/2010-2011/04_Corn… · • Corners are repeatable and distinctive. C.Harris and M.Stephens. "A Combined Corner