Upload
others
View
11
Download
0
Embed Size (px)
Citation preview
Feature Point Detection
Lecture 07 Computer Vision
• Dr George Stockman
Professor Emeritus, Michigan State University
• Dr Mubarak Shah
Professor, University of Central Florida
• Dr David A. Forsyth
Professor, University of Illinois at Urbana- Champaign
• Dr Kristen Grauman
Associate Professor, University of Texas at Austin
Material Citations
Suggested Readings
• Chapter 4
Richard Szeliski , “Computer Vision: Algorithms and Applications”, Springer; 2011.
• Excerpt chapter from Synthesis lecture draft: Visual Recognition, Kristen Grauman and Bastian Leibe.
Find a bottle:
4
Categories Instances Find these two objects
Can’t do
unless you do not
care about few errors…
Can nail it
Motivation
M. Brown and D. G. Lowe. Recognizing Panoramas. ICCV 2003
• Example: Build a Panorama
How do we build panorama?
• We need to match (align) images
How do we build panorama?
• We need to match (align) images
Matching with Features
• Detect feature points in both images
Matching with Features
• Detect feature points in both images • Find corresponding pairs
Matching with Features
• Detect feature points in both images • Find corresponding pairs • Use these pairs to align images
Local features: main components
• Detection: Identify the interest points • Description: Extract feature vector descriptor surrounding each interest point • Matching: Determine correspondence between descriptors in two views
(1) (1)
1 1[ ,........ ]dX x x
(2) (2)
2 1[ ,........ ]dX x x
Goal: interest operator repeatability
• Detect the same point independently in both images
No chance to find true matches!
We need a repeatable detector
Goal: descriptor distinctiveness
• For each point correctly recognize the corresponding one
We need a reliable and distinctive descriptor
Must provide some invariance to geometric and photometric differences between the two views
Matching with Features
• Some patches can be localized or matched with higher accuracy than others
Matching with Features - More problems
• How to find landmarks to match across two images?
• How to distinguish one landmark from another?
• How to match N landmarks from image 1 to M landmarks from image 2, assuming that image 2 has more and/or less than image 1 ?
Image 1 Image 2
Matching with Features - More problems
• The mapping between the two image spaces may be nonlinear or complicated
• There may be errors in the locations of detected point or area features
• There may be extra features in one image or the other
• There may be missed features in one image or the other
Where can we use it further?
• Automate object tracking
• Point matching for computing disparity
• Stereo calibration - Estimation of fundamental matrix
• Motion based segmentation
• Recognition
• 3D object reconstruction
• Robot navigation
• Image retrieval and indexing…………etc.
What is an interest point?
• Expressive texture - The point at which the direction of the boundary of object changes abruptly - Intersection point between two or more edge segments
Synthetic & Real Interest Points
• Corners are indicated in red
How to determine landmarks?
• By unique intensity/color structure of Neighborhood
• By geometry or topology of neighborhood
Image 1 Image 2
Vectors show how intensity neighborhoods moved: similar vectors are colored the same
Properties of Interest Point Detectors
• Detect all (or most) true interest points
• No false interest points
• Well localized
• Robust with respect to noise
• Efficient detection
Possible Approaches to Corner Detection
• Based on brightness of images - Usually image derivatives
• Based on boundary extraction
- First step edge detection
- Curvature analysis of edges
Harris Corner Detector
• Corner point can be recognized in a window
• Shifting a window in any direction should give a large change in intensity
C.Harris, M.Stephens. “A Combined Corner and Edge Detector”. 1988
Harris Corner Detector - Basic Idea
Flat region: No change in all directions
Edge: No change along the edge direction
Corner: Significant change in all directions
Correlation
Correlation
Cross correlation
Auto correlation
Correlation Vs SSD
SSD: Subtract and minimize Correlation: Multiply and maximize
The local image energy is approximately constant, so it can be neglected
Constant, so it can be neglected
• Change of intensity for the shift [u,v]:
Intensity Shifted
intensity
Window function
or Window function w(x,y) =
Gaussian 1 in window, 0 outside
Harris Detector: Mathematics
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y
Taylor Series
Representation of a function as an infinite sum of terms that are calculated from the values of the function's derivatives at a single point
• Change of intensity for the shift [u,v]:
Harris Detector: Mathematics
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y
2
,
( , ) ( , ) ( ) ( , )x y
x y
E u v w x y I x y uI vI I x y
Window function
Shifted intensity
Intensity
Window function
Shifted intensity
Intensity
2
,
( , ) ( , ) x y
x y
E u v w x y uI vI
2
,
( , ) ( , )x
x y y
IE u v w x y u v
I
Taylor series
• Change of intensity for the shift [u,v]:
Harris Detector: Mathematics
,
( , ) ( , )x
x y
x y y
I uE u v w x y u v I I
I v
,
( , ) ( , )x
x y
x y y
I uE u v u v w x y I I
I v
( , ) ,u
E u v u v Mv
• For small shifts [u,v] we have a bilinear approximation:
• Where M is a 22 matrix computed from image derivatives:
Harris Detector: Mathematics
( , ) ,u
E u v u v Mv
2
2,
( , )x x y
x y x y y
I I IM w x y
I I I
E(u,v) is an equation of an Ellipse
M is covariance matrix
• Intensity change in shifting window: eigen value analysis
1, 2 – eigen values of M
direction of the slowest
change
direction of the fastest change
(max)-1/2
(min)-1/2
Ellipse E(u,v) = const
Harris Detector: Mathematics
( , ) ,u
E u v u v Mv
Eigen Vectors and Eigen Values
• The eigen vector, x, of a matrix A is a special vector, with the following property Where is called eigen value • To find eigen values of a matrix A first find the roots of: • Then solve the following linear system for each eigen value to find corresponding eigen vector
Ax x
det( ) 0A I
( ) 0A I x
Example
1 2 0
0 3 4
0 0 7
A
Eigen Values
1 2 37, 3, 1
1 2 3
1 1 1
4 , 2 , 0
4 0 0
X X X
Eigen Vectors
Eigen Values
1 2 3
det( ) 0
1 2 0 1 0 0
det 0 3 4 0 1 0 0
0 0 7 0 0 1
1 2 0
det 0 3 4 0
0 0 7
( 1 ) (3 )(7 ) 0 0
( 1 )(3 )(7 ) 0
1, 3, 7
A I
Eigen Vectors
1
2
3
1
2
3
1
( ) 0
1 2 0 0
0 3 4 0
0 0 7 0
0 2 0 0
0 4 4 0
0 0 8 0
1
0
0
A I x
x
x
x
x
x
x
X
2
2 3
3
1 2 3
0 2 0 0
0 4 4 0
0 0 8 0
1, 0, 0
x
x x
x
x x x
1 1
Eigen value
Eigen vector
• Intensity change in shifting window: eigen value analysis
1, 2 – eigen values of M
direction of the slowest
change
direction of the fastest change
(max)-1/2
(min)-1/2
Ellipse E(u,v) = const
Harris Detector: Mathematics
( , ) ,u
E u v u v Mv
1 and 2 are small; E is almost constant in all directions
Classification of image points using eigen values of M:
1
2
“Corner” 1 and 2 are large, 1 ~ 2; E increases in all directions
“Edge” 1 >> 2
“Edge” 2 >> 1
“Flat” region
Harris Detector: Mathematics
Matrix - Determinant and Trace
Determinant
Trace
(k – empirical constant, k = 0.04-0.06)
Harris Detector: Mathematics
2
det traceR M k M
1 2
1 2
det
trace
M
M
• Measure of corner response in terms of and : 21
2
1 2 1 2R k
1
2
0
0M
• R depends only on eigen values of M
• R is large for a corner
• R is negative with large magnitude for an edge • |R| is small for a flat region
Harris Detector: Mathematics
1
2
“Corner”
R > 0
“Edge”
“Edge” R < 0
“Flat”
|R| small R < 0
• Algorithm - Brief:
– Find points with large corner response function R
(R > threshold)
– Take the points of local maxima of R
Harris Detector
Harris Detector: Workflow
• Compute corner response R
Harris Detector: Workflow
• Find points with large corner response: R>threshold
Harris Detector: Workflow
• Take only the points of local maxima of R
Harris Detector: Workflow
Harris Detector: Workflow
• If pixel value is greater than its neighbors then it is a local maxima
1 1R
Other Version of Harris Detectors
1 2
1 2
det( )
( )
MR
trace M
1R
Triggs
Szeliski (Harmonic mean)
Shi-Tomasi
Other Version of Harris Detectors
1
2
• Algorithm - Detailed:
– Compute horizontal and vertical derivatives of image Ix and Iy
– Compute three images corresponding to three terms in matrix M
– Convolve these three images with a larger Gaussian (window)
– Compute scalar cornerness value using one of the R
– Find local maxima above some threshold as detected interest points
Harris Detector
• Average intensity change in direction [u,v] can be expressed as a bilinear form:
• Describe a point in terms of eigen values of M: measure of corner response
• A good (corner) point should have a large intensity change in all directions, i.e. R should be large positive
( , ) ,u
E u v u v Mv
2
1 2 1 2R k
Harris Detector: Summary
• Rotation invariance
• Ellipse rotates but its shape (i.e. eigen values) remains the same
Corner response R is invariant to image rotation
Harris Detector: Some Properties
• Partial invariance to affine intensity change
- Only derivatives are used => invariance to intensity shift I I + b
- Intensity scale: I a I
R
x (image coordinate)
threshold
R
x (image coordinate)
Harris Detector: Some Properties
• But: non-invariant to image scale!
All points will be classified as edges Corner !
Harris Detector: Some Properties
• Quality of Harris detector for different scale changes
• Repeatability rate:
# correspondences # possible correspondences
C. Schmid et.al. “Evaluation of Interest Point Detectors”. IJCV 2000
Harris Detector: Some Properties
• Geometry – Rotation
– Similarity (rotation + uniform scale)
– Affine (scale dependent on direction)
valid for: orthographic camera, locally planar object
• Photometry – Affine intensity change (I a I + b)
Model of Image Change
• Harris Corner Detector
C. Schmid et.al. “Evaluation of Interest Point Detectors”. IJCV 2000
Rotation Invariant Detection
Assignment 02 – Will be uploaded !!!