Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Lecture 6 - !!!
Fei-Fei Li!
Lecture 6: Finding Features (part 1/2)
Professor Fei-‐Fei Li Stanford Vision Lab
2-‐Oct-‐14 1
Lecture 6 - !!!
Fei-Fei Li!
What we will learn today?
• Local invariant features – MoHvaHon – Requirements, invariances
• Keypoint localizaHon – Harris corner detector
• Scale invariant region selecHon – AutomaHc scale selecHon – Difference-‐of-‐Gaussian (DoG) detector
• SIFT: an image region descriptor
2-‐Oct-‐14 2
Next lecture (#7)
Lecture 6 - !!!
Fei-Fei Li!
What we will learn today?
• Local invariant features – MoHvaHon – Requirements, invariances
• Keypoint localizaHon – Harris corner detector
• Scale invariant region selecHon – AutomaHc scale selecHon – Difference-‐of-‐Gaussian (DoG) detector
• SIFT: an image region descriptor
2-‐Oct-‐14 3
Some background reading: Rick Szeliski, Chapter 14.1.1; David Lowe, IJCV 2004
Lecture 6 - !!!
Fei-Fei Li!
Image matching: a challenging problem
2-‐Oct-‐14 4
Lecture 6 - !!!
Fei-Fei Li!
by Diva Sian
by swashford
Slid
e cr
edit
: St
eve
Seit
z
2-‐Oct-‐14 5
Image matching: a challenging problem
Lecture 6 - !!!
Fei-Fei Li!
Harder Case
by Diva Sian by scgbt
Slid
e cr
edit
: St
eve
Seit
z
2-‐Oct-‐14 6
Lecture 6 - !!!
Fei-Fei Li!
Harder SHll?
NASA Mars Rover images
Slid
e cr
edit
: St
eve
Seit
z
2-‐Oct-‐14 7
Lecture 6 - !!!
Fei-Fei Li!
Answer Below (Look for Hny colored squares)
NASA Mars Rover images with SIFT feature matches (Figure by Noah Snavely) Sl
ide
cred
it:
Stev
e Se
itz
2-‐Oct-‐14 8
Lecture 6 - !!!
Fei-Fei Li!
MoHvaHon for using local features • Global representaHons have major limitaHons • Instead, describe and match only local regions • Increased robustness to
– Occlusions
– ArHculaHon
– Intra-‐category variaHons
θqφ
dq
φ
θ
d
2-‐Oct-‐14 9
Lecture 6 - !!!
Fei-Fei Li!
General Approach N
pix
els
N pixels
Similarity measure Af
e.g. color
Bf
e.g. color
B1 B2
B3 A1
A2 A3
Tffd BA <),(
1. Find a set of distinctive key- points
3. Extract and normalize the region content
2. Define a region around each keypoint
4. Compute a local descriptor from the normalized region
5. Match local descriptors
Slid
e cr
edit
: Ba
stia
n Le
ibe
2-‐Oct-‐14 10
Lecture 6 - !!!
Fei-Fei Li!
Common Requirements • Problem 1:
– Detect the same point independently in both images
No chance to match!
We need a repeatable detector! Slid
e cr
edit
: D
arya
Fro
lova
, D
enis
Sim
akov
2-‐Oct-‐14 11
Lecture 6 - !!!
Fei-Fei Li!
Common Requirements • Problem 1:
– Detect the same point independently in both images
• Problem 2: – For each point correctly recognize the corresponding one
We need a reliable and distinctive descriptor!
?
Slid
e cr
edit
: D
arya
Fro
lova
, D
enis
Sim
akov
2-‐Oct-‐14 12
Lecture 6 - !!!
Fei-Fei Li!
Invariance: Geometric TransformaHons
Slid
e cr
edit
: St
eve
Seit
z
2-‐Oct-‐14 13
Lecture 6 - !!!
Fei-Fei Li!
Levels of Geometric Invariance
Slid
e cr
edit
: Ba
stia
n Le
ibe
2-‐Oct-‐14 14
CS131 CS231a
Lecture 6 - !!!
Fei-Fei Li!
Invariance: Photometric TransformaHons
• Ofen modeled as a linear transformaHon: – Scaling + Offset
Slid
e cr
edit
: Ti
nne
Tuyt
elaa
rs
2-‐Oct-‐14 15
Lecture 6 - !!!
Fei-Fei Li!
Requirements • Region extracHon needs to be repeatable and accurate
– Invariant to translaHon, rotaHon, scale changes – Robust or covariant to out-‐of-‐plane (≈affine) transformaHons – Robust to lighHng variaHons, noise, blur, quanHzaHon
• Locality: Features are local, therefore robust to occlusion and cluier.
• QuanHty: We need a sufficient number of regions to cover the object.
• DisHncHveness: The regions should contain “interesHng” structure.
• Efficiency: Close to real-‐Hme performance.
Slid
e cr
edit
: Ba
stia
n Le
ibe
2-‐Oct-‐14 16
Lecture 6 - !!!
Fei-Fei Li!
Many ExisHng Detectors Available
• Hessian & Harris [Beaudet ‘78], [Harris ‘88] • Laplacian, DoG [Lindeberg ‘98], [Lowe ‘99] • Harris-‐/Hessian-‐Laplace [Mikolajczyk & Schmid ‘01] • Harris-‐/Hessian-‐Affine [Mikolajczyk & Schmid ‘04] • EBR and IBR [Tuytelaars & Van Gool ‘04] • MSER [Matas ‘02] • Salient Regions [Kadir & Brady ‘01] • Others…
• Those detectors have become a basic building block for many recent applica8ons in Computer Vision.
Slid
e cr
edit
: Ba
stia
n Le
ibe
2-‐Oct-‐14 17
Lecture 6 - !!!
Fei-Fei Li!
Keypoint LocalizaHon
• Goals:
– Repeatable detecHon – Precise localizaHon – InteresHng content
⇒ Look for two-‐dimensional signal changes
Slid
e cr
edit
: Ba
stia
n Le
ibe
2-‐Oct-‐14 18
Lecture 6 - !!!
Fei-Fei Li!
Finding Corners
• Key property: – In the region around a corner, image gradient has two or more dominant direcHons
• Corners are repeatable and dis8nc8ve
C.Harris and M.Stephens. "A Combined Corner and Edge Detector.“ Proceedings of the 4th Alvey Vision Conference, 1988.
Slid
e cr
edit
: Sv
etla
na L
azeb
nik
2-‐Oct-‐14 19
Lecture 6 - !!!
Fei-Fei Li!
Corners as DisHncHve Interest Points • Design criteria
– We should easily recognize the point by looking through a small window (locality)
– Shifing the window in any direc8on should give a large change in intensity (good localiza8on)
“edge”: no change along the edge direction
“corner”: significant change in all directions
“flat” region: no change in all directions Sl
ide
cred
it:
Alyo
sha
Efro
s
2-‐Oct-‐14 20
Lecture 6 - !!!
Fei-Fei Li!
Harris Detector FormulaHon
• Change of intensity for the shif [u,v]: [ ]2
,( , ) ( , ) ( , ) ( , )
x yE u v w x y I x u y v I x y= + + −∑
Intensity Shifted intensity
Window function
or Window function w(x,y) =
Gaussian 1 in window, 0 outside
Slid
e cr
edit
: Ri
ck S
zelis
ki
2-‐Oct-‐14 21
Lecture 6 - !!!
Fei-Fei Li!
Harris Detector FormulaHon • This measure of change can be approximated by:
where M is a 2×2 matrix computed from image derivaHves:
⎥⎦
⎤⎢⎣
⎡≈
vu
MvuvuE ][),(
M
Sum over image region – the area we are checking for corner
Gradient with respect to x, times gradient with respect to y
2
2,( , ) x x y
x y x y y
I I IM w x y
I I I⎡ ⎤
= ⎢ ⎥⎢ ⎥⎣ ⎦
∑
Slid
e cr
edit
: Ri
ck S
zelis
ki
2-‐Oct-‐14 22
Lecture 6 - !!!
Fei-Fei Li!
Harris Detector FormulaHon
Ix Image I IxIy Iy
where M is a 2×2 matrix computed from image derivaHves:
M
Sum over image region – the area we are checking for corner
Gradient with respect to x, times gradient with respect to y
2
2,( , ) x x y
x y x y y
I I IM w x y
I I I⎡ ⎤
= ⎢ ⎥⎢ ⎥⎣ ⎦
∑
Slid
e cr
edit
: Ri
ck S
zelis
ki
2-‐Oct-‐14 23
Lecture 6 - !!!
Fei-Fei Li!
What Does This Matrix Reveal? • First, let’s consider an axis-‐aligned corner:
• This means: – Dominant gradient direcHons align with x or y axis – If either λ is close to 0, then this is not a corner, so look for locaHons where both are large.
• What if we have a corner that is not aligned with the image axes?
⎥⎦
⎤⎢⎣
⎡=
⎥⎥⎦
⎤
⎢⎢⎣
⎡=
∑∑∑∑
2
12
2
00λ
λ
yyx
yxx
IIIIII
M
Slid
e cr
edit
: D
avid
Jac
obs
2-‐Oct-‐14 24
Lecture 6 - !!!
Fei-Fei Li!
What Does This Matrix Reveal? • First, let’s consider an axis-‐aligned corner:
• This means: – Dominant gradient direcHons align with x or y axis – If either λ is close to 0, then this is not a corner, so look for locaHons where both are large.
• What if we have a corner that is not aligned with the image axes?
⎥⎦
⎤⎢⎣
⎡=
⎥⎥⎦
⎤
⎢⎢⎣
⎡=
∑∑∑∑
2
12
2
00λ
λ
yyx
yxx
IIIIII
M
Slid
e cr
edit
: D
avid
Jac
obs
2-‐Oct-‐14 25
Lecture 6 - !!!
Fei-Fei Li!
General Case
• Since M is symmetric, we have
• We can visualize M as an ellipse with axis lengths determined by the eigenvalues and orientaHon determined by R
RRM ⎥⎦
⎤⎢⎣
⎡= −
2
11
00λ
λ
Direction of the slowest change
Direction of the fastest change
(λmax)-1/2
(λmin)-1/2
adap
ted
from
Dar
ya F
rolo
va,
Den
is S
imak
ov
(Eigenvalue decomposition)
2-‐Oct-‐14 26
Lecture 6 - !!!
Fei-Fei Li!
InterpreHng the Eigenvalues • ClassificaHon of image points using eigenvalues of M:
λ1
“Corner” λ1 and λ2 are large, λ1 ~ λ2; E increases in all direcHons
λ1 and λ2 are small; E is almost constant in all direcHons “Edge”
λ1 >> λ2
“Edge” λ2 >> λ1
“Flat” region
Slid
e cr
edit
: Kr
iste
n G
raum
an
λ2
2-‐Oct-‐14 27
Lecture 6 - !!!
Fei-Fei Li!
Corner Response FuncHon
• Fast approximaHon – Avoid compuHng the eigenvalues
– α: constant (0.04 to 0.06)
λ2
“Corner” θ > 0
“Edge” θ < 0
“Edge” θ < 0
“Flat” region
θ = det(M )−α trace(M )2 = λ1λ2 −α(λ1 +λ2 )2
λ1 Slid
e cr
edit
: Kr
iste
n G
raum
an
2-‐Oct-‐14 28
Lecture 6 - !!!
Fei-Fei Li!
Window FuncHon w(x,y)
• OpHon 1: uniform window – Sum over square window
– Problem: not rotaHon invariant
• OpHon 2: Smooth with Gaussian – Gaussian already performs weighted sum
– Result is rotaHon invariant
1 in window, 0 outside
2
2,( , ) x x y
x y x y y
I I IM w x y
I I I⎡ ⎤
= ⎢ ⎥⎢ ⎥⎣ ⎦
∑
Gaussian
2
2,
x x y
x y x y y
I I IM
I I I⎡ ⎤
= ⎢ ⎥⎢ ⎥⎣ ⎦
∑
2
2( ) x x y
x y y
I I IM g
I I Iσ
⎡ ⎤= ∗⎢ ⎥
⎢ ⎥⎣ ⎦
Slid
e cr
edit
: Ba
stia
n Le
ibe
2-‐Oct-‐14 29
Lecture 6 - !!!
Fei-Fei Li!
Summary: Harris Detector [Harris88] • Compute second moment matrix
(autocorrelaHon matrix) 1. Image derivatives
Ix Iy
2. Square of derivatives
Ix2 Iy2 IxIy
3. Gaussian filter g(σI) g(Ix2) g(Iy2) g(IxIy)
R
2
2
( ) ( )( , ) ( )
( ) ( )x D x y D
I D Ix y D y D
I I IM g
I I Iσ σ
σ σ σσ σ
⎡ ⎤= ∗⎢ ⎥
⎢ ⎥⎣ ⎦
2 2 2 2 2 2( ) ( ) [ ( )] [ ( ) ( )]x y x y x yg I g I g I I g I g Iα= − − +
θ = det[M (σ I ,σ D )]−α[trace(M (σ I ,σ D ))]2
4. Cornerness function – two strong eigenvalues
5. Perform non-maximum suppression
Slid
e cr
edit
: Kr
ysti
an M
ikol
ajcz
yk
2-‐Oct-‐14 30
Lecture 6 - !!!
Fei-Fei Li!
Harris Detector: Workflow
Slid
e ad
apte
d fr
om D
arya
Fro
lova
, D
enis
Sim
akov
2-‐Oct-‐14 31
Lecture 6 - !!!
Fei-Fei Li!
Harris Detector: Workflow -‐ computer corner responses θ
Slid
e ad
apte
d fr
om D
arya
Fro
lova
, D
enis
Sim
akov
2-‐Oct-‐14 32
Lecture 6 - !!!
Fei-Fei Li!
Slid
e ad
apte
d fr
om D
arya
Fro
lova
, D
enis
Sim
akov
Harris Detector: Workflow -‐ Take only the local maxima of θ, where θ>threshold
2-‐Oct-‐14 33
Lecture 6 - !!!
Fei-Fei Li!
Harris Detector: Workflow -‐ ResulHng Harris points
Slid
e ad
apte
d fr
om D
arya
Fro
lova
, D
enis
Sim
akov
2-‐Oct-‐14 34
Lecture 6 - !!!
Fei-Fei Li!
Harris Detector – Responses [Harris88]
Effect: A very precise corner detector.
Slid
e cr
edit
: Kr
ysti
an M
ikol
ajcz
yk
2-‐Oct-‐14 35
Lecture 6 - !!!
Fei-Fei Li!
Harris Detector – Responses [Harris88]
Slid
e cr
edit
: Kr
ysti
an M
ikol
ajcz
yk
2-‐Oct-‐14 36
Lecture 6 - !!!
Fei-Fei Li!
Harris Detector – Responses [Harris88]
Slid
e cr
edit
: Kr
iste
n G
raum
an
• Results are well suited for finding stereo correspondences
2-‐Oct-‐14 37
Lecture 6 - !!!
Fei-Fei Li!
Harris Detector: ProperHes
• TranslaHon invariance?
Slid
e cr
edit
: Kr
iste
n G
raum
an
2-‐Oct-‐14 38
Lecture 6 - !!!
Fei-Fei Li!
Harris Detector: ProperHes
• TranslaHon invariance • RotaHon invariance?
Ellipse rotates but its shape (i.e. eigenvalues) remains the same
Corner response θ is invariant to image rotation
Slid
e cr
edit
: Kr
iste
n G
raum
an
2-‐Oct-‐14 39
Lecture 6 - !!!
Fei-Fei Li!
Harris Detector: ProperHes
• TranslaHon invariance • RotaHon invariance • Scale invariance?
Not invariant to image scale!
All points will be classified as edges!
Corner
Slid
e cr
edit
: Kr
iste
n G
raum
an
2-‐Oct-‐14 40
Lecture 6 - !!!
Fei-Fei Li!
What we have learned today?
• Local invariant features – MoHvaHon – Requirements, invariances
• Keypoint localizaHon – Harris corner detector
• Scale invariant region selecHon – AutomaHc scale selecHon – Difference-‐of-‐Gaussian (DoG) detector
• SIFT: an image region descriptor
2-‐Oct-‐14 41
Next lecture (#7)
Some background reading: Rick Szeliski, Chapter 14.1.1; David Lowe, IJCV 2004