Upload
colleen-lang
View
244
Download
5
Tags:
Embed Size (px)
Citation preview
Motion Estimation IWhat affects the induced image motion?
• Camera motion• Object motion• Scene structure
Example Flow Fields
• This lesson – estimation of general flow-fields• Next lesson – constrained by global transformations
The Aperture Problem
So how much information is there locally…?
The Aperture Problem
Copyright, 1996 © Dale Carnegie & Associates, Inc.
Not enough info in local regions
The Aperture Problem
Copyright, 1996 © Dale Carnegie & Associates, Inc.
Not enough info in local regions
The Aperture Problem
Copyright, 1996 © Dale Carnegie & Associates, Inc.
The Aperture Problem
Copyright, 1996 © Dale Carnegie & Associates, Inc.
Information is propagated from regions with high certainty (e.g., corners) to regions with low certainty.
Such info propagation can cause optical illusions…
Illusory corners
1. Gradient based methods (Horn &Schunk, Lucase & Kanade)
2. Region based methods (Correlation, SSD, Normalized correlation)
• Direct (intensity-based) Methods
• Feature-based Methods
Image J (taken at time t)
yx,
Brightness Constancy Assumption
yxJ , vyuxI ,
Image I(taken at time t+1)
vyux ,
Brightness Constancy Equation:
The Brightness Constancy Constraint
),(),( ),(),( yxyx vyuxIyxJ
),(),(),(),(),(),( yxvyxIyxuyxIyxIyxJ yx Linearizing (assuming small (u,v)):
),(),(),(),(),(),(0 yxJyxIyxvyxIyxuyxI yx
),(),(),(),(),( yxIyxvyxIyxuyxI tyx
),(),(
),(),( yxI
yxv
yxuyxI t
T
* One equation, 2 unknowns
* A line constraint in (u,v) space.
* Can recover Normal Flow.
Observations:
Need additional constraints…
Horn and Schunk (1981)
Add global smoothness term
2222
),(yxyx
yx
vvuu
Smoothness error
2
),(tyx
yx
IvIuIE
Error in brightness constancy equation
sc EE Minimize:
Solve by using calculus of variations
Horn and Schunk (1981)
Problems…
* Smoothness assumption wrong at motion/depth discontinuities over-smoothing of the flow field.
* How is Lambda determined…?
Lucas-Kanade (1984)
Windowyx
tyx IvyxIuyxIvuE),(
2),(),(),(
Assume a single displacement (u,v) for all pixels within a small window
Minimize E(u,v):
Geometrically -- Intersection of multiple line constraints
Algebraically --
Lucas-Kanade (1984)
Windowyx
tyx IvyxIuyxIvuE),(
2),(),(),(
Differentiating w.r.t u and v and equating to 0:
ty
tx
yyx
yxx
II
II
v
u
III
III2
2
tT IIUII
Solve for (u,v)[ Repeat this process for each and every pixel in the image ]
Minimize E(u,v):
Problems…
* Still smoothes motion discontinuities (but unlike Horn & Schunk, does not propagate error across the entire image)
* Singularities (partially solved by coarse-to-fine)
Lucas-Kanade (1984)
Singularites
2
2
yyx
yxx
III
III
Where in the image will this matrix be invertible and where not…?
Homework
Linearization approximation iterate & warp
xx0
Initial guess:
Estimate:
estimate update
xx0
estimate update
Initial guess:
Estimate:
Linearization approximation iterate & warp
xx0
Initial guess:
Estimate:
Initial guess:
Estimate:
estimate update
Linearization approximation iterate & warp
xx0
Linearization approximation iterate & warp
Revisiting the small motion assumption
Is this motion small enough?Probably not—it’s much larger than one pixel (2nd order terms
dominate)How might we solve this problem?
0 tyx IvIuI ==> small u and v ...
u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
image Iimage J
u
iterate refine
u
uΔ
+
Pyramid of image J Pyramid of image I
image Iimage J
Coarse-to-Fine EstimationAdvantages: (i) Larger displacements. (ii) Speedup. (iii) Information from multiple window sizes.
Optical Flow Results
Optical Flow Results
1. Gradient based methods (Horn &Schunk, Lucase & Kanade, …)
2. Region based methods (SSD, Normalized correlation, etc.)
Copyright, 1996 © Dale Carnegie & Associates, Inc.
But… (despite coarse-to-fine estimation)
• rely on B.C.
• cannot handle very large motions (no more than 10%-15% of image width/height)
• small object moving fast…?
Region-Based Methods
* Define a small area around a pixel as the region.
* Match the region against each pixel within a search area in next image.
* Use a match measure (e.g., sum of-squares difference, normalized correlation, etc.)
* Choose the maximum (or minimum) as the match.
SSD Surface – Textured area
SSD Surface -- Edge
SSD – homogeneous area
[Anandan’89 - Use coarse-to-fine SSD of local windows to find matches. - Propagate information using directional confidence measures extracted from each local SSD surface]
B.C. + Additional constraints:
Copyright, 1996 © Dale Carnegie & Associates, Inc.
• Increase aperture: [e.g., Lucas & Kanade]
Local singularities at degenerate image regions.
• Increase analysis window to large image regions => Global model constraints: (NEXT LESSON)
Numerically stable, but requires prior model selection:
• Planar (2D) world model
[e.g., Bergen-et-al:92, Irani-et-al:92+94, Black-et-al]
• 3D world model[e.g., Hanna-et-al:91+93, Stein & Shashua:97, Irani-et-al:1999]
• Spatial smoothness: [e.g., Horn & Schunk:81, Anandan:89]
Violated at depth/motion discontinuities