Ivo Ihrke / Autumn 2015
Computational Optical Imaging -Optique Numerique
-- Single and Multiple View Geometry, Stereo matching --
Autumn 2015
Ivo Ihrke
with slides by Thorsten Thormaehlen
Ivo Ihrke / Autumn 2015
Reminder: Feature Detection and Matching
Wide-Baseline-Matching
Ivo Ihrke / Autumn 2015
SIFT = Scale Invariant Feature Transform David G. Lowe: “Distinctive image features from scale-invariant keypoints” (IJCV 2004)
http://www.cs.ubc.ca/~lowe/keypoints/
Suited for wide-baseline matching
Invariance to changes in illumination, scale, and rotation
No gradient approach, but compares feature description of all candidates
Many applications
Wide-baseline Matching with SIFT
Ivo Ihrke / Autumn 2015
Application example: Mosaic generation
Wide-baseline Matching with SIFT
Ivo Ihrke / Autumn 2015
Application example: Object recognition
Wide-baseline Matching with SIFT
Ivo Ihrke / Autumn 2015
Stereo pair with super-imposed SIFT features
SIFT: Keypoint localization
Arrow size indicates scale
729 out of 832 feature are left after contrast thresholding
(try cross-eye fusion to identify the outliers :)
Ivo Ihrke / Autumn 2015
SIFT: Descriptor matching
Ivo Ihrke / Autumn 2015
VLFeat (http://www.vlfeat.org/index.html)
Implements SIFT and other modern features
C-implementation with MATLAB bindings
─ Win/Linux
Good tutorials on parameter selection
Software
Ivo Ihrke / Autumn 2015
Towards Multiple Views and Self-Calibration
-- Two-View Geometry and Basic Stereo --
Ivo Ihrke / Autumn 2015
Fundamental Matrix (F-Matrix):
Fundamental Matrix
Ivo Ihrke / Autumn 2015
Epipolar Geometry in our example
Ivo Ihrke / Autumn 2015
F is a rank 2 homogeneous matrix with 7 degrees of freedom
Epipolar lines
Epipoles
Fundamental Matrix
Ivo Ihrke / Autumn 2015
F can be computed from camera matrices
General projective cameras:
with and
Canonical cameras not at infinity and
Fundamental Matrix
Ivo Ihrke / Autumn 2015
Typical left and right image with parallel optical axes and only horizontally displaced
Use Case – Stereo Matching
[implementation by Rohit Singh and Mitul Sara]
Left image Right image
Ivo Ihrke / Autumn 2015
Epipolar lines are parallel lines if optical axes are parallel
Epipolar Geometry of a Stereo Pair
Ivo Ihrke / Autumn 2015
Simplest: Search a moving window and perform correlation
E.g. SSD (sum of squared differences)
Alternatively, SAD (sum of absolute
differences), CC (cross correlation), etc.
Usually restricted search range
Epipolar Geometry of a Stereo Pair
Ivo Ihrke / Autumn 2015
Example, SSD, win=5, search range=15
Correlation-Based Stereo Matching
[implementation by Rohit Singh and Mitul Saha]
Ivo Ihrke / Autumn 2015
Example, SSD, win=5, search range=8
Correlation-Based Stereo Matching
[implementation by Rohit Singh and Mitul Saha]
Ivo Ihrke / Autumn 2015
Example, SAD, win=5, search range=8
Correlation-Based Stereo Matching
[implementation by Rohit Singh and Mitul Saha]
Ivo Ihrke / Autumn 2015
Example, SSD, win=20, search range=8
Correlation-Based Stereo Matching
[implementation by Rohit Singh and Mitul Saha]
Ivo Ihrke / Autumn 2015
ADCensus [Mei’11] (#1 Middlebury stereo benchmark, Sept. 2013)
A Modern Technique
Ivo Ihrke / Autumn 2015
Towards Multiple Views and Self-Calibration
-- The Structure-from-motion Pipeline --
Ivo Ihrke / Autumn 2015
Voodoo Camera Tracker - Steps of camera tracking
image sequence
Feature detection
and
correspondence
analysis
Outlier EliminationIncremental
Bundle AdjustmentSelf-Calibration
camera parameters
Ivo Ihrke / Autumn 2015
Correspondence Analysis
For video KLT-Tracker (Kanade, Lucas and Tomasi)
Minimization of the SSD between two windows in subsequent camera images to find the displacement vector
Correspondence Analysis – KLT Tracker
window
image K-1 image K
Ivo Ihrke / Autumn 2015
Linearization of the image signal with a Taylor series
KLT Tracker
Finding the minimum by setting the derivative to zero
Ivo Ihrke / Autumn 2015
KLT Tracker
with
Ivo Ihrke / Autumn 2015
Optical Flow – Horn & Schunck’81
Optical Flow as seen by a
person at the back of a train
• Continuous version of tracking: optical flow
• Apparent motion of brightness patterns in an image
sequence (typically two frames)
• For images: 𝑢( 𝑥): 𝑅2 𝑅2, is a vector valued fct.
• Often visualized as vector field or color coded
Ivo Ihrke / Autumn 2015
Example Yosemite sequencele
ftrigh
t
Flow field (middlebury coding)
Flow field (IPOL coding)
IPOL coding
middlebury coding
Ivo Ihrke / Autumn 2015
Assume a video 𝐼( 𝑥, 𝑡): 𝑅3 𝑅
Brightness constancy implies𝐼 𝑥 + 𝑢( 𝑥, 𝑡), 𝑡 = 𝐼( 𝑥, 𝑡 + 1)
Look at one particular time step with flow vectors 𝑢 = (𝑢𝑥, 𝑢𝑦)
perform Taylor expansion of 𝐼 𝑥, 𝑡 + 1 :
𝐼 𝑥, 𝑡 + 1 ≈ 𝐼 𝑥, 𝑡 +𝜕𝐼
𝜕𝑥𝑢𝑥 +
𝜕𝐼
𝜕𝑦𝑢𝑦 +
𝜕𝐼
𝜕𝑡+ 𝑂(𝛻2)
implies𝜕𝐼
𝜕𝑥𝑢𝑥 +
𝜕𝐼
𝜕𝑦𝑢𝑦 +
𝜕𝐼
𝜕𝑡= 0
Alternative form: 𝛻𝐼 ∙ 𝑢 +𝜕𝐼
𝜕𝑡= 0
Optical Flow- Derivation
Ivo Ihrke / Autumn 2015
KLT and optical flow descriptions are differential in nature only suitable for small displacements (up to 2 pixels)
Large displacements are small on a coarser scale
compute displacements on coarse scale
Upsample
Iteratively compute residuals on finer scales
KLT-Tracker: Resolution pyramids
resolution
Ivo Ihrke / Autumn 2015
KLT-Tracker: Affine mapping for longer sequences
Ivo Ihrke / Autumn 2015
For video (small displacements)
Minimization of the SSD between two windows in subsequent camera images
Linearization of the image signal with a Taylor series → linear equation system
Linearization only valid for small displacements → Resolution Pyramids
KLT-Tracker summary
Ivo Ihrke / Autumn 2015
Voodoo Camera Tracker - Steps of camera tracking
image sequence
Feature detection
and
correspondence
analysis
Outlier EliminationIncremental
Bundle AdjustmentSelf-Calibration
camera parameters
Ivo Ihrke / Autumn 2015
RANSAC (Random Sample Consensus) method
Outlier Elimination
Ivo Ihrke / Autumn 2015
RANSAC example – line fit
Least-squares fit
LS-fit after RANSAC
Ivo Ihrke / Autumn 2015
Fundamental Matrix (F-Matrix):
Outlier Elimination - Fundamental Matrix
Ivo Ihrke / Autumn 2015
Fundamental Matrix Estimation
multiple of these equations gives a linear equation system
Estimating the fundamental matrix from
feature correspondence:
Ivo Ihrke / Autumn 2015
2D Homography (H-Matrix):
Outlier Elimination - 2D Homography
Ivo Ihrke / Autumn 2015
2D Homography Matrix Estimation
multiple of these equations gives a linear equation system
Estimating the 2D homography matrix
from feature correspondence:
Ivo Ihrke / Autumn 2015
Outlier Elimination - Camera Matrix
Camera matrix (A-Matrix):
Ivo Ihrke / Autumn 2015
Incremental Bundle Adjustment
Ivo Ihrke / Autumn 2015
Bundle Adjustment
Bundle Adjustment
Ivo Ihrke / Autumn 2015
Measurement vector
Levenberg Marquardt –Non-linear Least Squares
Parameter vector
Taylor approximation:
with N x M Jacobian Matrix
Ivo Ihrke / Autumn 2015
Levenberg Marquardt –Non-linear Least Squares
transformed to linear least squares problem for each iteration
Ivo Ihrke / Autumn 2015
Levenberg Marquardt –Non-linear Least Squares
Linear least squares problem can be solved with normal equations
use to update solution iteratively
Ivo Ihrke / Autumn 2015
Levenberg Marquardt uses slightly different normal equations
Levenberg Marquardt –Non-linear Least Squares
Original normal equations
Modified normal equations
Lambda is changed during optimization
successful iteration
failed iteration
small ~ Newton style (quadratic convergence)
large ~ Gradient descent style (guaranteed decrease)
Ivo Ihrke / Autumn 2015
Requirements for Levenberg Marquardt minimization
Function to compute f
Start value X0
Optionally, function to compute J(but numerical derivation works as well)
Levenberg Marquardt
Ivo Ihrke / Autumn 2015
For bundle adjustment the problem becomes to large
(100 cameras + 10000 3D object points = 31200 parameters)
Can achieve huge speed-up by exploiting sparse structure of Jacobianmatrix
Partition parameters
partition A
partition B (only dependent on A and itself)
(typically A contains camera parameters, and B contains 3D points)
Sparse Levenberg Marquardt
Ivo Ihrke / Autumn 2015
Sparse Levenberg Marquardt
Jacobian becomes
Normal equations
become
Ivo Ihrke / Autumn 2015
Corresponding block structure is
where denotes augmented by multiplying its diagonal entries by a factor of 1 + , and likewise. Left multiplication with
yields
which can be used to find with
which may be back-substituted to get with
Sparse Levenberg Marquardt
Ivo Ihrke / Autumn 2015
Jacobian for bundle adjustment has sparse block structure
Sparse Levenberg Marquardt
U1
U2
U3
WT
W
V
A1 A2 A3 P
J JJT
12 x m 3 x n(in general much larger)
im.pts.
A1
Needed for non-linear minimization
Ivo Ihrke / Autumn 2015
Voodoo camera tracker – demo session
Ivo Ihrke / Autumn 2015
Why self-calibration?
Allows flexible acquisition
No prior calibration necessary
Possibility to vary intrinsic camera parameters
Use archive footage
Self-calibration
Ivo Ihrke / Autumn 2015
We want to find a 4x4 transformation matrix that transforms all projective cameras into metric cameras
This does not change the back-projections onto the feature points
Self-calibration
Ivo Ihrke / Autumn 2015
Voodoo Camera Tracker –Some applications
Virtual advertising
Ivo Ihrke / Autumn 2015
Voodoo Camera Tracker –More applications
Car navigation
Architectural visualisation
3D Endoscopy
UAV terrain reconstruction
Ivo Ihrke / Autumn 2015
Matchmoving in Cloverfield
Ivo Ihrke / Autumn 2015
Matchmoving in Cloverfield
Ivo Ihrke / Autumn 2015
References
1. Multiple View Geometry in Computer Vision, Richard
Hartley and Andrew Zisserman, 2nd Edition
2. Triggs, B.: “Autocalibration and the absolute quadric”.
In “IEEE Conference on Computer Vision and Pattern
Recognition”, S. 609–614. 1997
3. Pollefeys, M., Koch, R., Gool, L. V.: “Self-Calibration
and Metric Reconstruction in Spite of Varying and
Unknown Internal Camera Parameters”. In “EEE
International Conference on Computer Vision, S. 90–
95. 1998