Embed Size (px)
Citation preview
ComputerVision
Projective structure from motion
Marc PollefeysCOMP 256
Some slides and illustrations from J. Ponce, …
ComputerVision
2
Last class: Affine camera
• The affine projection equations are
1
~
~
j
j
j
yi
xi
ij
ij
ZYX
P
Py
x
how to find the origin? or for that matter a 3D reference point?
affine projection preserves center of gravity
i
ijijij xxx~ i
ijijij yyy~
2
1
tt
~
~
j
j
j
yi
xi
ij
ij
Z
Y
X
P
Py
x
ComputerVision
3
Orthographic factorization
• The orthographic projection equations are
• where
njmijiij ,...,1,,...,1,Xx P
• All equations can be collected for all i and j
• where
n
mmnmm
n
n
X,...,X,X , ,
xxx
xxxxxx
212
1
21
22221
11211
X
P
P
P
Px
XPx
X ~
~x
j
j
j
jyi
xi
iij
ijij
Z
Y
X
,P
P,
y
xP
Note that P and X are resp. 2mx3 and 3xn matrices and
therefore the rank of x is at most 3
2
1
tt
~
~
j
j
j
yi
xi
ij
ij
Z
Y
X
P
Py
x
ComputerVision
4
Orthographic factorization
• Factorize m through singular value decomposition
• An affine reconstruction is obtained as follows
TVUx
T~,
~VXUP
n
mmnmm
n
n
X,...,X,X
xxx
xxxxxx
min 212
1
21
22221
11211
P
P
P
Closest rank-3 approximation yields MLE!
ComputerVision
5
Jan 16/18 - Introduction
Jan 23/25 Cameras Radiometry
Jan 30/Feb1 Sources & Shadows Color
Feb 6/8 Linear filters & edges Texture
Feb 13/15 Multi-View Geometry Stereo
Feb 20/22 Optical flow Project proposals
Feb27/Mar1 Affine SfM Projective SfM
Mar 6/8 Camera Calibration Silhouettes and Photoconsistency
Mar 13/15 Springbreak Springbreak
Mar 20/22 Segmentation Fitting
Mar 27/29 Prob. Segmentation Project Update
Apr 3/5 Tracking Tracking
Apr 10/12 Object Recognition Object Recognition
Apr 17/19 Range data Range data
Apr 24/26 Final project Final project
Tentative class schedule
ComputerVision
6
PROJECTIVE STRUCTURE FROM MOTION
Reading: Chapter 13.
• The Projective Structure from Motion Problem• Elements of Projective Geometry• Projective Structure and Motion from Two Images • Projective Motion from Fundamental Matrices• Projective Structure and Motion from Multiple Images
ComputerVision
7
The Projective Structure-from-Motion Problem
Given m perspective images of n fixed points P we can write
Problem: estimate the m 3x4 matrices M andthe n positions P from the mn correspondences p .
i
j ij
2mn equations in 11m+3n unknowns
Overconstrained problem, that can be solvedusing (non-linear) least squares!
j
ComputerVision
8
The Projective Ambiguity of Projective SFM
If M and P are solutions, i j
So are M’ and P’ wherei j
and Q is an arbitrary non-singular 4x4 matrix.
When the intrinsic and extrinsic parameters are unknown
Q is a projective transformation.
ComputerVision
9
Projective Spaces: (Semi-Formal) Definition
ComputerVision
10
A Model of P( R )3
ComputerVision
11
Projective Subspaces and Projective Coordinates
ComputerVision
12
Projective Subspaces and Projective Coordinates
Projectivecoordinates
P
ComputerVision
13
Projective Subspaces
Given a choice of coordinate frame
Line: Plane:
ComputerVision
14
Affine and Projective Spaces
ComputerVision
15
Affine and Projective Coordinates
ComputerVision
16
Cross-Ratios
Collinear points
Pencil of coplanar lines Pencil of planes
{A,B;C,D}=sin(+)sin(+)
sin(++)sin
ComputerVision
17
Cross-Ratios and Projective Coordinates
Along a line equipped with the basis
In a plane equipped with the basis
In 3-space equipped with the basis
*
*
ComputerVision
18
Projective Transformations
Bijective linear map:
Projective transformation:( = homography )
Projective transformations map projective subspaces ontoprojective subspaces and preserve projective coordinates.
Projective transformations map lines onto lines andpreserve cross-ratios.
ComputerVision
19
Perspective Projections induce projective transformationsbetween planes.
ComputerVision
20
Geometric SceneReconstruction
Idea: use (A,O”,O’,B,C)as a projective basis.
ComputerVision
21
Reprinted from “Relative Stereo and Motion Reconstruction,” by J. Ponce, T.A. Cass and D.H. Marimont, Tech. ReportUIUC-BI-AI-RCV-93-07, Beckman Institute, Univ. of Illinois (1993).
ComputerVision
22
Motion estimation from fundamental matrices
Q
Facts:
b’ can be found using LLS.
Once M and M’ are known,P can be computed with LLS.
ComputerVision
23
Projective Structure from Motion and Factorization
Factorization??
Algorithm (Sturm and Triggs, 1996) • Guess the depths;• Factorize D;• Iterate.
Does it converge? (Mahamud and Hebert, 2000)
ComputerVision
24
Bundle adjustment - refining structure and motion
• Minimize reprojection error
– Maximum Likelyhood Estimation (if error zero-mean Gaussian
noise)– Huge problem but can be solved
efficiently (exploit sparseness)
m
i
n
jjiijD
ji 1 1
2
X̂,P̂X̂P̂,xmin
ComputerVision
25
• Developed in photogrammetry in 50´s
Bundle adjustment
ComputerVision
26
Non-linear least squares• Linear approximation of residual• allows quadratic approximation of sum- of-squares
J0e
0T1-T
0TT
JJJ
0J2J2J
e
e
JJ 00 ee T
J)λdiag(JJJN' TT
N
(extra term = descent term)
Minimization corresponds to finding zeros of derivative
Levenberg-Marquardt: extra term to deal with singular N(decrease/increase if success/failure to descent)
ComputerVision
27
Bundle adjustment
U1
U2
U3
WT
W
V
P1 P2 P3 M
• Jacobian of has sparse block structure– cameras independent of other cameras,– points independent of other points
J JJN T
12xm 3xn(in general
much larger)
im.pts. view 1
m
i
n
jjiijD
1 1
2M̂P̂,m
Needed for non-linear minimization
ComputerVision
28
Bundle adjustment
Eliminate dependence of camera/motion parameters on structure parametersNote in general 3n >> 11m
WT V
U-WV-1WT
NI0WVI 1
11xm 3xn
Allows much more efficient computations
e.g. 100 views,10000 points,
solve 1000x1000, not 30000x30000Often still band diagonaluse sparse linear algebra algorithms
ComputerVision
29
Sequential SfM
• Initialize motion from two images• Initialize structure• For each additional view
– Determine pose of camera– Refine and extend structure
• Refine structure and motion
ComputerVision
30
Initial projective camera motion
• Choose P and P´compatible with F
Reconstruction up to projective ambiguity
(reference plane;arbitrary)
•Initialize motion•Initialize structure•For each additional view
•Determine pose of camera•Refine and extend structure
•Refine structure and motion
Same for more views?
different projective basis
ComputerVision
31
Initializing projective structure
• Reconstruct matches in projective frame by minimizing the reprojection error
Non-iterative optimal solution •Initialize motion•Initialize structure•For each additional view
•Determine pose of camera•Refine and extend structure
•Refine structure and motion
ComputerVision
32
Projective pose estimation
• Infere 2D-3D matches from 2D-2D matches
• Compute pose from (RANSAC,6pts)
F
X
x
Inliers: inx,X x X DD iii P
•Initialize motion•Initialize structure•For each additional view
•Determine pose of camera•Refine and extend structure
•Refine structure and motion
ComputerVision
33
• Refining structure
• Extending structure2-view triangulation
X~
P
1
3
(Iterative linear)
•Initialize motion•Initialize structure•For each additional view
•Determine pose of camera•Refine and extend structure
•Refine structure and motion
Refining and extending structure
ComputerVision
34
Refining structure and motion
• use bundle adjustment
Also model radial distortion to avoid bias!
ComputerVision
35
Hierarchical structure and motion recovery
• Compute 2-view• Compute 3-view• Stitch 3-view reconstructions• Merge and refine reconstruction
FT
H
PM
ComputerVision
36
Metric structure and motion
Note that a fundamental problem of the uncalibrated approach is that it fails if a purely planar scene is observed (in one or more views)
(solution possible based on model selection)
use self-calibration (see next class)
ComputerVision
37
Dealing with dominant planes
ComputerVision
38
PPPgric
HHgric
ComputerVision
39
Farmhouse 3D models
(note: reconstruction much larger than camera field-of-view)
ComputerVision
40
Application: video augmentation
ComputerVision
41
Next class: Camera calibration (and self-calibration)
Reading: Chapter 2 and 3
