Upload
preston-mcfadden
View
39
Download
2
Embed Size (px)
DESCRIPTION
Affine structure from motion. Marc Pollefeys COMP 256. Some slides and illustrations from J. Ponce, A. Zisserman, R. Hartley, Luc Van Gool, …. Last time: Optical Flow. I x u. I x. u. I x u= - I t. I t. Aperture problem:. two solutions: - regularize (smoothness prior) - PowerPoint PPT Presentation
Citation preview
ComputerVision
Affine structure from motion
Marc PollefeysCOMP 256
Some slides and illustrations from J. Ponce, A. Zisserman, R. Hartley, Luc Van Gool, …
ComputerVision
2
Last time: Optical Flow
0 tyx IvIuI uIx
It
Ixu
Ixu=It
Aperture problem:
two solutions:- regularize (smoothness prior)- constant over window (i.e. Lucas-Kanade)
Coarse-to-fine, parametric models, etc…
ComputerVision
3
Jan 16/18 - Introduction
Jan 23/25 Cameras Radiometry
Jan 30/Feb1 Sources & Shadows Color
Feb 6/8 Linear filters & edges Texture
Feb 13/15 Multi-View Geometry Stereo
Feb 20/22 Optical flow Project proposals
Feb27/Mar1 Affine SfM Projective SfM
Mar 6/8 Camera Calibration Silhouettes and Photoconsistency
Mar 13/15 Springbreak Springbreak
Mar 20/22 Segmentation Fitting
Mar 27/29 Prob. Segmentation Project Update
Apr 3/5 Tracking Tracking
Apr 10/12 Object Recognition Object Recognition
Apr 17/19 Range data Range data
Apr 24/26 Final project Final project
Tentative class schedule
ComputerVision
4
AFFINE STRUCTURE FROM MOTION
Reading: Chapter 12.
• The Affine Structure from Motion Problem• Elements of Affine Geometry• Affine Structure from Motion from two Views
• A Geometric Approach• Affine Epipolar Geometry• An Algebraic Approach
• Affine Structure from Motion from Multiple Views• From Affine to Euclidean Images• Structure from motion of multiple and deforming object
ComputerVision
5
Affine Structure from Motion
Reprinted with permission from “Affine Structure from Motion,” by J.J. (Koenderink and A.J.Van Doorn, Journal of the Optical Society of America A, 8:377-385 (1990). 1990 Optical Society of America.
Given m pictures of n points, can we recover• the three-dimensional configuration of these points?• the camera configurations?
(structure)(motion)
ComputerVision
6
Orthographic Projection
Parallel Projection
ComputerVision
7
Weak-Perspective Projection
Paraperspective Projection
ComputerVision
8
The Affine Structure-from-Motion Problem
Given m images of n fixed points P we can write
Problem: estimate the m 2x4 matrices M andthe n positions P from the mn correspondences p .
i
j ij
2mn equations in 8m+3n unknowns
Overconstrained problem, that can be solvedusing (non-linear) least squares!
j
ComputerVision
9
The Affine Ambiguity of Affine SFM
If M and P are solutions, i j
So are M’ and P’ wherei j
and
Q is an affinetransformation.
When the intrinsic and extrinsic parameters are unknown
ComputerVision
10
Affine Spaces: (Semi-Formal) Definition
ComputerVision
11
Example: R as an Affine Space2
ComputerVision
12
In General
The notation
is justified by the fact that choosing some origin O in Xallows us to identify the point P with the vector OP.
Warning: P+u and Q-P are defined independently of O!!
ComputerVision
13
Barycentric Combinations
• Can we add points? R=P+Q NO!
• But, when we can define
• Note:
ComputerVision
14
Affine Subspaces
ComputerVision
15
Affine Coordinates
• Coordinate system for U:
• Coordinate system for Y=O+U:
• Coordinate system for Y:
• Affine coordinates:
• Barycentric coordinates:
ComputerVision
16
When do m+1 points define a p-dimensional subspace Y of an n-dimensional affine space X equipped with some coordinate frame basis?
Writing that all minors of size (p+2)x(p+2) of D are equal to zero gives the equations of Y.
Rank ( D ) = p+1, where
ComputerVision
17
Affine Transformations
Bijections from X to Y that:• map m-dimensional subspaces of X onto m-dimensional subspaces of Y;• map parallel subspaces onto parallel subspaces; and• preserve affine (or barycentric) coordinates.
In E they are combinations of rigid transformations, non-uniform scalings and shears.
Bijections from X to Y that:• map lines of X onto lines of Y; and• preserve the ratios of signed lengths of line segments.
3
ComputerVision
18
Affine Transformations II
• Given two affine spaces X and Y of dimension m, and two coordinate frames (A) and (B) for these spaces, there exists a unique affine transformation mapping (A) onto (B).
• Given an affine transformation from X to Y, one can always write:
• When coordinate frames have been chosen for X and Y,this translates into:
ComputerVision
19
Affine projections induce affine transformations from planesonto their images.
ComputerVision
20
Affine Shape
Two point sets S and S’ in some affine space X are affinely equivalent when there exists an affine transformation : X X such that X’ = ( X ).
Affine structure from motion = affine shape recovery.
= recovery of the corresponding motion equivalence classes.
ComputerVision
21
Geometric affine scene reconstruction from two images(Koenderink and Van Doorn, 1991).
ComputerVision
22
Affine Structure from Motion
(Koenderink and Van Doorn, 1991)
Reprinted with permission from “Affine Structure from Motion,” by J.J. (Koenderink and A.J.Van Doorn, Journal of the Optical Society of America A, 8:377-385 (1990). 1990 Optical Society of America.
ComputerVision
23
The Affine Epipolar Constraint
Note: the epipolar lines are parallel.
ComputerVision
24
Affine Epipolar Geometry
ComputerVision
25
The Affine Fundamental Matrix
where
ComputerVision
26
An Affine Trick.. Algebraic Scene Reconstruction
ComputerVision
27
The Affine Structure of Affine Images
Suppose we observe a scene with m fixed cameras..
The set of all images of a fixed scene is a 3D affine space!
ComputerVision
28
has rank 4!
ComputerVision
29
From Affine to Vectorial Structure
Idea: pick one of the points (or their center of mass)as the origin.
ComputerVision
30
What if we could factorize D? (Tomasi and Kanade, 1992)
Affine SFM is solved!
Singular Value Decomposition
We can take
ComputerVision
31
From uncalibrated to calibrated cameras
Weak-perspective camera:
Calibrated camera:
Problem: what is Q ?
Note: Absolute scale cannot be recovered. The Euclidean shape(defined up to an arbitrary similitude) is recovered.
ComputerVision
32
Reconstruction Results (Tomasi and Kanade, 1992)
Reprinted from “Factoring Image Sequences into Shape and Motion,” by C. Tomasi andT. Kanade, Proc. IEEE Workshop on Visual Motion (1991). 1991 IEEE.
ComputerVision
33
More examples
Tomasi Kanade’92,Poelman & Kanade’94
ComputerVision
34
More examples
Tomasi Kanade’92,Poelman & Kanade’94
ComputerVision
35
More examples
Tomasi Kanade’92,Poelman & Kanade’94
ComputerVision
36
Further Factorization work
Factorization with uncertainty
Factorization for indep. moving objects (now)
Factorization for articulated objects (now)
Factorization for dynamic objects (now)
Perspective factorization (next week)
Factorization with outliers and missing pts.
(Irani & Anandan, IJCV’02)
(Costeira and Kanade ‘94)
(Bregler et al. 2000, Brand 2001)
(Jacobs ‘97 (affine), Martinek & Pajdla‘01 Aanaes’02 (perspective))
(Sturm & Triggs 1996, …)
(Yan and Pollefeys ‘05)
ComputerVision
37
Structure from motion of multiple moving objects
ComputerVision
38
Structure from motion of multiple moving objects
ComputerVision
39
Shape interaction matrixShape interaction matrix for articulated
objects looses block diagonal structure
Costeira and Kanade’s approach is not usable for articulated bodies(assumes independent motions)
ComputerVision
40
Articulated motion subspaces
Joint (1D intersection)
Hinge (2D intersection)
(joint=origin)
(hinge=z-axis)
Motion subspaces for articulated bodies intersect
(rank=8-1)
(rank=8-2)
(Yan and Pollefeys, CVPR’05)(Tresadern and Reid, CVPR’05)
Exploit rank constraint to obtain better estimate
(Yan & Pollefeys, 06?)Also for non-rigid parts if
ComputerVision
41
Student Segmentation
Intersection
Results
Toy truckSegmentation
Intersection
ComputerVision
42
Articulated shape and motion factorization
Automated kinematic chain building for articulated & non-rigid obj.– Estimate principal angles between subspaces – Compute affinities based on principal angles– Compute minimum spanning tree
(Yan and Pollefeys, 2006?)
ComputerVision
43
Structure from motion of deforming objects
Extend factorization approaches to deal with dynamic shapes
(Bregler et al ’00; Brand ‘01)
ComputerVision
44
Representing dynamic shapes
represent dynamic shape as varying linear combination of basis shapes
k
kk (t)ScS(t)
(fig. M.Brand)
ComputerVision
45
Projecting dynamic shapes
PtScR k
kk
(figs. M.Brand)Rewrite:
ComputerVision
46
Dynamic image sequences
One image:
Multiple images(figs. M.Brand)
ComputerVision
47
Dynamic SfM factorization?
Problem: find J so that M has proper structure
ComputerVision
48
Dynamic SfM factorization
(Bregler et al ’00)
Assumption: SVD preserves order and orientation of basis shape components
ComputerVision
49
Results(Bregler et al ’00)
ComputerVision
50
Dynamic SfM factorization(Brand ’01)
constraints to be satisfied for M
constraints to be satisfied for M, use to compute J
hard!
(different methods are possible, not so simple and also not optimal)
ComputerVision
51
Non-rigid 3D subspace flow
• Same is also possible using optical flow in stead of features, also takes uncertainty into account
(Brand ’01)
ComputerVision
52
Results (Brand ’01)
ComputerVision
53
(Brand ’01)
Results
ComputerVision
54
Results(Bregler et al ’01)
ComputerVision
55
Next class: Projective structure from motion
Reading: Chapter 13