Upload
mia-stephens
View
222
Download
6
Tags:
Embed Size (px)
Citation preview
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.20051
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
Algorithms
• Point correspondences– Salient point detection – Local descriptors
• Matrix decompositions– RQ decomposition– Singular value decomposition - SVD
• Estimation– Systems of linear equations– Solving systems of linear equations
• Direct Linear Transform – DLT• Normalization• Iterative error / cost minimization• Outliers Robustness, RANSAC
– Pose estimation• Perspective n-point problem – PnP
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.20052
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
Relevant Issues in Practice
• Poor condition of A Normalization
• Algebraic error vs.geometric error, Iterative minimizationnonlinearities (lens dist.)
• Outliers Robust algorithms (RANSAC)
0 0 0
fph AAA
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.20053
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
Normalization (1)
0
0
000
000:0
9
1
'11
'11
'11
'11
'11
'11
'11
'11
'11
'11
'11
'11
1
h
h
xwxyxxwwwywx
ywyyyxwwwywxh
A
Homography H:
• Entries of A are quadratic in point coordinates• SVD is not robust against coordinate transform !
– change of coordinate system (translation, scaling) will influence result !
– algebraic vs. geometric error !
• Normalization recommended, e.g.:– translate origin (0,0,1) image center
– “isotropic” scaling such that:• either average distance to (0,0,1) is ,• or “average point” is (1,1,1)
2
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.20054
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
Normalization (2)
Fundamental Matrix F:
• “poor condition” of ATA• Normalization is mandatory• “normalized 8-point algorithm” to estimate F [Hartley’95]
“in defense of the 8-point algorithm”
)1,10,10,10,10,10,10,10,10(: of diagonal
)1,10,10,10,10,10,10,10,10(:in line
)1,100,100(' assume
44488488
22244244
AA
AT
TT uu
note: some algorithms useeigenvalues of ATA insteadof singular values (SVD) !
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.20055
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
Iterative Minimization
• DLT minimizes “algebraic error”• “geometric distance” is more complex
• Lens distortion is non-linear
• “Standard” approach:– estimate linear parameters by DLT initialization for– subsequent iterative minimization over all parameters
• E.g.: “gold standard” for estimation of H
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.20056
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
“Gold Standard” for Estimation of H (1)[Hartley+Zisserman]
ObjectiveGiven n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi
Algorithm
(i) For each correspondence xi ↔xi’ compute Ai. Usually only two first rows needed.
(ii) Assemble n 2x9 matrices Ai into a single 2nx9 matrix A
(iii) Obtain SVD of A. Solution for h is last column of V
(iv) Determine H from h[adapted from Pollefeys’ course]
DLT algorithm to estimate H:
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.20057
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
“Gold Standard” for Estimation of H (2)[Hartley+Zisserman]
[adapted from Pollefeys’ course]
normalized DLT algorithm to estimate H:
ObjectiveGiven n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi
Algorithm
(i) Normalize points
(ii) Apply DLT algorithm to
(iii) Denormalize solution
inorminormi 'x'x,xx
TT i~~
,xx ii~~
norm-
norm THTH~1
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.20058
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
“Gold Standard” for Estimation of H (3)[Hartley+Zisserman]
[adapted fromPollefeys’ course]
ObjectiveGiven n≥4 2D to 2D point correspondences {xi↔xi’}, determine the Maximum Likelihood Estimation of H
Algorithm
(i) Initialization: compute an initial estimate using normalized DLT or RANSAC
(ii) Geometric minimization of either Sampson error:
● Minimize the Sampson error
● Minimize using Levenberg-Marquardt over 9 entries of h
or Gold Standard error:
● compute initial estimate for “subsidiary”
● minimize cost over
● if many points, use sparse method
22 ˆˆ iiii x',xdx,xd }ˆ{ ix
nixi 1,ˆ and ˆ H
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.20059
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
Robust Estimation (RANSAC) [Hartley+Zisserman]
Handling of outliers !
“RANSAC” = “RANdom Sample Consensus”
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.200510
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
RANSAC Algorithm [Hartley+Zisserman]
ObjectiveRobust fit of model to data set S which contains outliers
Algorithm
(i) Randomly select a sample of s data points from S and instantiate the model from this subset.
(ii) Determine the set of data points Si which are within a distance threshold t of the model. The set Si is the consensus set of samples and defines the inliers of S.
(iii) If the subset of Si is greater than some threshold T, re-estimate the model using all the points in Si and terminate
(iv) If the size of Si is less than T, select a new subset and repeat the above.
(v) After N trials the largest consensus set Si is selected, and the model is re-estimated using all the points in the subset Si
[adapted FromPollefeys’course]
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.200511
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
RANSAC Algorithm [Hartley+Zisserman]
proportion of outliers es 5% 10% 20% 25% 30% 40% 50%2 2 3 5 6 7 11 173 3 4 7 9 11 19 35
4 3 5 9 13 17 34 725 4 6 12 17 26 57 1466 4 7 16 24 37 97 2937 4 8 20 33 54 163 5888 5 9 26 44 78 272 1177
sample size vs. proportion of outliers:
[adapted from Pollefeys’ course]
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.200512
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
More Problems“critical” cases ! e.g. [Torr+Murray, IJCV 1997]
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.200513
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
Pose Estimation
X
Y
Z
• calibrated camera, known K
C
xC
yC
zC
R, t
• Camera
xV
yV
zV
• Visualization (screen, HMD)
R, t
• determine camera pose: R, t
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.200514
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
Perspective n-Point Problem – PnP (1)
• Calibrated camera– Known K
• Known points Pi in the scene• Given n point correspondences
– pi ↔ Pi
• What can be measured with one calibrated camera?
angle θ between two lines of sight
pa
pb
Pb
Pa
dabθ
C
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.200515
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
Perspective n-Point Problem – PnP (2)
• PnP uses just this information:
• P3P will give up to 4 solutions• P4P is already overdetermined
– Perform 4 x P3P– Find consensus
pa
pb
Pb
Pa
dabθ
C
abbabaab CPCPCPCPd cos2222
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.200516
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
Pose Estimation Tracking
• In theory, tracking is simple !– Calibrate your camera (K)– Measure some points Pi in the scene (“ground truth”)– Perform pose estimation in real-time (for each frame)
• In practice, tracking is a hard problem !– Point detection – Correspondence– Motion prediction – Occlusion– Unknown scene– …
• Many solutions have been proposed !
“Tracking beyond 15 minutes of thought”SIGGRAPH 2001 Turorial #15[Allen, Bishop, Welch]
“An introduction to the Kalman filter”SIGGRAPH 2001 Turorial #8[Welch, Bishop]
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.200517
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
Tracking Systems (vision-based / hybrid)some of my own contributions (1)
Hybrid“inside out”magnetic + stereo vision[Auer 1999]
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.200518
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
Tracking Systems (vision-based / hybrid)some of my own contributions (2)
stereo vision“outside in”[Ribo ca. 2000]
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.200519
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
Tracking Systems (vision-based / hybrid)some of my own contributions (3)
inertial“inside out”hybrid inertal + vision[many 2000-2004]
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.200520
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
Tracking Systems (vision-based / hybrid)some of my own contributions (4)
vision (stereo or mono)“inside out”speed solves correspondence ! [Mühlmann 2005]
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.200521
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
Our Current View [Schweighofer 2008]
stereo vision“inside out”structure and motion
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.200522
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
Summary
In these four lectures, I gave an introduction to:• Projective geometry• Perspective cameras• Homographies, camera projection matrices, fundamental and
essential matrices• Algorithms that are typically applied to solve for
– Camera calibration– Stereo reconstruction– Camera pose estimation
I consider this the basis for further reading in topics including:• Vision-based pose tracking• Structure and motion analysis (sometimes termed “SLAM”)
Many aspects were, of course, not covered, but would also be important !
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.200523
22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz
What could not be covered ?
• Self calibration (see Pollefeys, absolute conic,…)• Bundle adjustment• Levenberg-Marquardt• The full presentation of algorithms for the estimation of H, P, K, F, …
– see the Hartley, Zisserman book for all about “multiple view geometry”• Tracking in general, Kalman filter (two UNC Siggraph 2001Tutorials)• Several prominent variants of vision-based tracking algorithms/systems:
– KLT– Rapid, RoRapid– Condensation, ICondensation– [Lu, Hager]– [Ansar, Daniilidis]– [Wunsch, Hirzinger]– [Klein, Murray]– …
Another reference to Pollefeys:http://www.cs.unc.edu/~marc/tutorial/node159.html
interested in more detail ?2 VO “Image based measurement” WS1 LU “Image based measurement” SSseminar, project, bachelor, diploma, PhD, …