Registration for Robotics
Kurt Konolige
Willow Garage
Stanford University
Patrick Mihelich
JD Chen
James Bowman
Helen Oleynikova
Freiburg TORO group:
Giorgio Grisetti
Cyrill Stachness
Rainer Kummerle
Outline
• Applications• Overview of Registration• Front end and image matching• Visual Odometry• Place recognition• Global SBA• Extensions
Outline
• Applications• Overview of Registration• Front end, image matching• Visual Odometry• Place recognition• Global SBA• Extensions
Registration Elements
Incremental
Extract keypointsCompute features
Match against local views
Estimate pose
Optimize
Global
Add new view to map
Match against global views[Place recognition]
Estimate pose
Optimize
Mapping Systems
MonoSLAM [Davison 2003]
Parallel Tracking and Mapping [Klein and Murray 2007]
View-based Maps [Konolige et al. 2009]
Map type
World points
Views and world points
Views [and implicit world points]
Map RepresentationsMap type
World points
Views and world points
Views [and implicit world points]
Covariance Matrix over pointsp0 p1 p2 …
p0 p1 p2 …
Extended Kalman Filter update
Map RepresentationsMap type
World points
Views and world points
Views [and implicit world points]
Information matrix over views and world points
Nonlinear least squares updateHDx = -g
c0 c1 c2 … p0 p1 p2 …
c0c1c2…
p0 p1 p2 …
Map RepresentationsMap type
World points
Views and world points
Views [and implicit world points]
Information matrix over views
c0 c1 c2 …
c0c1c2…
Nonlinear least squares updateHDx = -g
Comparison of Mapping MethodsMap type
World pointsUseful in small spaces - World points for global matching - World point Covar matrix update
Not as accurate [Strasdat et al. 2010]
Views and world points
Useful in small spaces - World points for global matching - View and point Info matrix update
Highly accurateViews [and implicit world points]
Useful in larger spaces - View matching - View Info matrix update
Not as accurate
Outline
• Applications• Overview of Registration• Front end and image matching• Visual Odometry• Place recognition• Global SBA• Extensions
Interest points• FAST [Rosten and Drummond 2006]
• Harris
• SIFT [Lowe 1999]
• SUSAN, CenSure, MSER, …
• Adaptive threshold
• Gridding for spatial diversity
Feature matching• SIFT / SURF features
• Planar patches
• Random-tree signatures [Calonder et al. 2009]
• …
• Tracking with motion model
• Windowed brute-force
• KD tree nearest-neighbor
930 features 354 matched 183 inliers
Outlier detection and pose estimation• RANSAC to estimate pose
– 3 points for stereo (3D-3D)
– 3-point PnP for monocular (2D-3D)
• Outlier rejection
• Refinement – Least-squares
– Reprojection error
• Scavenging
930 features 354 matched 183 inliers
Outline
• Applications• Overview of Registration• Front end and image matching• Visual Odometry• Place recognition• Global SBA• Extensions
Visual Odometry – Sliding Window• 2-view pose estimates are unreliable
– Triangulation is narrow-baseline
• Multi-view estimates are more accurate– Triangulation is wide-baseline
– More points, wider baseline => more accuracy
• Keyframes– Many close-together views don’t add much
– Too far apart leads to low inlier count
• Sliding window of views– Track points as long as possible
– ~20 – 40 keyframes in window
Visual Odometry – Bundle AdjustmentMeasurement model:
For Gaussian distributions ( is covariance)
)(xzzz
i
iiTi
iii
Ti
ii zWzzWzzp )exp(log)|(log
2
1x
1W
[Gauss-Newton / Normal]
Sparseness of H
zWJWJJ TT x)(x
z
J
H
20 camera views 5000 featuresH is (20x6 + 5000x3)2
c0 c1 c2 … p0 p1 p2 …
c0c1c2…
p0 p1 p2 …
Visual Odometry examples[courtesy Andrew Comport, INRIA]
• Outdoor sequence in Versailles
• 1 m stereo baseline, narrow FOV
• ~400 m sequence
• Average frame distance: 0.6 m
• Max frame distance: 1.1 m
26
Visual Odometry examples[courtesy Andrew Comport, INRIA]• Indoor Willow
Garage sequence
• 10 cm stereo, wide FOV
• ~100 m sequence
• Average frame distance: 0.3 m
Outline
• Applications• Overview of Registration• Front end and image matching• Visual Odometry• Place recognition• Global SBA• Extensions
Place Recognition• K-nearest neighbor feature
matching across database of images– KD tree
[Lowe 1999,
Eade and Drummond 2008,
Williams et al. 2007]
• Bag-of-words– visual vocabulary
[Stewenius and Nister 2006,
Cummins and Newman 2008]DB
test
DB
test
Place Recognition: Vocabulary Trees[Nister and Stewenius CVPR06]
- “Bag of words” retrieval- Vocab tree created offline- For recognition:
- Image keypoints extracted- Tree encodes approximate NN
search- Inverted index of images at leaves
[Image from Nister and Stewenius CVPR06]
[Cummins and Newman ICRA07 Cullmer et al. ACRA08 Fraundorfer et al. IROS07]
Outline
• Applications• Overview of Registration• Front end and image matching• Visual Odometry• Place recognition• Global SBA• Extensions
Comparison of Mapping MethodsMap type
World pointsUseful in small spaces - World points for global matching - World point Covar matrix update
Not as accurate [xxx and Davison 2010]
Views and world points
Useful in small spaces - World points for global matching - View and point Info matrix update
Highly accurateViews [and implicit world points]
Useful in larger spaces - View matching - View Info matrix update
Not as accurate
Fast SBA[Konolige BMVC 2010, Jeong et al. CVPR 2010]
• Take advantage of sparse secondary structure of Hessian
• Use fast linear solvers– Davis’ CHOLMOD
– Block preconditioned conjugate gradient