Registration for Robotics

Kurt Konolige

Willow Garage

Stanford University

Patrick Mihelich

JD Chen

James Bowman

Helen Oleynikova

Freiburg TORO group:

Giorgio Grisetti

Cyrill Stachness

Rainer Kummerle

Outline

• Applications• Overview of Registration• Front end and image matching• Visual Odometry• Place recognition• Global SBA• Extensions

Applications of Registration

• Photo tourism [Snavely, Seitz, Szeliski 2006]

Applications of Registration

• Object Modeling [Lovi et al. 2010]

• [Newcombe and Davison 2010]

Applications of Registration

• Face Tracking [WATSON: Morency 2003]

Applications of Registration

• Visual

Odometry

[Willow Garage]

Applications of Registration

• Visual

SLAM

[Willow Garage]

Outline

• Applications• Overview of Registration• Front end, image matching• Visual Odometry• Place recognition• Global SBA• Extensions

Registration Elements

Incremental

Extract keypointsCompute features

Match against local views

Estimate pose

Optimize

Global

Add new view to map

Match against global views[Place recognition]

Estimate pose

Optimize

Mapping Systems

MonoSLAM [Davison 2003]

Parallel Tracking and Mapping [Klein and Murray 2007]

View-based Maps [Konolige et al. 2009]

Map type

World points

Views and world points

Views [and implicit world points]

Map RepresentationsMap type

World points

Views and world points

Views [and implicit world points]

Covariance Matrix over pointsp0 p1 p2 …

p0 p1 p2 …

Extended Kalman Filter update

MonoSLAM

Map RepresentationsMap type

World points

Views and world points

Views [and implicit world points]

Information matrix over views and world points

Nonlinear least squares updateHDx = -g

c0 c1 c2 … p0 p1 p2 …

c0c1c2…

p0 p1 p2 …

Parallel Tracking and Mapping

Map RepresentationsMap type

World points

Views and world points

Views [and implicit world points]

Information matrix over views

c0 c1 c2 …

c0c1c2…

Nonlinear least squares updateHDx = -g

View-based Maps

Comparison of Mapping MethodsMap type

World pointsUseful in small spaces - World points for global matching - World point Covar matrix update

Not as accurate [Strasdat et al. 2010]

Views and world points

Useful in small spaces - World points for global matching - View and point Info matrix update

Highly accurateViews [and implicit world points]

Useful in larger spaces - View matching - View Info matrix update

Not as accurate

Outline

• Applications• Overview of Registration• Front end and image matching• Visual Odometry• Place recognition• Global SBA• Extensions

Interest points• FAST [Rosten and Drummond 2006]

• Harris

• SIFT [Lowe 1999]

• SUSAN, CenSure, MSER, …

• Adaptive threshold

• Gridding for spatial diversity

Feature matching• SIFT / SURF features

• Planar patches

• Random-tree signatures [Calonder et al. 2009]

• …

• Tracking with motion model

• Windowed brute-force

• KD tree nearest-neighbor

930 features 354 matched 183 inliers

Outlier detection and pose estimation• RANSAC to estimate pose

– 3 points for stereo (3D-3D)

– 3-point PnP for monocular (2D-3D)

• Outlier rejection

• Refinement – Least-squares

– Reprojection error

• Scavenging

930 features 354 matched 183 inliers

Outline

• Applications• Overview of Registration• Front end and image matching• Visual Odometry• Place recognition• Global SBA• Extensions

Visual Odometry – Sliding Window• 2-view pose estimates are unreliable

– Triangulation is narrow-baseline

• Multi-view estimates are more accurate– Triangulation is wide-baseline

– More points, wider baseline => more accuracy

• Keyframes– Many close-together views don’t add much

– Too far apart leads to low inlier count

• Sliding window of views– Track points as long as possible

– ~20 – 40 keyframes in window

Visual Odometry – Bundle AdjustmentMeasurement model:

For Gaussian distributions ( is covariance)

)(xzzz

i

iiTi

iii

Ti

ii zWzzWzzp )exp(log)|(log

2

1x

1W

[Gauss-Newton / Normal]

Sparseness of H

zWJWJJ TT x)(x

z

J

H

20 camera views 5000 featuresH is (20x6 + 5000x3)2

c0 c1 c2 … p0 p1 p2 …

c0c1c2…

p0 p1 p2 …

Visual Odometry examples[courtesy Andrew Comport, INRIA]

• Outdoor sequence in Versailles

• 1 m stereo baseline, narrow FOV

• ~400 m sequence

• Average frame distance: 0.6 m

• Max frame distance: 1.1 m

26

Visual Odometry examples[courtesy Andrew Comport, INRIA]• Indoor Willow

Garage sequence

• 10 cm stereo, wide FOV

• ~100 m sequence

• Average frame distance: 0.3 m

Outline

• Applications• Overview of Registration• Front end and image matching• Visual Odometry• Place recognition• Global SBA• Extensions

Place Recognition• K-nearest neighbor feature

matching across database of images– KD tree

[Lowe 1999,

Eade and Drummond 2008,

Williams et al. 2007]

• Bag-of-words– visual vocabulary

[Stewenius and Nister 2006,

Cummins and Newman 2008]DB

test

DB

test

Place Recognition: Vocabulary Trees[Nister and Stewenius CVPR06]

- “Bag of words” retrieval- Vocab tree created offline- For recognition:

- Image keypoints extracted- Tree encodes approximate NN

search- Inverted index of images at leaves

[Image from Nister and Stewenius CVPR06]

[Cummins and Newman ICRA07 Cullmer et al. ACRA08 Fraundorfer et al. IROS07]

Place Recognition: Vocabulary Trees

Performance on Indoor dataset

View-based Maps [Konolige et al. 2009]

Performance on Indoor dataset

View-based Maps [Konolige et al. 2009]

Performance on Indoor dataset

Trajectory synthesis

Outline

• Applications• Overview of Registration• Front end and image matching• Visual Odometry• Place recognition• Global SBA• Extensions

Comparison of Mapping MethodsMap type

World pointsUseful in small spaces - World points for global matching - World point Covar matrix update

Not as accurate [xxx and Davison 2010]

Views and world points

Useful in small spaces - World points for global matching - View and point Info matrix update

Highly accurateViews [and implicit world points]

Useful in larger spaces - View matching - View Info matrix update

Not as accurate

Fast SBA[Konolige BMVC 2010, Jeong et al. CVPR 2010]

• Take advantage of sparse secondary structure of Hessian

• Use fast linear solvers– Davis’ CHOLMOD

– Block preconditioned conjugate gradient

Fast SBA in VSLAM

Fast SBA in Reconstruction

Open-Source in ROS

• sba package for fast SBA• frame_common and posest for front-end

and pose estimation• vocabulary_tree for place recognition• vslam_system for Visual Odometry and

VSLAM