1

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

6/1

7

Computer Vision – Lecture 19

Uncalibrated Reconstruction

23.01.2017

Bastian Leibe

RWTH Aachen

http://www.vision.rwth-aachen.de

leibe@vision.rwth-aachen.de Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

6/1

7

Course Outline

• Image Processing Basics

• Segmentation & Grouping

• Object Recognition

• Local Features & Matching

• Object Categorization

• 3D Reconstruction

Epipolar Geometry and Stereo Basics

Camera calibration & Uncalibrated Reconstruction

Active Stereo

Structure-from-Motion

• Motion and Tracking

3

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Recap: A General Point

• Equations of the form

• How do we solve them? (always!)

Apply SVD

Singular values of A = square roots of the eigenvalues of ATA.

The solution of Ax=0 is the nullspace vector of A.

This corresponds to the smallest singular vector of A. 4

Ax 0

11 11 1

1

T

N

T

NN N NN

d v v

d v v

A UDV U

SVD

Singular values Singular vectors

B. Leibe

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6Recap: Camera Parameters

• Intrinsic parameters

Principal point coordinates

Focal length

Pixel magnification factors

Skew (non-rectangular pixels)

Radial distortion

• Extrinsic parameters

Rotation R

Translation t

(both relative to world coordinate system)

• Camera projection matrix

General pinhole camera: 9 DoF

CCD Camera with square pixels: 10 DoF

General camera: 11 DoF 5B. Leibe

0

0

1 1 1

x x x

y y y

m f p x

K m f p y

s s

P K R | t

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Recap: Calibrating a Camera

Goal

• Compute intrinsic and extrinsic

parameters using observed camera

data.

Main idea

• Place “calibration object” with

known geometry in the scene

• Get correspondences

• Solve for mapping from scene to

image: estimate P=PintPext

6B. LeibeSlide credit: Kristen Grauman

? P

Xi

xi

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Recap: Camera Calibration (DLT Algorithm)

• P has 11 degrees of freedom.

• Two linearly independent equations per independent

2D/3D correspondence.

• Solve with SVD (similar to homography estimation)

Solution corresponds to smallest singular vector.

• 5 ½ correspondences needed for a minimal solution.7

B. LeibeSlide adapted from Svetlana Lazebnik

0pA 0

P

P

P

X0X

XX0

X0X

XX0

3

2

1111

111

T

nn

TT

n

T

nn

T

n

T

TTT

TTT

x

y

x

y

• Solve with …?

2

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Topics of This Lecture

• Revisiting Epipolar Geometry Triangulation

Calibrated case: Essential matrix

Uncalibrated case: Fundamental matrix

Weak calibration

Epipolar Transfer

• Active Stereo Kinect sensor

Structured Light sensing

Laser scanning

9B. Leibe

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Two-View Geometry

• Scene geometry (structure):

Given corresponding points in two or more images, where is the

pre-image of these points in 3D?

• Correspondence (stereo matching):

Given a point in just one image, how does it constrain the

position of the corresponding point x’ in another image?

• Camera geometry (motion):

Given a set of corresponding points in two images, what are the

cameras for the two views?

10B. LeibeSlide credit: Svetlana Lazebnik

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Revisiting Triangulation

• Given projections of a 3D point in two or more images

(with known camera matrices), find the coordinates of

the point

11B. Leibe

O1O2

x1x2

X?

Slide credit: Svetlana Lazebnik

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6Revisiting Triangulation

• We want to intersect the two visual rays corresponding

to x1 and x2, but because of noise and numerical errors,

they will never meet exactly. How can this be done?

12B. Leibe

O1O2

x1x2

X?

Slide credit: Svetlana Lazebnik

R1R2

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Triangulation: 1) Geometric Approach

• Find shortest segment connecting the two viewing rays

and let X be the midpoint of that segment.

13B. Leibe

O1O2

x1x2

X

Slide credit: Svetlana Lazebnik

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Triangulation: 2 )Linear Algebraic Approach

14B. Leibe

baba ][

0

0

0

z

y

x

xy

xz

yz

b

b

b

aa

aa

aa

XPx

XPx

222

111

0XPx

0XPx

22

11

0XP][x

0XP][x

22

11

Cross product as matrix multiplication:

Slide credit: Svetlana Lazebnik

3

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

• This approach is often preferable to the geometric

approach, since it nicely generalizes to multiple

cameras.

Triangulation: 2) Linear Algebraic Approach

15B. Leibe

XPx

XPx

222

111

0XPx

0XPx

22

11

0XP][x

0XP][x

22

11

Slide credit: Svetlana Lazebnik

Two independent equations each in terms of

three unknown entries of X

Stack them and solve using… ? Stack them and solve using SVD!

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Triangulation: 3) Nonlinear Approach

• Find X that minimizes

16B. Leibe

),(),( 22

2

11

2 XPxdXPxd

O1O2

x1x2

X?

x’1

x’2

Slide credit: Svetlana Lazebnik

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Triangulation: 3) Nonlinear Approach

• Find X that minimizes

• This approach is the most accurate, but unlike the other

two methods, it doesn’t have a closed-form solution.

• Iterative algorithm

Initialize with linear estimate.

Optimize with Gauss-Newton or Levenberg-Marquardt

(see F&P sec. 3.1.2 or H&Z Appendix 6).

17B. Leibe

),(),( 22

2

11

2 XPxdXPxd

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6Revisiting Epipolar Geometry

• Let’s look again at the epipolar constraint

For the calibrated case (but in homogenous coordinates)

For the uncalibrated case

18B. Leibe

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Epipolar Geometry: Calibrated Case

19B. Leibe

X

x x’

Camera matrix: [I|0]X = (u, v, w, 1)T

x = (u, v, w)T

Camera matrix: [RT | –RTt]Vector x’ in second coord.

system has coordinates Rx’ in

the first one.

t

The vectors x, t, and Rx’ are coplanar

R

Slide credit: Svetlana Lazebnik

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Epipolar Geometry: Calibrated Case

20B. Leibe

X

x x’

Slide credit: Svetlana Lazebnik

Essential Matrix

(Longuet-Higgins, 1981)

0)]([ xRtx RtExExT ][with0

4

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Epipolar Geometry: Calibrated Case

• E x’ is the epipolar line associated with x’ (l = E x’)

• ETx is the epipolar line associated with x (l’ = ETx)

• E e’ = 0 and ETe = 0

• E is singular (rank two)

• E has five degrees of freedom (up to scale)21

B. Leibe

X

x x’

Slide credit: Svetlana Lazebnik

0)]([ xRtx RtExExT ][with0

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Epipolar Geometry: Uncalibrated Case

• The calibration matrices K and K’ of the two cameras are unknown

• We can write the epipolar constraint in terms of unknown normalized coordinates:

22B. Leibe

X

x x’

Slide credit: Svetlana Lazebnik

0ˆˆ xExT xKxxKx ˆ,ˆ

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Epipolar Geometry: Uncalibrated Case

23B. Leibe

X

x x’

Slide credit: Svetlana Lazebnik

Fundamental Matrix(Faugeras and Luong, 1992)

0ˆˆ xExT

xKx

xKx

ˆ

ˆ

1with0 KEKFxFx TT

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6Epipolar Geometry: Uncalibrated Case

• F x’ is the epipolar line associated with x’ (l = F x’)

• FTx is the epipolar line associated with x (l’ = FTx)

• F e’ = 0 and FTe = 0

• F is singular (rank two)

• F has seven degrees of freedom24

B. Leibe

X

x x’

Slide credit: Svetlana Lazebnik

0ˆˆ xExT 1with0 KEKFxFx TT

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

• The Fundamental matrix defines the epipolar geometry

between two uncalibrated cameras.

• How can we estimate F from an image pair?

We need correspondences…

Estimating the Fundamental Matrix

25B. Leibe

xi

Xi

x0i

P? P0? Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

The Eight-Point Algorithm

26B. Leibe

x = (u, v, 1)T, x’ = (u’, v’, 1)T

This minimizes:

2

1

)( i

N

i

T

i xFx

Slide adapted from Svetlana Lazebnik

[u0u; u0v; u0; uv0; vv0; v0; u; v; 1]

26666666666664

F11

F12

F13

F21

F22

F23

F31

F32

F33

37777777777775

= 0

266666666664

u01u1 u0

1v1 u01 u1v

01 v1v

01 v0

1 u1 v1 1

u02u2 u0

2v2 u02 u2v

02 v2v

02 v0

2 u2 v2 1

u03u3 u0

3v3 u03 u3v

03 v3v

03 v0

3 u3 v3 1

u04u4 u0

4v4 u04 u4v

04 v4v

04 v0

4 u4 v4 1

u05u5 u0

5v5 u05 u5v

05 v5v

05 v0

5 u5 v5 1

u06u6 u0

6v6 u06 u6v

06 v6v

06 v0

6 u6 v6 1

u07u7 u0

7v7 u07 u7v

07 v7v

07 v0

7 u7 v7 1

u08u8 u0

8v8 u08 u8v

08 v8v

08 v0

8 u8 v8 1

377777777775

26666666666664

F11

F12

F13

F21

F22

F23

F31

F32

F33

37777777777775

=

266666666664

0

0

0

0

0

0

0

0

377777777775

Af = 0

Solve using… ?Solve using… SVD!

• Taking 8 correspondences:

5

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Excursion: Properties of SVD

• Frobenius norm

Generalization of the Euclidean norm to matrices

• Partial reconstruction property of SVD

Let i i=1,…,N be the singular values of A.

Let Ap = UpDpVpT be the reconstruction of A when we set

p+1,…, N to zero.

Then Ap = UpDpVpT is the best rank-p approximation of A in the

sense of the Frobenius norm

(i.e. the best least-squares approximation).27

2

1 1

m n

ijFi j

A a

min( , )

2

1

m n

i

i

B. Leibe

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

The Eight-Point Algorithm

• Problem with noisy data

The solution will usually not fulfill the constraint that F only has

rank 2.

There will be no epipoles through which all epipolar lines pass!

• Enforce the rank-2 constraint using SVD

• As we have just seen, this provides the best least-

squares approximation to the rank-2 solution.

28B. Leibe

11 11 13

22

33 31 33

T

T

d v v

F d

d v v

UDV U

SVD

Set d33 to

zero and

reconstruct F

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Problem with the Eight-Point Algorithm

29B. Leibe

266666666664

u01u1 u0

1v1 u01 u1v

01 v1v

01 v0

1 u1 v1 1

u02u2 u0

2v2 u02 u2v

02 v2v

02 v0

2 u2 v2 1

u03u3 u0

3v3 u03 u3v

03 v3v

03 v0

3 u3 v3 1

u04u4 u0

4v4 u04 u4v

04 v4v

04 v0

4 u4 v4 1

u05u5 u0

5v5 u05 u5v

05 v5v

05 v0

5 u5 v5 1

u06u6 u0

6v6 u06 u6v

06 v6v

06 v0

6 u6 v6 1

u07u7 u0

7v7 u07 u7v

07 v7v

07 v0

7 u7 v7 1

u08u8 u0

8v8 u08 u8v

08 v8v

08 v0

8 u8 v8 1

377777777775

26666666666664

F11

F12

F13

F21

F22

F23

F31

F32

F33

37777777777775

=

266666666664

0

0

0

0

0

0

0

0

377777777775

Slide adapted from Svetlana Lazebnik

• In practice, this often looks as follows:

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6Problem with the Eight-Point Algorithm

• In practice, this often looks as follows:

Poor numerical conditioning

Can be fixed by rescaling the data

30B. Leibe

266666666664

u01u1 u0

1v1 u01 u1v

01 v1v

01 v0

1 u1 v1 1

u02u2 u0

2v2 u02 u2v

02 v2v

02 v0

2 u2 v2 1

u03u3 u0

3v3 u03 u3v

03 v3v

03 v0

3 u3 v3 1

u04u4 u0

4v4 u04 u4v

04 v4v

04 v0

4 u4 v4 1

u05u5 u0

5v5 u05 u5v

05 v5v

05 v0

5 u5 v5 1

u06u6 u0

6v6 u06 u6v

06 v6v

06 v0

6 u6 v6 1

u07u7 u0

7v7 u07 u7v

07 v7v

07 v0

7 u7 v7 1

u08u8 u0

8v8 u08 u8v

08 v8v

08 v0

8 u8 v8 1

377777777775

26666666666664

F11

F12

F13

F21

F22

F23

F31

F32

F33

37777777777775

=

266666666664

0

0

0

0

0

0

0

0

377777777775

Slide adapted from Svetlana Lazebnik

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

The Normalized Eight-Point Algorithm

1. Center the image data at the origin, and scale it so the

mean squared distance between the origin and the data

points is 2 pixels.

2. Use the eight-point algorithm to compute F from the

normalized points.

3. Enforce the rank-2 constraint using SVD.

4. Transform fundamental matrix back to original units: if

T and T’ are the normalizing transformations in the two

images, than the fundamental matrix in original

coordinates is TT F T’.31

B. Leibe [Hartley, 1995]Slide credit: Svetlana Lazebnik

11 11 13

22

33 31 33

T

T

d v v

F d

d v v

UDV U

SVD

Set d33 to

zero and

reconstruct F

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

The Eight-Point Algorithm

• Meaning of error

Sum of Euclidean distances between points xi and

epipolar lines Fx’i (or points x’i and epipolar lines FTxi),

multiplied by a scale factor

• Nonlinear approach: minimize

Similar to nonlinear minimization

approach for triangulation.

Iterative approach (Gauss-Newton,

Levenberg-Marquardt,…)32

B. Leibe

:)( 2

1

i

N

i

T

i xFx

N

i

i

T

iii xFxxFx1

22 ),(d),(d

O1O2

x1x2

X?

x’1

x’2

Slide credit: Svetlana Lazebnik

6

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Comparison of Estimation Algorithms

33B. Leibe

8-point Normalized 8-point Nonlinear least squares

Av. Dist. 1 2.33 pixels 0.92 pixel 0.86 pixel

Av. Dist. 2 2.18 pixels 0.85 pixel 0.80 pixel

Slide credit: Svetlana Lazebnik

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

3D Reconstruction with Weak Calibration

• Want to estimate world geometry without requiring calibrated cameras.

• Many applications: Archival videos

Photos from multiple unrelated users

Dynamic camera system

• Main idea:

Estimate epipolar geometry from a (redundant) set of point correspondences between two uncalibrated cameras.

34B. LeibeSlide credit: Kristen Grauman

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Stereo Pipeline with Weak Calibration

• So, where to start with uncalibrated cameras?

Need to find fundamental matrix F and the correspondences

(pairs of points (u’,v’) ↔ (u,v)).

• Procedure

1. Find interest points in both images

2. Compute correspondences

3. Compute epipolar geometry

4. Refine35

B. Leibe Example from Andrew ZissermanSlide credit: Kristen Grauman

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6Stereo Pipeline with Weak Calibration

1. Find interest points (e.g. Harris corners)

36B. Leibe Example from Andrew ZissermanSlide credit: Kristen Grauman

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Stereo Pipeline with Weak Calibration

2. Match points using only proximity

37B. Leibe Example from Andrew ZissermanSlide credit: Kristen Grauman

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Putative Matches based on Correlation Search

38B. Leibe Example from Andrew Zisserman

7

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

RANSAC for Robust Estimation of F

• Select random sample of correspondences

• Compute F using them

This determines epipolar constraint

• Evaluate amount of support – number of inliers within

threshold distance of epipolar line

• Choose F with most support (#inliers)

39B. LeibeSlide credit: Kristen Grauman

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Putative Matches based on Correlation Search

40B. Leibe Example from Andrew Zisserman

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Pruned Matches

• Correspondences consistent with epipolar geometry

41B. Leibe Example from Andrew Zisserman

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6Resulting Epipolar Geometry

42B. Leibe Example from Andrew Zisserman

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Epipolar Transfer

• Assume the epipolar geometry is known

• Given projections of the same point in two images, how

can we compute the projection of that point in a third

image?

43B. Leibe

x1 x2

? x3

Slide credit: Svetlana Lazebnik

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Extension: Epipolar Transfer

• Assume the epipolar geometry is known

• Given projections of the same point in two images, how

can we compute the projection of that point in a third

image?

44B. Leibe

x1 x2 x3l32l31

l31 = FT13 x1

l32 = FT23 x2

When does epipolar transfer fail?

Slide credit: Svetlana Lazebnik

8

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Topics of This Lecture

• Revisiting Epipolar Geometry Triangulation

Calibrated case: Essential matrix

Uncalibrated case: Fundamental matrix

Weak calibration

Epipolar Transfer

• Active Stereo Kinect sensor

Structured Light sensing

Laser scanning

45B. Leibe

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Microsoft Kinect – How Does It Work?

• Built-in IR

projector

• IR camera for

depth

• Regular camera

for color

46B. Leibe

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Recall: Optical Triangulation

47

O1

x1

X?

Image plane

Camera center

3D Scene point

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6Recall: Optical Triangulation

• Principle: 3D point given by intersection of two rays.

Crucial information: point correspondence

Most expensive and error-prone step in the pipeline…

48

O1 O2

x1 x2

X

Image plane

Camera center

3D Scene point

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Active Stereo with Structured Light

• Idea: Replace one camera by a projector.

Project “structured” light patterns onto the object

Simplifies the correspondence problem

49

O1 O2

x1 x2

X

Image plane

Camera center

3D Scene point

Projector

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

What the Kinect Sees…

50B. Leibe

9

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

3D Reconstruction with the Kinect

51B. Leibe

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Active Stereo with Structured Light

• Idea: Project “structured” light patterns onto the object

Simplifies the correspondence problem

Allows us to use only one camera

• The Kinect uses one such approach (“structured noise“)

What other approaches are possible?

52B. Leibe

camera

projector

Slide credit: Steve Seitz

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Laser Scanning

• Optical triangulation

Project a single stripe of laser light

Scan it across the surface of the object

This is a very precise version of structured light scanning

53B. Leibe

Digital Michelangelo Projecthttp://graphics.stanford.edu/projects/mich/

Slide credit: Steve Seitz

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6Laser Scanned Models

54B. Leibe

The Digital Michelangelo Project, Levoy et al.

Slide credit: Steve Seitz

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Laser Scanned Models

55B. Leibe

The Digital Michelangelo Project, Levoy et al.

Slide credit: Steve Seitz

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Laser Scanned Models

56B. Leibe

The Digital Michelangelo Project, Levoy et al.

Slide credit: Steve Seitz

10

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Laser Scanned Models

57B. Leibe

The Digital Michelangelo Project, Levoy et al.

Slide credit: Steve Seitz

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Laser Scanned Models

58B. Leibe

The Digital Michelangelo Project, Levoy et al.

Slide credit: Steve Seitz

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Multi-Stripe Triangulation

• To go faster, project multiple stripes

• But which stripe is which?

• Answer #1: assume surface continuity

e.g. Eyetronics’ ShapeCam

Slide credit: Szymon Rusienkiewicz

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6Multi-Stripe Triangulation

• To go faster, project multiple stripes

• But which stripe is which?

• Answer #2: colored stripes (or dots)

Slide credit: Szymon Rusienkiewicz

L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured

Light and Multi-pass Dynamic Programming. 3DPVT 2002

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Multi-Stripe Triangulation

• To go faster, project multiple stripes

• But which stripe is which?

• Answer #3: time-coded stripes

O. Hall-Holt, S. Rusienkiewicz, Stripe Boundary Codes for Real-Time Structured-

Light Scanning of Moving Objects, ICCV 2001.

Slide credit: Szymon Rusienkiewicz

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Time-Coded Light Patterns

• Assign each stripe a unique illumination code

over time [Posdamer 82]

Space

Time

Slide credit: Szymon Rusienkiewicz

11

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Better codes…

• Gray code

Neighbors only differ one bit

Slide credit: David Gallup

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

• Faster procedure by projecting continuous patterns

Project 3 sinusoid grating patterns shifted by 120 in phase.

For each pixel, compute relative phase from 3 intensities.

Recover absolute phase by adding a 2nd camera.

Phase-Shift Structured Light Scanning

64

ProjectorCameraCameras

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

A High-Speed 3D Scanner

65

Dense 3D reconstructions

at 30 fps (on the GPU )

Project patterns in

all 3 color channels

(color wheel removed)

[Weise, Leibe, Van Gool, CVPR’07]

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6Problems with Dynamically Moving Objects

• Moving objects lead to artifacts

Measurements correspond to different 3D points!

• Derived a geometric model for the error

Designed a motion compensation method

Result: Cleaned-up geometry + motion estimate!66

[Weise, Leibe, Van Gool, CVPR’07]

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Effect of Motion Compensation

67[Weise, Leibe, Van Gool, CVPR’07]

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Application: Online Model Reconstruction

68[Weise, Leibe, Van Gool, CVPR’08; 3DIM’09]

12

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Poor Man’s Scanner

69Bouget and Perona, ICCV’98

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

Slightly More Elaborate (But Still Cheap)

70B. Leibe

http://www.david-laserscanner.com/

Software freely available from Robotics Institute TU Braunschweig

Perc

eptu

al

and S

enso

ry A

ugm

ente

d C

om

puti

ng

Co

mp

ute

r V

isio

n W

S 1

5/1

6

References and Further Reading

• Background information on camera models and

calibration algorithms can be found in Chapters 6 and 7

of

• Also recommended: Chapter 9 of the same book on

Epipolar geometry and the Fundamental Matrix and

Chapter 11.1-11.6 on automatic computation of F.

B. Leibe71

R. Hartley, A. Zisserman

Multiple View Geometry in Computer Vision

2nd Ed., Cambridge Univ. Press, 2004