Lecture’9’&’10:’’ Stereo’Vision’vision.stanford.edu/.../lectures/lecture9_10_stereo_cs131.pdf · Lecture 9 & 10 - !!! Fei-Fei Li! Algorithm: • Re-project image planes

Lecture 9 & 10 - !!!

Fei-Fei Li!

Lecture 9 & 10: Stereo Vision

Professor Fei-‐Fei Li Stanford Vision Lab

21-‐Oct-‐14 1

Lecture 9 & 10 - !!!

Fei-Fei Li!

Point of observation

Figures © Stephen E. Palmer, 2002

Dimensionality ReducBon Machine (3D to 2D)

3D world 2D image

Lecture 9 & 10 - !!!

Fei-Fei Li! 21-‐Oct-‐14 3

Pinhole camera

•  Common to draw image plane in front of the focal point •  Moving the image plane merely scales the image.

f f

⎪⎪⎩

⎪⎪⎨

⎧

=

=

zyf'y

zxf'x

Lecture 9 & 10 - !!!


RelaBng a real-‐world point to a point on the camera

In homogeneous coordinates:

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=

10100000000

'zyx

ff

zyfxf

P

M ideal world

Intrinsic Assumptions •  Unit aspect ratio •  Optical center at (0,0) •  No skew

Extrinsic Assumptions •  No rotation •  Camera at (0,0,0)

Lecture 9 & 10 - !!!


RelaBng a real-‐world point to a point on the camera

In homogeneous coordinates:

P ' =f xf yz

!

"

####

$

%

&&&&

=

f 0 0 00 f 0 00 0 1 0

!

"

###

$

%

&&&

xyz1

!

"

####

$

%

&&&&

= K I 0[ ]P

Intrinsic Assumptions •  Unit aspect ratio •  Optical center at (0,0) •  No skew


K

Lecture 9 & 10 - !!!

Fei-Fei Li!

Real-‐world camera

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

10100000

10

0

zyx

vus

vu

w β

α


Intrinsic Assumptions •  Optical center at (u0, v0) •  Rectangular pixels •  Small skew

Intrinsic parameters

P ' = K I 0!"

#$P

Slide inspiraBon: S. Savarese

Lecture 9 & 10 - !!!

Fei-Fei Li!

Real-‐world camera + Real-‐world transformaBon

Extrinsic Assumptions •  Allow rotation •  Camera at (tx,ty,tz)

Intrinsic Assumptions •  Optical center at (u0, v0) •  Rectangular pixels •  Small skew

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

1100

01 333231

232221

131211

0

0

zyx

trrrtrrrtrrr

vus

vu

w

z

y

x

β

α

P ' = K R t[ ] P

Slide inspiraBon: S. Savarese

Lecture 9 & 10 - !!!

Fei-Fei Li!

What we will learn today?

•  IntroducBon to stereo vision •  Epipolar geometry: a gentle intro •  Parallel images & image recBficaBon •  Solving the correspondence problem •  Homographic transformaBon •  AcBve stereo vision system

21-‐Oct-‐14 8

Reading: [HZ] Chapters: 4, 9, 11 [FP] Chapters: 10

Lecture 9 & 10 - !!!

Fei-Fei Li!



21-‐Oct-‐14 9


Lecture 9 & 10 - !!!

Fei-Fei Li!

Recovering 3D from Images

•  How can we automaBcally compute 3D geometry from images? – What cues in the image provide 3D informaBon?

Point of observation

Real 3D world 2D image

? 21-‐Oct-‐14 10

Lecture 9 & 10 - !!!

Fei-Fei Li!

•  Shading

Visual Cues for 3D

Merle Norman Cosme5cs, Los Angeles

Slide cred

it: J. Hayes

21-‐Oct-‐14 11

Lecture 9 & 10 - !!!

Fei-Fei Li!

Visual Cues for 3D

•  Shading

•  Texture

The Visual Cliff, by William Vandivert, 1960

Slide cred

it: J. Hayes

21-‐Oct-‐14 12

Lecture 9 & 10 - !!!

Fei-Fei Li!

Visual Cues for 3D

From The Art of Photography, Canon

•  Shading

•  Texture

•  Focus

Slide cred

it: J. Hayes

21-‐Oct-‐14 13

Lecture 9 & 10 - !!!

Fei-Fei Li!

Visual Cues for 3D

•  Shading

•  Texture

•  Focus

•  MoBon

Slide cred

it: J. Hayes

21-‐Oct-‐14 14

Lecture 9 & 10 - !!!

Fei-Fei Li!

Visual Cues for 3D

•  Others: –  Highlights –  Shadows –  Silhoueaes –  Inter-‐reflecBons –  Symmetry –  Light PolarizaBon –  ...

•  Shading

•  Texture

•  Focus

•  MoBon Shape From X

•  X = shading, texture, focus, moBon, ... •  We’ll focus on the moBon cue

Slide cred

it: J. Hayes

21-‐Oct-‐14 15

Lecture 9 & 10 - !!!

Fei-Fei Li!

Stereo ReconstrucBon

•  The Stereo Problem –  Shape from two (or more) images –  Biological moBvaBon

known camera

viewpoints

Slide cred

it: J. Hayes

21-‐Oct-‐14 16

Lecture 9 & 10 - !!!

Fei-Fei Li!

Why do we have two eyes?

Cyclope vs. Odysseus

Slide cred

it: J. Hayes

21-‐Oct-‐14 17

Lecture 9 & 10 - !!!

Fei-Fei Li!

1. Two is beaer than one

Slide cred

it: J. Hayes

21-‐Oct-‐14 18

Lecture 9 & 10 - !!!

Fei-Fei Li!

2. Depth from Convergence

Human performance: up to 6-8 feet

Slide cred

it: J. Hayes

21-‐Oct-‐14 19

Lecture 9 & 10 - !!!

Fei-Fei Li!



21-‐Oct-‐14 20


Lecture 9 & 10 - !!!


Epipolar geometry

•  Epipolar Plane •  Epipoles e, e’

•  Epipolar Lines •  Baseline

O O’

p’

P

p

e e’

= intersecBons of baseline with image planes = projecBons of the other camera center = vanishing points of camera moBon direcBon

Lecture 9 & 10 - !!!


Example: Converging image planes

Lecture 9 & 10 - !!!


Epipolar Constraint

-‐  Two views of the same object -‐  Suppose I know the camera posiBons and camera matrices -‐ Given a point on lem image, how can I find the corresponding point on right image?

Lecture 9 & 10 - !!!


Epipolar Constraint

•  PotenBal matches for p have to lie on the corresponding epipolar line l’.

•  PotenBal matches for p’ have to lie on the corresponding epipolar line l.

Lecture 9 & 10 - !!!


Epipolar Constraint

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=→

1vu

PMp

O O’

p p’

P

R, T

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡ʹ′

ʹ′

=ʹ′→

1vu

PMp

M = K I 0!"

#$ M ' = K ' R T!

"#$

Lecture 9 & 10 - !!!


Epipolar Constraint

O O’

p p’

P

R, T

K1 and K2 are known (calibrated cameras)

[ ]0IM = [ ]TRM ='

M = K I 0!"

#$ M ' = K ' R T!

"#$

Lecture 9 & 10 - !!!


Epipolar Constraint

O O’

p p’

P

R, T

[ ] 0)pR(TpT =ʹ′×⋅Perpendicular to epipolar plane

)( pRT ʹ′×

Lecture 9 & 10 - !!!


Cross product as matrix mulBplicaBon

baba ][0

00

×=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−

−

−

=×

z

y

x

xy

xz

yz

bbb

aaaaaa

“skew symmetric matrix”

Lecture 9 & 10 - !!!


Epipolar Constraint

O O’

p p’

P

R, T

[ ] 0)pR(TpT =ʹ′×⋅ [ ] 0pRTpT =ʹ′⋅⋅→ ×

E = essenBal matrix (Longuet-‐Higgins, 1981)

Lecture 9 & 10 - !!!


Epipolar Constraint

•  E p’ is the epipolar line associated with p’ (l = E p’) •  ET p is the epipolar line associated with p (l’ = ET p) •  E is singular (rank two) •  E e’ = 0 and ET e = 0 •  E is 3x3 matrix; 5 DOF

O O’

p’

P

p

e e’

l l’

Lecture 9 & 10 - !!!

Fei-Fei Li!



21-‐Oct-‐14 31


Lecture 9 & 10 - !!!

Fei-Fei Li!

Simplest Case: Parallel images •  Image planes of cameras

are parallel to each other and to the baseline

•  Camera centers are at same height

•  Focal lengths are the same

Slide cred

it: J. Hayes

21-‐Oct-‐14 32

Lecture 9 & 10 - !!!

Fei-Fei Li!

•  Image planes of cameras are parallel to each other and to the baseline

•  Camera centers are at same height

•  Focal lengths are the same •  Then, epipolar lines fall

along the horizontal scan lines of the images

Simplest Case: Parallel images

Slide cred

it: J. Hayes

21-‐Oct-‐14 33

Lecture 9 & 10 - !!!

Fei-Fei Li!

Essential matrix for parallel images

pTE !p = 0, E = [t× ]R

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−== ×

0000

000][

TTRtE

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−

−

−

=×

00

0][

xy

xz

yz

aaaaaa

a

R = I t = (T, 0, 0)

Epipolar constraint:

t

p

p’

Reminder: skew symmetric matrix

Slide cred

it: J. Hayes

21-‐Oct-‐14 34

Lecture 9 & 10 - !!!

Fei-Fei Li!

( ) ( ) vTTvvTTvuv

u

TTvu ʹ′==

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

ʹ′

−=⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛ʹ′

ʹ′

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

− 00

10100

00000

1

The y-coordinates of corresponding points are the same!

t

p

p’

Essential matrix for parallel images

pTE !p = 0, E = [t× ]R

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−== ×

0000

000][

TTRtE

R = I t = (T, 0, 0)

Epipolar constraint:

Slide cred

it: J. Hayes

21-‐Oct-‐14 35

Lecture 9 & 10 - !!!

Fei-Fei Li!

Triangulation -- depth from disparity

f

p p’

Baseline B

z

O O’

P

f

disparity = u− "u = B ⋅ fz

Disparity is inversely proportional to depth!

Slide cred

it: J. Hayes

21-‐Oct-‐14 36

Lecture 9 & 10 - !!!

Fei-Fei Li!

Stereo image rectification

Slide cred

it: J. Hayes

21-‐Oct-‐14 37

Lecture 9 & 10 - !!!

Fei-Fei Li!

Algorithm: •  Re-project image planes onto a

common plane parallel to the line between optical centers

•  Pixel motion is horizontal after this transformation

•  Two transformation matrices, one for each input image reprojection

Ø  C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.

Stereo image rectification

Slide cred

it: J. Hayes

21-‐Oct-‐14 38

Lecture 9 & 10 - !!!

Fei-Fei Li!

Rectification example

21-‐Oct-‐14 39

Lecture 9 & 10 - !!!


Applica5on: view morphing S. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 1996, 21-‐30

Lecture 9 & 10 - !!!


Applica5on: view morphing

Lecture 9 & 10 - !!!



Lecture 9 & 10 - !!!



Lecture 9 & 10 - !!!

Fei-Fei Li!

Removing perspective distortion

Hp

(rectification)

21-‐Oct-‐14 44

Lecture 9 & 10 - !!!

Fei-Fei Li!



21-‐Oct-‐14 45


Lecture 9 & 10 - !!!

Fei-Fei Li!

Stereo matching: solving the correspondence problem

•  Goal: finding matching points between two images

21-‐Oct-‐14 46

Lecture 9 & 10 - !!!

Fei-Fei Li!

Basic stereo matching algorithm

•  For each pixel in the first image –  Find corresponding epipolar line in the right image –  Examine all pixels on the epipolar line and pick the best match –  Triangulate the matches to get depth information

•  Simplest case: epipolar lines are scanlines

–  When does this happen?

Slide cred

it: J. Hayes

21-‐Oct-‐14 47

Lecture 9 & 10 - !!!

Fei-Fei Li!

Basic stereo matching algorithm

•  If necessary, rectify the two stereo images to transform epipolar lines into scanlines

•  For each pixel x in the first image –  Find corresponding epipolar scanline in the right image –  Examine all pixels on the scanline and pick the best match x’ –  Compute disparity x-x’ and set depth(x) = 1/(x-x’)

corresponding

21-‐Oct-‐14 48

Lecture 9 & 10 - !!!

Fei-Fei Li!

•  Let’s make some assumptions to simplify the matching problem –  The baseline is relatively small (compared to the

depth of scene points) –  Then most scene points are visible in both views –  Also, matching regions are similar in appearance

Correspondence problem

Slide cred

it: J. Hayes

21-‐Oct-‐14 49

Lecture 9 & 10 - !!!

Fei-Fei Li!

Matching cost

disparity

Left Right

scanline

Correspondence search with similarity constraint

•  Slide a window along the right scanline and compare contents of that window with the reference window in the left image

•  Matching cost: SSD or normalized correlation

Slide cred

it: J. Hayes

21-‐Oct-‐14 50

Lecture 9 & 10 - !!!

Fei-Fei Li!

Left Right

scanline


SS

D (s

um o

f squ

ared

diff

eren

ces)

21-‐Oct-‐14 51

Lecture 9 & 10 - !!!

Fei-Fei Li!

Left Right

scanline


Nor

mal

ized

cor

rela

tion

21-‐Oct-‐14 52

Lecture 9 & 10 - !!!

Fei-Fei Li!

Effect of window size

– Smaller window +  More detail •  More noise

–  Larger window

+  Smoother disparity maps •  Less detail

W = 3 W = 20

21-‐Oct-‐14 53

Lecture 9 & 10 - !!!

Fei-Fei Li!

The similarity constraint

•  Corresponding regions in two images should be similar in appearance

•  …and non-corresponding regions should be different •  When will the similarity constraint fail?

Slide cred

it: J. Hayes

21-‐Oct-‐14 54

Lecture 9 & 10 - !!!

Fei-Fei Li!

Limitations of similarity constraint

Textureless surfaces Occlusions, repetition

Specular surfaces Slide cred

it: J. Hayes

21-‐Oct-‐14 55

Lecture 9 & 10 - !!!

Fei-Fei Li!

Results with window search

Window-based matching Ground truth

Left image Right image

21-‐Oct-‐14 56

Lecture 9 & 10 - !!!

Fei-Fei Li!

Better methods exist... (CS231a)

Graph cuts Ground truth

Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001

21-‐Oct-‐14 57

Lecture 9 & 10 - !!!

Fei-Fei Li!

The role of the baseline

• Small baseline: large depth error • Large baseline: difficult search problem

Large Baseline Small Baseline

Slide credit: S. Seitz

21-‐Oct-‐14 58

Lecture 9 & 10 - !!!

Fei-Fei Li!

Problem for wide baselines: Foreshortening

•  Matching with fixed-size windows will fail! •  Possible solution: adaptively vary window size •  Another solution: model-based stereo (CS231a)

Slide credit: J. Hayes

21-‐Oct-‐14 59

Lecture 9 & 10 - !!!

Fei-Fei Li!


•  IntroducBon to stereo vision •  Epipolar geometry: a gentle intro •  Parallel images •  Solving the correspondence problem •  Homographic transformaBon •  AcBve stereo vision system

21-‐Oct-‐14 60


Lecture 9 & 10 - !!!

Fei-Fei Li!

Reminder: transformaBons in 2D

Special case from lecture 2 (planar rotaBon & translaBon)

u 'v '1

!

"

###

$

%

&&&= R t

0 1

!

"#

$

%&

uv1

!

"

###

$

%

&&&= He

uv1

!

"

###

$

%

&&&

-  3 DOF -  Preserve distance (areas) -  Regulate motion of rigid object

Lecture 9 & 10 - !!!

Fei-Fei Li!

u 'v '1

!

"

###

$

%

&&&=

a1 a2 a3a4 a5 a6a7 a8 a9

!

"

####

$

%

&&&&

uv1

!

"

###

$

%

&&&= H

uv1

!

"

###

$

%

&&&

-  8 DOF -  Preserve colinearity

Reminder: transformaBons in 2D

Generic case (rotaBon in 3D, scale & translaBon)

Lecture 9 & 10 - !!!


Goal: esBmate the homographic transformaBon between two images

H

AssumpBon: Given a set of corresponding points.

Lecture 9 & 10 - !!!


Goal: esBmate the homographic transformaBon between two images

H

AssumpBon: Given a set of corresponding points.

QuesBon: How many points are needed?

Hint: DoF for H? 8!

At least 4 points (8 equaBons)

Lecture 9 & 10 - !!!

Fei-Fei Li!

DLT algorithm (Direct Linear Transformation)

pi !pi

!pi = H pi

21-‐Oct-‐14 65

H

Lecture 9 & 10 - !!!

Fei-Fei Li!

DLT algorithm (direct Linear Transformation)

!pi ×H pi = 0

9x1

0Ai =hFunction of measurements

Unknown [9x1]

[2x9]

2 independent equaBons

h =

h1h2h9

!

"

#####

$

%

&&&&&

21-‐Oct-‐14 66

H =

h1 h2 h3h4 h5 h6h7 h8 h9

!

"

####

$

%

&&&&

Lecture 9 & 10 - !!!

Fei-Fei Li!


0hAi =9x2A

0hA1 =0hA2 =

0hAN =

0hA 199N2 =××

1x9h

Over determined Homogenous system

21-‐Oct-‐14 67

pi !pi

H

Lecture 9 & 10 - !!!

Fei-Fei Li!


0hA 199N2 =××How to solve ?

Singular Value Decomposition (SVD)!

21-‐Oct-‐14 68

Lecture 9 & 10 - !!!

Fei-Fei Li!


0hA 199N2 =××

999992 ××× Σ Tn VU

Last column of V gives h! H!

How to solve ?

Singular Value Decomposition (SVD)!

Why? See pag 593 of AZ

21-‐Oct-‐14 69

Lecture 9 & 10 - !!!

Fei-Fei Li!


How to solve ? 0hA 199N2 =××

21-‐Oct-‐14 70

Lecture 9 & 10 - !!!

Fei-Fei Li!

Clarification about SVD Tnnnnnmnm VDUP ×××× =

Tnnnmmmnm VDUP ×××× =

Has n orthogonal columns

Orthogonal matrix

•  This is one of the possible SVD decompositions •  This is typically used for efficiency •  The classic SVD is actually:

orthogonal Orthogonal

21-‐Oct-‐14 71

Lecture 9 & 10 - !!!


H

Lecture 9 & 10 - !!!

Fei-Fei Li!



21-‐Oct-‐14 73


Lecture 9 & 10 - !!!


Ac5ve stereo (point)

Replace one of the two cameras by a projector -‐  Single camera -‐  Projector geometry calibrated -‐  What’s the advantage of having the projector?

P

O2

p’

Correspondence problem solved!

projector

Lecture 9 & 10 - !!!


Ac5ve stereo (stripe)

O -‐ Projector and camera are parallel -‐  Correspondence problem solved!

projector

Lecture 9 & 10 - !!!


Ac5ve stereo (shadows)

Light source O

-‐  1 camera,1 light source -‐  very cheap setup -‐  calibrated light source

J. Bouguet & P. Perona, 99 S. Savarese, J. Bouguet & Perona, 00

Lecture 9 & 10 - !!!


Ac5ve stereo (shadows) J. Bouguet & P. Perona, 99 S. Savarese, J. Bouguet & Perona, 00

Lecture 9 & 10 - !!!


Ac5ve stereo (color-‐coded stripes)

-‐ Dense reconstrucBon -‐  Correspondence problem again -‐  Get around it by using color codes

projector O

L. Zhang, B. Curless, and S. M. Seitz 2002 S. Rusinkiewicz & Levoy 2002

Lecture 9 & 10 - !!!


Ac5ve stereo (color-‐coded stripes)

L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape AcquisiBon Using Color Structured Light and MulB-‐pass Dynamic Programming. 3DPVT 2002

Rapid shape acquisiBon: Projector + stereo cameras

Lecture 9 & 10 - !!!


Ac5ve stereo (stripe)

•  OpBcal triangulaBon –  Project a single stripe of laser light –  Scan it across the surface of the object –  This is a very precise version of structured light scanning

Digital Michelangelo Project http://graphics.stanford.edu/projects/mich/

Lecture 9 & 10 - !!!

Fei-Fei Li!

Laser scanned models

The Digital Michelangelo Project, Levoy et al. Slide credit: S. Seitz

21-‐Oct-‐14 81

Lecture 9 & 10 - !!!

Fei-Fei Li!


21-‐Oct-‐14

The Digital Michelangelo Project, Levoy et al.


82

Lecture 9 & 10 - !!!

Fei-Fei Li!




21-‐Oct-‐14 83

Lecture 9 & 10 - !!!

Fei-Fei Li!




21-‐Oct-‐14 84

Lecture 9 & 10 - !!!

Fei-Fei Li!

1.0 mm resoluBon (56 million triangles)




21-‐Oct-‐14 85

Lecture 9 & 10 - !!!

Fei-Fei Li!



21-‐Oct-‐14 86


Documents

Lecture’9’&’10:’’ Stereo’Vision’vision.stanford.edu/.../lectures/lecture9_10_stereo_cs131.pdf · Lecture 9 & 10 - !!! Fei-Fei Li! Algorithm: • Re-project image planes