Previously

Previously

Two view geometry: epipolar geometry Stereo vision: 3D reconstruction

epipolar linesepipolar linesepipolar linesepipolar lines

BaselineBaselineOO O’O’

epipolar planeepipolar plane

' 0Tp Ep

Today

Orthographic projection Two views 3 Views: geometric interpretation >3 Views: factorization – simultaneous recovery

of motion and shape

Orthographic Projection (Reminder) Parallel projection rays,

orthogonal to image plane Focal center at infinity

x X

y Y

Two Views

Implies that

Eliminating Z

Since

Therefore

23 13 23 11 13 21 23 12 13 22 23 13' ' ( ) ( ) ( )x yr x r y r r r r x r r r r y r t r t

Q RP t

11 12 13

21 22 23

'

'x

y

x r x r y r z t

y r x r y r z t

32 23 11 13 21

31 23 12 13 22

( )r r r r r

r r r r r

23 13 32 31 23 13' ' ( ) 0x yr x r y r x r y r t r t

TR R I

Epipolar lines

This is a linear equation

23

13

32 31 23 13

0 0

' ' 1 0 0 0

( ) 1x y

r x

x y r y

r r r t r t

23 13 32 31 23 13' ' ( ) 0x yr x r y r x r y r t r t

Further Simplification

Select one point in first image and its corresponding point in the second image to be the origin of the two images

In this coordinate frame translation is 0 Expression for epipolar lines:

23 13 32 31' ' 0r x r y r x r y

Epipolar Line Recovery

We need 4 corresponding points:

1 to eliminate translation 3 to determine the 4 components of R up to scale The rest of the components cannot be determined In particular, cannot be determined from

, because these components are known only up to scale

23 13 32 31' ' 0r x r y r x r y

2 2 213 23 33 1r r r

33r

Shape Recovery from Two Views Perspective:

Translation recovered up to scale 3D shape recovered up to scale Recovery only if non-zero translation No calibration – recovery up to a projective transformation

(“projective shape”) Orthographic:

Rotation along epipolar line cannot be recovered 3D shape cannot be recovered Recovery is possible up to an affine transformation

(“affine shape”) Recovery only if non-trivial rotation Translation along line at infinity = rotation

Recovery from Three Views

Under orthographic projection metric recovery is possible from three views

Only rotation matters Rotation has three degrees of freedom Given an image, one rotation is in the image and

two are out of plane rotations Ignoring the in-plane rotation we can associate

the image with a point on the unit sphere


Im1

Im2

Im3


a

b

c

Im1

Im2

Im3


a, b, c are unknown –

rotation angles are known –

angles between epipolar lines Can we determine lengths

from angles?

a

b

c


In the plane the angles determine the sides of a triangle up to scale


On a sphere the sides are determined completely by the angles

Therefore three views determine all the components of rotation (up to reflection)

Sides (rotation angles) can be computed using the analogue of the cosine theorem on the sphere

Once the rotation is known structure can be recovered

a

b

c

Factorization

Simultaneous recovery of shape and motion Input:

A video sequence Tracked feature points

Assumptions: Rigid scene Orthographic projection All tracked points appear in all frames

Observation: tracked point locations satisfy linear relations that can be exploited for robust recovery

Singular Value Decomposition (SVD) Every real matrix can be decomposed to a

product of three matrices:

With diagonal U,V orthonormal

U orthonormal basis to row space of M V orthonormal basis to column space of M

nmM

Tnnnmmmnm VUM

,T TU U I V V I

Singular Value Decomposition (SVD)

1

2

0 0 0 0 0

0 0 0 0 0

0 0 0 0 0m

1

2

0 0 0

0 0 0

0 0 0

0 0 0 0

0 0 0 0

n

or

0...321 k

are called the singular values i

Tnnnmmmnm VUM

Relation to Eigenvalue Problems

2

2

2

2

2

2

( )( )

( )( )

T T T T

T

Ti i i

T T T T

T

Ti i i

M M V U U V V V

M MV V

M Mv v

MM U V V U U U

MM U U

MM u u

Singular Value Decomposition (SVD)Rank k least squares approximation of M Example: k=3

Take the 3 largest singular values:

Rank 3 approximation of M:

1

2

3

0 0

0 0

0 0

Tnmnm VUM 3333

~~~~

Factorization

Goal: given p corresponding points in f frames, compute the 3-D location of each point and the transformation between the frames

M T S

Measurements Transformation Shape (3-D locations)

Factorization

Step 1: eliminate translationSet the centroid of the points in each frame to be the

origin

Now11 12 13

21 22 23

Xr r rx

Yr r ry

Z

Factorization

11

11

y

x

Constructing M :

Factorization

1211

1211

yy

xx

Constructing M :

Factorization

p

p

yyy

xxx

11211

11211

...

...

Constructing M :

Factorization

p

p

p

p

yyy

yyy

xxx

xxx

22221

11211

22221

11211

...

...

...

...

Constructing M :

Factorization

11 12 1

1 2

11 12 1

1 2 2

...

...

...

...

p

f f fp

p

f f fp f p

x x x

x x x

y y y

y y y

Constructing M :

Factorization

p

fpffpf

p

fpf

p

ZZ

YY

XX

rrr

rrr

rrr

rrr

yy

yy

xx

xx

321

21

21

32

223

222

221

123

122

121

213

212

211

113

112

111

21

111

1

111

...

...

...

.........

.........

...

...

...

M T S

Goal: given M, find S and T

What should rank(M) be?

Factorization

Goal: given M, find S and T Compute the SVD of M

rank(M) should be 3, since rank(T)=rank(S)=3 Noise cleaning: find the rank 3 approximation of M

using the 3 largest singular values

Tpppfffpf VUM 2222

1

2

3

0 0

0 0

0 0

Tnmnm VUM 3333

~~~~

MM~

So far:

Define

The decomposition can now be written as

Factorization is not unique, since

, A invertible

Factorization

Tnmnm VUM 3333

~~~~

~~~UT TVS

~~~

STM~~~

STSAATM ˆˆ)~

()~(

~ 1

SASATT~ˆ~ˆ 1

T should contain valid rotations 3f equations, 6 unknowns:

Each row is defined as And should maintain

Factorization1 1 1

111 12 13 1

2 2 2211 12 13 2

1 1 121 22 23 1 1

2 2 221 22 23 2 2

TT

TT

T T

T T

T T

ar r r rar r r r

T T A Ar r r s b

r r r s b

,i i i iAa r Ab s

1

0

1

Ti i

Ti i

Ti i

r r

r s

s s

1

0

1

T Ti i

T Ti i

T Ti i

a A Aa

a A Ab

b A Ab

( )TR R I

Factorization

is 3x3, symmetric Linear system of equations in 6 unknowns Once B is recovered it can be factored to find A Solution is unique up to a global rotation

TB A A

TR R I

1

0

1

T Ti i

T Ti i

T Ti i

a A Aa

a A Ab

b A Ab

Input: tracked sequence

Eliminate translation, produce M

Use SVD to find the rank 3 approximation of M

Factorization ambiguous, up to invertible matrix

Find matrix A such that T contains valid rotations

Solution is unique (up to a global rotation)

Output: motion , shape

Factorization (Algorithm)

1 0 1T T T T T Ti i i i i ia A Aa a A Ab b A Ab

STSAATM ˆˆ)~

()~(

~ 1

STVUVUM TTnmnm

~~)

~~()

~~(

~~~~3333

T S

Factorization

Advantages Simultaneous recovery of shape and motion Simple algorithm, based on linear equations Robust to noise

Disadvantages Orthographic projection All points should appear in all frames

(factorization with missing data is difficult)

Summary

Shape and motion recovery under orthographic projection

Two views: Parallel epipolar lines 4 corresponding points are needed Recovery of affine shape

Three or more views Metric recovery Simultaneous recovery of shape and motion using

SVD factorization

Documents

Previously