1

Basic Single-View Geometry

2

Finite cameras

Man Drawing a Lute (The Draughtsman of the Lute), woodcut 1525, Albrecht Dürer

3

The basic pinhole model Central projection of pts in space

onto a plane Center of projection in the origin Plane of projection

(principal/image plane) in z=f Principal Point in (0,0, f)

=

10100000000

1ZYX

ff

ZfYfX

ZYX

( )[ ]0IPPXx

| 1,,diag ff==

∝

=

ZfYfX

ZYfZXf

vu

11

Camera Center, Optical Center

Principal Plane

Principal Point Principal Ray,

Optical axis

4

Principal Point offset

K: camera calibration matrix Xcam: camera located at the origin of a coordinate

system with principal axis aligned with the z axis

=

++

∝

+

+

=

101000000

111

ZYX

pfpf

ZZpfYZpfX

pZYf

pZXf

vu

ZYX

y

x

y

x

y

x

[ ] [ ]0I0IKP

PXx

|100

00

|

==

=

y

x

pfpf

cam

Position of the Principal Point ( px,py,f )

5

Camera Rotation and Translation (1)

xcam

ycam

zcam

C

O

Z

X Y

R,t

( )CRt

tXRCXRX~

~~~~

−=

+=−=cam

6

Camera Rotation and Translation (2)

In a world coordinate frame:

General mapping given by a pinhole camera 9 dof (3 for K, 3 for R, and 3 for C)

( ) [ ] [ ] XCRRCRRXCXRX ~

1

~~~~camcam −=

−=⇒−=ZYX

[ ] [ ] | ~ | tRKCIKRP

PXx

=−=

=

7

Digital cameras

Intrinsic camera parameters αx=fmx, αy =fmy mx, my : no. of pixels per unit distance s : skew (usually zero) x0 =mxpx, y0 =mypy coords of the principal pt in

terms of pixel dimensions A total of 10 d.o.f.

4 intrinsic 6 extrinsic (camera position and orientation)

[ ]

==

=

1000 , ~ | 0

0

yxs

- y

x

αα

KCIKRP

PXx

s≠0

8

Decomposition of the camera matrix Problem: given the projection matrix P, find the (intrinsic

and extrinsic) camera parameters Camera center

Point C for which PC=0 It can be shown that

Position, orientation and intrinsic parameters

Decomposition of M=KR done with the “RQ-decomposition” method (orthogonal matrix times upper triangular matrix)

• No ambiguities if K is set to have positive diagonal entries

[ ]( )[ ]( )[ ]( )[ ]( )

=

=

321

421

431

432

,,det,,det,,det,,det

pppppppppppp

C

tzyx

[ ] [ ] KRMCIMCMMP === , ~ | ~ | --

9

Euclidean vs projective spaces The 3D world is Euclidean and so is the image coordinate

system Projective geometry is used for turning central projection

into a linear operator Directions are pts on π∞ Homogeneous coordinates

Why not use a projective coordinate frame at all times? Tf from the projective world coordinate frame to the camera

coordinate frame is a 4x4 homogeneous matrix Map from P3 to P2 is a 3x4 matrix of rank 3

HMP

=

010000100001

4x4 homography 3x3 homography

10

Vanishing points

11

Vanishing line

12

Vanishing lines and points

13

Computation of the projection matrix Given a number of correspondences between world pts Xi ∈P3 and image pts xi

∈P2, compute the 3x4 matrix P such that xi=PXi

DLT (Direct Linear Transformation) algorithm Since the equation xi=PXi is homogeneous, xi and PXi are not equal. They

have the same direction but they may differ in magnitude, therefore we can write

We will use this expression to produce a linear estimation algorithm for the projection matrix

0=× ii PXx

14

DLT method for the estimation of P The j-th row of P is denoted by pj T (4-vector)

since pj Txi= xiT pj, we can write

0pA

0ppp

XXXX

XX

=

=

−−

−

i

Tii

Tii

Tii

Tii

Tii

Tii

xyxz

yz

3

2

1

00

0

( )

−−

−==×⇒

= ×

iT

iT

iT

ii

ii

ii

iiii

iT

iT

iT

i

xyxz

yz

XpXpXp

PXxPXxXpXpXp

PX3

2

1

3

2

1

00

0

15

DLT method for the estimation of P Only two of the three eqs are linearly independent,

therefore it can be rewritten as

Pay attention to which row you drop! E.g. If one of the pts is at infinity (zi=0), then we should not drop the third row, otherwise we would end up with just one equation

Keeping all three rows could also lead to a better conditioned set of eqs.

( )( ) ( )12112122

00

3

2

1

×=××=

=

−−

0pA

0ppp

XXXX

i

Tii

Tii

Tii

Tii

xzyz

16

DLT method for the estimation of P

The equation is linear in the unknown p The matrix elements of Ai are quadratic in the known

coordinates of the pts Each point correspondence gives rise to two independent

eqs in the entries of P, which has 11 d.o.f. Since we need 11 equations, we will use “5 ½

correspondences” (we will only use one of the eqs. for the last correspondence)

Overdetermined solutions are often advisable (noisy data)

Degenerate configurations of 3D points could prevent a complete calibration of the system (for example when the 3D points lie on a single plane)*

( )( ) ( )12112122 ×=××= 0pAi

* For more details see the book of Hartley and Zisserman (Ch. 21) and “A Flexible New Technique for Camera Calibration”. Z. Zhang. Technical Report MSRTR-98-71, Microsoft Research, December 1998. Available together with the software at http://research.microsoft.com/˜zhang/Calib/

17

Calibration through geometric error minimization

It is also possible to estimate the P elements by minimizing the geometric error between real image (xi) points and those computed by applying the projection matrix (P) to the 3D known points (Xi).

An iterative technique (i.e. Levenberg-Marquardt) is required for solving the problem. The DLT solution can be used as a starting point for the minimization.

( )

( )( ) ( ) ( )222

2

ˆ/ˆ/ˆ/ˆ/ˆ,

,,

,minˆ

iiiiiiiiii

iiii

iii

zyzyzxzxd

zyx

d

−+−=

=

= ∑

xx

x

PXxpP

18

Camera calibration Calibration:

An image pt x back-projects to a ray defined by x and the camera center

Calibration relates x to the direction of the ray In the Euclidean coords of the camera:

• Points on the ray

• Projection of points on the ray

( ) )~(~~1~ CdCCdX −+=−+= λλλ

[ ] Kdd

0IKx =

=

1|

λ

The camera calibration matrix K is the (affine) tf btw x and the ray’s direction d=K-1x measured in camera’s Euclidean coordinates

Note: generally d=K-1x is not a unit vector!

19

Instrinsic Camera calibration Angle between two rays with directions d1 and d2,

corresponding to the image pts x1 and x2, respectively, is given by

( ) ( )( ) ( ) ( ) ( )

( )( ) ( ) 2

112

11

21

1

21

21

11

11

21

11

2211

21cos

xKKxxKKxxKKx

xKxKxKxK

xKxKdddd

dd

−−−−

−−

−−−−

−−

=

==

TTTT

TT

TT

T

TT

T

θ

If the camera is calibrated (K known), then the angle btw rays can be measured from their corresponding image pts

20

The image of the absolute conic The calibration matrix K is related to the image of the

absolute conic ω The image of pts on π∞ can be easily derived

[ ][ ] KRd

dCIKRPXx

CIKRP

πd

X=

−==⇒

−=

∈

=

∞

∞∞

0~|

~|

0

The mapping btw π∞ and an image is given by the planar homography x=Hd, with H=KR

Note: this map does not depend on the position of the camera (C) but only on orientation and intrinsics!

21

The image of the absolute conic Since the absolute conic Ω∞ is on π∞, we can compute its

image through the homography H=KR ( ) ( ) ( ) ( )

( ) ( ) ( )( )( ) 1111

1111

11

1

−−−−−−

−−−−−−

−−−−∞

−−

===

===

====Ω=⇒

⇓

TTT

TTTT

TT

T

KKKKKRRK

KRRKKRKR

KRKRKRIKRωICCHHC

Hxx

The Image of the Absolute Conic (IAC) is the conic ω=(KKT)-1

22

Properties of the IAC The IAC depends only on the intrinsic camera

parameters The angle between two rays is

The Dual of the IAC (DIAC) is a line conic with matrix ( ) ( )( ) TT KKKKωωωω ==∝=

−−−−1111* det

23

Properties of the IAC

Once ω is identified in an image, then K is also identified A symmetric matrix ω*=KKT can be uniquely decomposed

into a product of an upper triangular matrix and its transposed (Cholesky factorization)

A plane π intersects π∞ in a line, which in turn intersects Ω∞ in two pts which are the circular pts of π

The imaged circular pts lie on ω at the pts of intersection btw the vanishing line of π and ω

These two properties can be used for calibration purposes

24

A simple calibration method The image of three squares (on non-

parallel planes) provides enough constraints to compute K The correspondence between the

4 corners of each square and their image allows us to compute the homography H between the plane of the square and the image plane

The same homography can be used to determine the image of the circular pts on π as H(1, ±i,0)T

Repeating that for each one of the three squares we obtain three distinct pairs of images of circular pts, which are enough to characterize the IAC ω (5 pts are enough to determine a conic)

K can be computed from ω∗=KKT through Cholesky factorization

Object coordinates on each object plane

0,0

0,A

A,0

A,A

25

A calibration method Compute the homography H between the plane of the square and the

image plane

Detemine the image of the circular pts as H(1, ±i,0)T Z

X

Y Z=0

H

21

0

1~ hhHm ii ±=

±=∞

26

A calibration method The image of circular points lies on the image of the

absolute conic:

Requiring that both real and imaginary parts be zero yields:

Two constraints for every image of the model plane

three images are required

( ) ( ) ( ) ( ) 0211

212121 =±±=±± −− hhKKhhhhωhh iiii TTT

21

211

121

211

1

11

221

1

0

0

hKKhhKKhhKKhhKKh

hKKhhKKh−−−−−−−−

−−−−

=⇒=−

=TTTTTTTT

TTTT

27

Lens Distortion (1)

Ideal projection

Observed position

dr dt

Xr

Yr

dr: radial distortion dt: tangential distortion

Pu Pd

28

Lens Distortion (2)

a

b

Xr

Axis of a maximum tangential distortion

Axis of a minimum tangential distortion

Radial Distortion Tangential Distortion

a) Barrel Distortion (typical in wide wide-angle photos) b) Pincushion distortion (typical in tele-photos)

Tangential distortions are typical due to misalignments among optical elements of the lens. Normally are negligeble.

29

Lens Distortion (3)

tele wideangle

30

Lens Distortion (4)

ZX

fxu =

ZY

fyu =

xdu dxx += ydu dyy +=

( )( )

++=

++=4

22

1

42

21

rkrkydrkrkxd

dy

dx 22dd yxr +=

Radial Component Description Camera Coordinate System

0

0

yyyxxx

u

u

−=−=

31

Lens Distortion (5) An image before and after radial distortion compensation

32

Calibration with planar target Using only a set of coplanar 3D reference points it is possible to calibrate a

camera system when: the position of the principal point (x0,y0) is known; the skew factor (s) is 0.

The considered target plane must be non parallel to the camera image plane.

Camera

Y

Z

Zw

c

Yw

c

c

wX

X

Zw

Yw

wX

Z c

Yc

cX

=R +T

+

=

z

y

x

w

w

w

c

c

c

TTT

ZYX

rrrrrrrrr

ZYX

987

654

321

33

The TSAI’s algorithm (1) Algorithm

Extrinsic Parameters estimated exploiting the Radial Alignment Constraint (R.A.C.)

Intrinsic Parameters estimated through nonlinear minimization

Key features + Efficient estimation + Fast and robust calculations - Based on a simplified camera model

(the P.P. is assumed as known, only the first term of the radial distortion is considered)

[ ] [ ][ ]

( ) ( )( ) ( )

( ) ( )( ) ( )xWWWdyWWWd

xWWWuyWWWu

dcdcccdd

ucucccuu

Tccc

Tddd

Tuuu

TZrYrXryTZrYrXrxTZrYrXryTZrYrXrx

yXxYYXyxyXxYYXyx

RAC

ZYXP

fyxpfyxp

+++=+++

+++=+++=−=×=−=×

⇒

=

==

321654

321654

0,,0,,

,,

,, ;,,

O

x

y

P

Q

p

q

R.A.C. : PQ // pq Image coord. system with origin in the P.P.

34

The TSAI’s algorithm (2)

( ) ( )

dyWdyWdyxdyWdyWd

ydWdWdxdWdWd

xWWdyWWd

xTrYxTrXxTTyTrYyTrXy

TxYrxXrxTyYryXry

TYrXryTYrXrx

=−−++

=−−−++⇓

++=++

−−−−− 15

14

112

11

5421

2154

0

If we assume Zw=0 for all the targets points and we consider distorted image coordinates, we have:

The projection equation assumes the form:

(1)

( ) ( )( ) ( ) uWWuzyyWW

uWWuzxxWW

zWW

yWW

y

u

zWW

xWW

x

u

yYrXryTTYrXrxYrXrxTTYrXr

TYrXrTYrXry

TYrXrTYrXrx

8754

8721

87

54

87

21 ;

+=−+++=−++

⇓

++

++=

++++

=

αα

αα

(2)

35

Tsai`s Two-Stage Camera Calibration Stage 1: Compute 3D orientation and x and y positions:

Compute (Xd, Yd) from calibration points in frame memory. Compute r1/Ty, r2/Ty, Tx/Ty, r4/Ty, and r5/Ty from the eq. (1) of the

previous slide (one eq. for each reference point). Compute r1 to r9, Tx, and Ty using orthogonality constraints on the R

matrix.

Stage 2: Compute αx, αy, k1, and Tz: Compute approximate f and Tz by ignoring lens distortion

(k1=0→xu=xd; yu=yd) by using eq. (2) of the previous slide (two eq.s for each reference point)..

Compute exact f, Tz, k1 by nonlinear optimization using the above initial values.

For details see: "A versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off-the-Shelf TV Cameras and Lenses", Roger Y. Tsai, IEEE Journal of Robotics and Automation, Vol. RA-3, No. 4, August 1987, pages 323-344.

36

Tsai’s Calibration improvement The quality of the calibration can be improved though the use of a full non-linear optimization based on the minimization of the global error between the known 2D positions of the target points and those estimated through the camera model

Different views of a 2D calibration pattern can be considered (in this case also the relative positions-orientations of the calibration plate must be estimated)

This way all the camera parameters (including the position of the principal point) may be considered.

For details see: "Flexible Camera Calibration By Viewing a Plane From Unknown orientations”. Z. Zhang. International Conference on Computer Vision (ICCV'99), Corfu, Greece, pages 666-673, September 1999

37

Two-view geometry

38

Point correspondence

39

Epipolar geometry The epipolar geometry btw two views is the geometry of

the intersection of the image planes with the pencil of planes having the baseline as axis

Any plane containing the baseline is an epipolar plane Epipolar lines: intersection of the epipolar plane with the image planes Epipoles: intersection of the baseline with the image planes

40

Epipolar geometry Converging cameras

Motion parallel to the image plane

41

Examples

42

Examples

43

The fundamental matrix Geometric derivation

A pt x in one image is transfered through the plane p to a pt x’ in the second image

π

x

e

x'

e' c c '

X

Hπ

xHx π='

44

The fundamental matrix Geometric derivation

The epipolar line through x’ is obtained joining x’ with the epipole e’.

π

x

e

x'

e' c

X

l'

c’

45

The fundamental matrix The fundamental matrix F may be written as

where Hπ is the transfer mapping from one image to the other through any plane π

Since has rank 2 and Hπ has rank 3, F is a matrix of rank 2 It is a mapping from the 2-D projective plane of the first

image to the pencil of epipolar lines through the epipole e’, therefore it must be rank-2

In order to compute F we need at least 7 correspondences between pts in the two images F has 7 d.o.f.:

• A 3x3 homogeneous matrix has 8 d.o.f • The constraint det F=0 reduces it to 7

46

Correspondence condition For any pair of corresponding pts in the two images we

have

In fact, if the two pts correspond, then x’ lies on the

epipolar line l’=Fx corresponding to the pt x If image pts satisfy the correspondence condition, then the

rays defined by these pts are coplanar

47

Properties of F Epipolar lines:

Epipole:

F has 7 dof: 3x3 homogeneous matrix:

8 dof det F=0: -1 dof

''' xFlxFxlx T=⇒=

( ) ( ) 0 , ' contains ' , ==≠∀⇒=≠∀ xFeFxeexeFxlex TT ''

0' , 0 == FeFe T

( ) ( ) ( ) 0' , '' contains ' , '' ===≠∀⇒=≠∀ xFexFexFeexexFlex TTTTTT

48

Computation of the fundamental matrix Basic equations Need at least 7 pts to determine F from this equation

Given a pair of pts (x,y,1)T and (x’,y’,1)T the eq. Becomes

From a set of n pt matches we have a set of linear eqs.

' , 0' xxFxx ↔∀=T

0Af =

=

33

11

''''''

11'11

'11

'1

'11

'11

'1

1

1

f

f

yxyyyxyxxyxy

yxyyyxyxyxxx

nnnnnnnnnnnn

( ) 01,,,',',',',','

0'''''' 333231232221131211

=⇓

=++++++++

fyxyyyxyxyxxx

fyfxffyyfyxfyfxyfxxfx

49

From the camera projection matrices to F (1)

( ) ( )( ) IPPPPPPPPPPP

xXPxPX

==⇒==

==

−+−+ 11 of inverse pseudo

''TTTT

( ) ( ) ( ) [ ] ( )[ ] ( )+

+++

=

==∧=∧=

PPeFFxxPPexPPexPPCPl

'''''''''

x

x

P+x lies on r, in fact it is projected by the first camera onto x P P+x=x

c

π

x e

x' e'

X l'

c’

r r’

c

50

From the camera projection matrices to F (2)

[ ] [ ]

[ ] [ ] 1

1

''''

;1

;

;|'' ;|

−+

−+

==

=

=

==

RKKtKPPCPF

0C

0K

P

tRKP0IKP

xx

T

For example we have:

π

x e

x' e'

X l'

c’

r r’

c

51

More on camera matrices and F If H is a 4x4 matrix representing a projective

transformation of P3, then the fundamental matrices corresponding to the pairs of camera matrices (P;P’) and (PH;P’H) are the same

A pair of projection matrices uniquely determine the fundamental matrix

[ ] [ ]

[ ] [ ] [ ]

1|'''

|' , |

MmFMxmMx0

mM0x

PCPl

mMP0IP

×× =⇒=∧

=

∧=

⇓==

Converse not true!

52

Canonical cameras given F (1) The fundamental matrix determines a pair of camera

matrices at best up to a projective tf Derivation of a specific formula for a pair of cameras

with canonical form given F based on two results:

0' have we, and setting 0' symmetric-skew '===

∀=⇔

FxxXP'x'PXxXFPXPXFPP

T

TTT

Result 1: A non-zero matrix F is the fundamental matrix corresponding to a pair of camera matrices P and P' iff P'TFP is skew-symmetric

53

Canonical cameras given F (2) Result 2: Let F be a fundamental matrix and S any

skew-symmetric matrix. Define

then F is the fundamental matrix of the pair (P,P’)

Proof is based on Result 1:

which is skew-symmetric, as

[ ] [ ] 3 rank with (epipole), , , '''|'| P0FeeSFP0IP TT ===

[ ] [ ]

=

==

00'|'|' T

TT

T

TTTT

00FSF

Fe0FSF

0IFeSFFPP

( ) ( ) 0FSSFSFFFSFFSFFSF =+=+=+ TTTTTTTTTT

54

Canonical cameras given F (3) Result 3: The camera matrices corresponding to a

fundamental matrix F can be chosen as This result follows from choosing and leads to the following

[ ]×= 'eS

Result 4: The most general formula for a pair of canonic camera matrices corresponding to a fundamental matrix F is

where v is any 3-vector and λ a non-zero scalar

55

F estimation with 7 correspondences We must solve the system Af=0 where, in this case A is

a 7×9 matrix It has generally rank 7, therefore the solution of the

system is a 2-dimensional space with form αF1+(1-α)F2. F1 and F2 are obtained as the matrices corresponding to the generators f1 and f2 of the right-null space of A (for example by using Z=null(A) in Matlab)

The condition det(F)=0 becomes det(αF1+(1-α)F2)=0. It is a cubic equation in α. One or three real solutions are possible. Therefore one or three different F matrices are the results of this estimation process

56

Computation of the fundamental matrix 8-pt method

The equation Af=0 is a homogeneous one, therefore f is determined only up to a scale factor

Enforcement of the singularity constraint F must be rank-2 and this is not guaranteed by the above procedure Replace F with the rank-2 matrix F’ that minimizes the Froboenius norm

||F’-F||frob

( )

( ) T

TT

srdiag

tsrtsrdiag

VUF

VUUDVF

0,,'

,,,

=

↓

≥≥==

[ ]AA

AAA

of conjugatecomplex theof transpose*

,

*2

,

T

ji

Tjifrob

tracea∑ ==

57

The normalized 8-point algorithm (1) 8-point algorithm performs badly in presence of noise Normalization of data

8-pt algorithm is sensitive to origin of coordinates and scale

Data must be translated and scaled to “canonical” coordinate frame

Normalizing transformation is applied to both images Translated so that centroid is in the origin Scaled so that RMS distance of points from origin is √2

58

The normalized 8-point algorithm (2) Normalization: Transform the image coordinates: Solution: Compute F from the matches

Singularity constraint: Find singular that lies the closest to

Denormalization:

ii

ii

xTxTxx

′=′=

'ˆˆ

ii xx ˆˆ ′↔

0ˆˆˆ =′iTi xFx

TFTFTxFTxxFx 'ˆ'0'ˆ''0ˆ'ˆ'ˆ Ti

TTii

Ti =⇒=⇒=

F′ˆ F

59

Automatic computation of F (1) 1. Interest pts: compute interest pts in each image 2. Putative correspondences: compute a set of pt matches based

on proximity and luminance profile similarity 3. RANSAC robust estimation: repeat for N samples

a. Select a random sample of 7 correspondences and compute F using the 7-pt method

b. Compute the distance d⊥ of each putative match (given x, the distance between x’ and the epipolar line l’=Fx).

c. Compute the No. of inliers consistent with F as the no. of correspondences for which d⊥<t pixels

d. If there are 3 real solutions for F, the No. of inliers is computed for each solution and the solution with most inliers retained

⇒ Choose F with the largest no. of inliers

4. Non-linear estimation (Levemberg-Marquardt) 5. Guided matching search for further interest pts using

F The last two steps are iterated until until the number of

correspondencs is stable

60

Automatic computation of F (2) Why use the 7-points algorithm instead of the 8-points algorithm? Given w the probability that the match between a couple of points is correct, the probability p that, in N extractions of groups of s couples of points, there is at least one set of couples all correctly matched is: p=1-(1-ws)N

For p=0.99, w=0.5, s=7 N>586 For p=0.99, w=0.5, s=8 N>1176 Therefore the use in the proposed procedure of the 7-point alg. for F estimation allows us to significantly reduce the number of trials

61

Automatic computation of F (3) What is d⊥? Given x and x’, d⊥ is the minimum Euclidian distance between x’ and the epipolar line calculated from the available F (l’=Fx).

x’ l’ d⊥

[ ][ ]

22

;,,'

;,,'

ba

cwyb

wxa

d

cba

wyxT

T

+

++=

=

=

⊥

l

x

62

Automatic computation of F (4) What is the step #4 of the proposed procedure? Given a first estimation of the fundamental matrix Chose camera matrices as From the correspondences xi↔xi’ and determine in order to minimize Iteratively minimize Over There are 12+3N variables.

F

[ ] [ ][ ]0'

;'|' ' ;|=

==

FeeFeP0IP

Tx

F iX

( ) ( )22 'ˆ,'ˆ, iiii dd xxxx +

( ) ( )∑ +i

iiii dd 22 'ˆ,'ˆ, xxxx

N1i ˆ,ˆ=iXF

12 parameters is an over-parametrization of F: . Also minimal parameterization is possible. For example: Singularity when epipole of 1° imm is. ∞

+++

=βαβαβα

eedcdcbaba

F1

63

Motion vectors of the corners

Inliers Outliers

Example

64

RANSAC results

65

The essential matrix Specialization of F when using normalized

coords:

Adopting normalized coords, the fundamental matrix becomes

Properties: 5 dof: 3 (R) + 3 (t) – 1 (scale) 2 extra constraints w.r.t. F

[ ] [ ]

====

→

== −−

tRPXPPXKxKx

tRKPPXx K

|ˆˆˆ

|

11

1-

[ ] [ ]×× == tRRRtE T

66

Computation of the essential matrix The essential matrix can be computed from point correspondences

(expressed in normalized coordinates):

Or directly from: 0ˆ'ˆ =xEx T

FKKE T'=

A 3x3 matrix is an essential matrix iff two of its singular values are equal and the third one is zero

Note: From E it is possible to retrieve and up to a scale factor and a four-fold ambiguity. From F, P and P’ are retrieved up to a projective transformation.

P 'P

67

Retrieving the camera matrices from E Assume that E=Udiag(1,1,0)VT (SVD)

A reconstructed point X will be in front of both cameras in only one of these four solutions

For a given essential matrix E=Udiag(1,1,0)VT, and first camera matrix P=[I|0], there are four possible choices for the second camera matrix P’:

where

[ ][ ][ ][ ]3

3

3

3

'

'

'

'

uVUWPuVUWPuUWVP

uUWVP

−=

=

−=

=

TT

TT

T

T

−=

100001010

W

68

Triangulation Triangulation:

Knowing x and x’ Knowing P and P ’

Compute X such that X

x

e

x'

e' c c '

;PXx = ;'' XPx =

69

Triangulation in presence of noise In the presence of noise, back-projected lines do not

meet

x

c c '

'x

X

x

'x

70

Linear triangulation methods We need the equations x=PX and x'=P'X to be satisfied as

best as possible For P is true that:

Pp

0XpXp

XpXpXpXp

0Xppp

PXxPXx

of rows theare where

1

31

31

32

3

2

1

iT

TT

TT

TT

T

T

T

xyxy

yx

=

−−−

⇓

=

∧

=∧⇒∝

Only two rows are linear independent

71

Linear triangulation methods Write such conditions in the form of linear equations in X

Solve for X (right null space of A) It is possible to generalize for a multiple-view case Notice that we are not minimizing any geometrically-

meaningful quantity – not optimal

72

Minimizing geometric error Find X that minimizes difference between projected and measured

points

xx e

x'

e' c c '

X

'xd d'

Minimization of the Cost Function: 22 )'ˆ,'(')ˆ,()( xxxxX ddC +=Can be done by an iterative minimization technique (for example Levemberg-Marquardt). A close form solution is also possible (see the Hartley-Zisserman book), it requires to find the real roots of a 6-degree polynomial.

73

Point at minimum distance from 2 lines

xx

c c'

'x

X

'x

d

d’

[ ] [ ]

−

−+

−

−

=

−

−+

−

−

=

−

−+

−

−=

=

−+

−=∧+∧=

'ˆ''ˆˆˆ

'ˆ'''ˆˆˆ

'ˆ''ˆ'ˆˆ

'ˆ'ˆˆ''ˆD2222

2

cXcXcXcX

cXcXcXcX

cXcXcXcX

cXacXaXcaXca

TT

TT

TT

TT

xx

WW

AAAA

AAAA

22

2 ˆ'''''ˆD

;ˆ''''' ;ˆ

XcxcxcXc

cxcx

XcxcxcdXc

cxcxd

∧+∧=

∧=∧=

( ) ( ) viewsmoreor three toextendedeasely becan n formulatio The

'''ˆ

0'ˆ'ˆ2

point 3D searched theis 0Dfor which X The

1

2

2

ccX

cXcXD

opt

WWWW

WW

++=

=

−+

−=∇

=∇

−