Self-calibrationClass 13

Read Chapter 6

Assignment 3

• Collect potential matches from all algorithms for all pairs• Matlab ASCII format, exchange data

• Implement RANSAC that uses combined match dataset

• Compute consistent set of matches and epipolar geometry• Report thresholds used, match sets used, number of

consistent matches obtained, epipolar geometry, show matches and epipolar geometry (plot some epipolar lines).

Due next Tuesday, Nov. 2

naming convention: firstname_ij.dat

chris_56.dat

[F,inliers]=FRANSAC([chris_56; brian_56; …])

http://www.unc.edu/courses/2004fall/comp/290/089/assignment3/

Papers

• Each should present a paper during 20-25 minutes followed by discussion. Partially outside of class schedule to make up for missed classes.(When?)

• List of proposed papers will come on-line by Thursday, feel free to propose your own (suggestion: something related to your project).

• Make choice by Thursday, assignments will be made in class.

• Everybody should have read papers that are being discussed.

PapersChris

Nathan

Brian

Li

Chad

Seon Joo

Jason

Sudipta

Sriram

Christine

http://www.unc.edu/courses/2004fall/comp/290b/089/papers/

3D photography course schedule

Introduction

Aug 24, 26 (no course) (no course)

Aug.31,Sep.2

(no course) (no course)

Sep. 7, 9 (no course) (no course)

Sep. 14, 16 Projective Geometry Camera Model and Calibration

(assignment 1)

Feb. 21, 23 Camera Calib. and SVM Feature matching(assignment 2)

Feb. 28, 30 Feature tracking Epipolar geometry(assignment 3)

Oct. 5, 7 Computing F Triangulation and MVG

Oct. 12, 14 (university day) (fall break)

Oct. 19, 21 Stereo Active ranging

Oct. 26, 28 Structure from motion SfM and Self-calibration

Nov. 2, 4 Shape-from-silhouettes Space carving

Nov. 9, 11 3D modeling Appearance Modeling Nov.12 papers(2-3pm SN115)

Nov. 16, 18 (VMV’04) (VMV’04)

Nov. 23, 25 papers & discussion (Thanksgiving)

Nov.30,Dec.2

papers & discussion papers and discussion Dec.3 papers(2-3pm SN115)

Dec. 7? Project presentations

Ideas for a project?

Chris Wide-area display reconstruction

Nathan ?

Brian ?

Li Visual-hulls with occlusions 

Chad Laser scanner for 3D environments 

Seon Joo Collaborative 3D tracking

Jason SfM for long sequences

SudiptaCombining exact silhouettes and photoconsistency

Sriram Panoramic cameras self-calibration

Christine desktop lamp scanner

Dealing with dominant planar scenes

• USaM fails when common features are all in a plane

• Solution: part 1 Model selection to detect problem

(Pollefeys et al., ECCV‘02)

Dealing with dominant planar scenes

• USaM fails when common features are all in a plane• Solution: part 2 Delay ambiguous computations

until after self-calibration(couple self-calibration over all 3D

parts)

(Pollefeys et al., ECCV‘02)

Non-sequential image collections

4.8im/pt64 images

3792

po

ints

Problem:Features are lost and reinitialized as new features

Solution:Match with other close views

For every view iExtract featuresCompute two view geometry i-1/i and matches Compute pose using robust algorithmRefine existing structureInitialize new structure

Relating to more views

Problem: find close views in projective frame

For every view iExtract featuresCompute two view geometry i-1/i and matches Compute pose using robust algorithmFor all close views k

Compute two view geometry k/i and matchesInfer new 2D-3D matches and add to list

Refine pose using all 2D-3D matchesRefine existing structureInitialize new structure

Determining close views

• If viewpoints are close then most image changes can be modelled through a planar homography

• Qualitative distance measure is obtained by looking at the residual error on the best possible planar homography

Distance = m´,mmedian min HD

9.8im/pt

4.8im/pt

64 images

64 images

3792

po

ints

2170

po

ints

Non-sequential image collections (2)

Hierarchical structure and motion recovery

• Compute 2-view• Compute 3-view• Stitch 3-view reconstructions• Merge and refine reconstruction

FT

H

PM

Stitching 3-view reconstructions

Different possibilities1. Align (P2,P3) with (P’1,P’2) -1

23-1

12H

HP',PHP',Pminarg AA dd

2. Align X,X’ (and C,C’) j

jjAd HX',XminargH

3. Minimize reproj. error

jjj

jjj

d

d

x',HXP'

x,X'PHminarg 1-

H

4. MLE (merge) j

jjd x,PXminargXP,

Refining structure and motion

• Minimize reprojection error

• Maximum Likelyhood Estimation (if error zero-mean

Gaussian noise)• Huge problem but can be solved

efficiently (Bundle adjustment)

m

k

n

iikD

ik 1 1

2

kiM̂,P̂

M̂P̂,mmin

Sparse bundle adjustment

U1

U2

U3

WT

W

V

P1 P2 P3 M

Non-linear min. requires to solve Jacobian of has sparse block

structure

J JJN T

12xm 3xn(in general

much larger)

im.pts. view 1

m

k

n

iikD

1 1

2

ki M̂P̂,m

Needed for non-linear minimization

0

T-1T JJJ e

Sparse bundle adjustment• Eliminate dependence of

camera/motion parameters on structure parametersNote in general 3n >> 11m

WT V

U-WV-1WT

NI0WVI 1

11xm 3xn

Allows much more efficient computations

e.g. 100 views,10000 points,

solve 1000x1000, not 30000x30000Often still band diagonaluse sparse linear algebra algorithms

Self-calibration

• Introduction• Self-calibration• Dual Absolute Quadric• Critical Motion Sequences

Motivation

• Avoid explicit calibration procedure• Complex procedure• Need for calibration object • Need to maintain calibration

Motivation

• Allow flexible acquisition• No prior calibration necessary• Possibility to vary intrinsics• Use archive footage

Projective ambiguity

Reconstruction from uncalibrated images

projective ambiguity on reconstruction

´M´M))((Mm 1 PTPTP

Stratification of geometry

15 DOF 12 DOFplane at infinity

parallelism

More general

More structure

Projective Affine Metric

7 DOFabsolute conicangles, rel.dist.

Constraints ?

Scene constraints• Parallellism, vanishing points, horizon, ...• Distances, positions, angles, ...Unknown scene no constraints

Camera extrinsics constraints–Pose, orientation, ...

Unknown camera motion no constraints Camera intrinsics constraints

–Focal length, principal point, aspect ratio & skew

Perspective camera model too general some constraints

Euclidean projection matrix

tRRKP TT

1yy

xx

uf

usf

K

Factorization of Euclidean projection matrix

Intrinsics:

Extrinsics: t,R

Note: every projection matrix can be factorized,

but only meaningful for euclidean projection matrices

(camera geometry)

(camera motion)

Constraints on intrinsic parameters

Constant e.g. fixed camera:

Knowne.g. rectangular pixels:

square pixels: principal point known:

21 KK

0s

1yy

xx

uf

usf

K

0, sff yx

2,

2,

hwuu yx

Self-calibration

Upgrade from projective structure to metric structure using constraints on intrinsic camera parameters• Constant intrinsics

• Some known intrinsics, others varying

• Constraints on intrincs and restricted motion(e.g. pure translation, pure rotation, planar motion)

(Faugeras et al. ECCV´92, Hartley´93,

Triggs´97, Pollefeys et al. PAMI´99, ...)

(Heyden&Astrom CVPR´97, Pollefeys et al. ICCV´98,...)

(Moons et al.´94, Hartley ´94, Armstrong ECCV´96, ...)

A counting argument

• To go from projective (15DOF) to metric (7DOF) at least 8 constraints are needed

• Minimal sequence length should satisfy

• Independent of algorithm• Assumes general motion (i.e. not critical)

8#1# fixedmknownm

Outline

• Introduction• Self-calibration• Dual Absolute Quadric• Critical Motion Sequences

The Dual Absolute Quadric

00

0I*T

The absolute dual quadric Ω*∞ is a fixed conic under

the projective transformation H iff H is a similarity

1. 8 dof2. plane at infinity π∞ is the nullvector of Ω∞

3. Angles:

2*

21*

1

2*

1

ππππ

ππcos

TT

T

Absolute Dual Quadric and Self-calibration

Eliminate extrinsics from equation

Equivalent to projection of Dual Abs.Quadric

))(Ω)((Ω *1* TTTTT PTTTPTPPKK

Dual Abs.Quadric also exists in projective world

T´Ω´´ * PP Transforming world so thatreduces ambiguity to similarity

** ΩΩ´

*

*

projection

constraints

Absolute conic = calibration object which is always present but can only be observed through constraints on the intrinsics

Tii

Tiii Ωω KKPP

Absolute Dual Quadric and Self-calibration

Projection equation:Projection equation:

Translate constraints on K through projection equation to constraints on *

Constraints on *

1

ω 22

222

*

yx

yyyyxy

xyxyxx

cc

ccfccsf

cccsfcsf

Zero skew quadratic m

Principal point linear 2m

Zero skew (& p.p.)

linear m

Fixed aspect ratio (& p.p.& Skew)

quadratic m-1

Known aspect ratio (& p.p.& Skew)

linear m

Focal length (& p.p. & Skew)

linear m

*23

*13

*33

*12 ωωωω

0ωω *23

*13

0ω*12

*11

*22

*22

*11 ω'ωω'ω

*22

*11 ωω

*11

*33 ωω

condition constraint type #constraints

Linear algorithm

Assume everything known, except focal length

0Ω

0Ω

0Ω

0ΩΩ

23T

13T

12T

22T

11T

PP

PP

PP

PPPP

(Pollefeys et al.,ICCV´98/IJCV´99)

TPP *2

2

*

100

0ˆ0

00ˆ

ω

f

f

Yields 4 constraint per imageNote that rank-3 constraint is not enforced

Linear algorithm revisited

0Ω

0Ω

0Ω

0ΩΩ

23T

13T

12T

22T

11T

PP

PP

PP

PPPP

100

0ˆ0

00ˆ2

2

f

fTKK

9

1

9

1

)3log()1log()ˆlog( f)1.1log()1log()log( ˆ

ˆ

y

x

f

f1.00xc1.00yc

0s

0ΩΩ

0ΩΩ

33T

22T

33T

11T

PPPP

PPPP

(Pollefeys et al., ECCV‘02)

1.0

11.0

101.0

12.0

1

assumptions

Weighted linear equations

Projective to metric

Compute T from

using eigenvalue decomposition of and then obtain metric

reconstruction as

00

0

~ withΩ

~or Ω

~ **T

T-1-T IITITTTI

M and TPT-1

Ω*

Alternatives: (Dual) image of absolute conic

• Equivalent to Absolute Dual Quadric

• Practical when H can be computed first• Pure rotation (Hartley’94, Agapito et al.’98,’99)

• Vanishing points, pure translations, modulus constraint, …

T** ωω HH ea)( HH

TPP ** Ωω

1ω 22

22

*

yx

yyyyx

xyxxx

ccccfcc

ccccf

22222222

22

22

220

01

ω

yxxyyxyxxy

yxx

xyy

yx cfcfffcfcf

cff

cff

ff

Note that in the absence of skew the IAC can be more practical than the DIAC!

Kruppa equations

Limit equations to epipolar geometryOnly 2 independent equations per pairBut independent of plane at infinity

T*TT*T* ωe'ωe'e'ωe' FFHH

Refinement

• Metric bundle adjustment

Enforce constraints or priors on intrinsics during minimization(this is „self-calibration“ for photogrammetrist)

Outline

• Introduction• Self-calibration• Dual Absolute Quadric• Critical Motion Sequences

Critical motion sequences

• Self-calibration depends on camera motion

• Motion sequence is not always general enough

• Critical Motion Sequences have more than one potential absolute conic satisfying all constraints

• Possible to derive classification of CMS

(Sturm, CVPR´97, Kahl, ICCV´99, Pollefeys,PhD´99)

Critical motion sequences:constant intrinsic parameters

Most important cases for constant intrinsics

Critical motion type

ambiguity

pure translation affine transformation (5DOF)pure rotation arbitrary position for (3DOF)orbital motion proj.distortion along rot. axis

(2DOF)planar motion scaling axis plane (1DOF)

Note relation between critical motion sequences and restricted motion algorithms

Critical motion sequences:varying focal length

Most important cases for varying focal length (other parameters known)Critical motion type

ambiguity

pure rotation arbitrary position for (3DOF)forward motion proj.distortion along opt. axis

(2DOF)translation and rot. about opt. axis

scaling optical axis (1DOF)

hyperbolic and/or elliptic motion

one extra solution

Critical motion sequences:algorithm dependent

Additional critical motion sequences can exist for some specific algorithms• when not all constraints are enforced

(e.g. not imposing rank 3 constraint)• Kruppa equations/linear algorithm: fixating

a pointSome spheres also project to circles located in the image and hence satisfy all the linear/kruppa self-calibration constraints

Non-ambiguous new views for CMS

• restrict motion of virtual camera to CMS• use (wrong) computed camera parameters

(Pollefeys,ICCV´01)

Next class: shape from silhouettes