CSCE 641 Computer Graphics: Image-based Modeling

Jinxiang Chai

Image-based modeling

Estimating 3D structure

Estimating motion, e.g., camera motion

Estimating lighting

Estimating surface model

Traditional modeling and rendering

User input Texture map survey data

Geometry Reflectance Light source

Camera model

Images modeling rendering

For photorealism:

- Modeling is hard

- Rendering is slow

Can we model and render this?

What do we want to do for this model?

Image based modeling and rendering

Images user input range

scansModel Images

Image-based modeling

Image-based rendering

Spectrum of IBMR

Images user input range

scans

Model

Images

Image based modeling

Image-based renderingGeometry+ Images

Geometry+ Materials

Images + Depth

Light field

Panoroma

Kinematics

Dynamics

Etc.

Camera + geometry

Spectrum of IBMR

Images user input range

scans

Model

Images

Image based modeling

Image-based renderingGeometry+ Images

Geometry+ Materials

Images + Depth

Light field

Panoroma

Kinematics

Dynamics

Etc.

Camera + geometry

Spectrum of IBMR

Images user input range

scans

Model

Images

Image based modeling

Image-based renderingGeometry+ Images

Geometry+ Materials

Images + Depth

Light field

Panoroma

Kinematics

Dynamics

Etc.

Camera + geometry

Stereo reconstruction

Given two or more images of the same scene or object, compute a representation of its shape

How can we estimate camera parameters?

knownknowncameracamera

viewpointsviewpoints

How can we estimate the camera parameters?

Camera calibration

Augmented pin-hole camera - focal point, orientation

- focal length, aspect ratio, center, lens distortion

Known 3DKnown 3D

Classical calibration - 3D 2D

- correspondence

Camera calibration online resources

Camera and calibration target

Classical camera calibration

Known 3D coordinates and 2D coordinates - known 3D points on calibration targets

- find corresponding 2D points in image using feature detection

algorithm

Camera parameters

u0

v0

100-sy0

sx аuv1

Perspective proj. View trans.Viewport proj.

Known 3D coords and 2D coordsKnown 3D coords and 2D coords

Camera parameters

u0

v0

100-sy0

sx аuv1

Perspective proj. View trans.Viewport proj.

Known 3D coords and 2D coordsKnown 3D coords and 2D coords

Intrinsic camera parameters (5 parameters)

extrinsic camera parameters (6 parameters)

Camera matrix

Fold intrinsic calibration matrix K and extrinsic pose parameters (R,t) together into acamera matrix

M = K [R | t ]

(put 1 in lower r.h. corner for 11 d.o.f.)

Camera matrix calibration

Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi)

Camera matrix calibration

Linear regression:• Bring denominator over, solve set of (over-determined) linear

equations. How?

Camera matrix calibration

Linear regression:• Bring denominator over, solve set of (over-determined) linear

equations. How?

• Least squares (pseudo-inverse) - 11 unknowns (up to scale) - 2 equations per point (homogeneous coordinates) - 6 points are sufficient

Nonlinear camera calibration

Perspective projection:

1100

0

1 3

2

1

3

2

1

0

0

i

i

i

T

T

T

y

x

i

i

z

y

x

t

t

t

r

r

r

vf

uf

v

u

Nonlinear camera calibration

Perspective projection:

1100

0

1 3

2

1

3

2

1

0

0

i

i

i

T

T

T

y

x

i

i

z

y

x

t

t

t

r

r

r

vf

uf

v

u

K R T P

Nonlinear camera calibration

Perspective projection:

2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters:

1100

0

1 3

2

1

3

2

1

0

0

i

i

i

T

T

T

y

x

i

i

z

y

x

t

t

t

r

r

r

vf

uf

v

u

K R T P

33

32302

33

30213021

)(

)(

tPr

ttfPrvrfv

tPr

tuttfPrurrfu

T

yTT

yi

Tx

TTTx

i

Nonlinear camera calibration

Perspective projection:

2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters:

1100

0

1 3

2

1

3

2

1

0

0

i

i

i

T

T

T

y

x

i

i

z

y

x

t

t

t

r

r

r

vf

uf

v

u

33

32302

33

30213021

)(

)(

tPr

ttfPrvrfv

tPr

tuttfPrurrfu

T

yTT

yi

Tx

TTTx

i

K

);,,( iPTRKf

);,,( iPTRKg

R T P

Multiple calibration images

Find camera parameters which satisfy the constraints from M images, N points: for j=1,…,M

for i=1,…,N

This can be formulated as a nonlinear optimization problem:

);,,(

);,,(

ijjji

ijjji

PTRKgv

PTRKfu

M

j

N

iijj

jiijj

ji PTRKgvPTRKfu

1 1

22 ));,,(());,,((

Multiple calibration images

Find camera parameters which satisfy the constraints from M images, N points: for j=1,…,M for i=1,…,N

This can be formulated as a nonlinear optimization problem:

);,,(

);,,(

ijjji

ijjji

PTRKgv

PTRKfu

M

j

N

iijj

jiijj

ji PTRKgvPTRKfu

1 1

22 ));,,(());,,((

Solve the optimization using nonlinear optimization techniques:

- Gauss-newton

- Levenberg-Marquardt

Nonlinear approach

Advantages:• can solve for more than one camera pose at a time

• fewer degrees of freedom than linear approach

• Standard technique in photogrammetry, computer vision, computer graphics

- [Tsai 87] also estimates lens distortions (freeware @ CMU)http://www.cs.cmu.edu/afs/cs/project/cil/ftp/html/v-source.html

Disadvantages:• more complex update rules

• need a good initialization (recover K [R | t] from M)

Camera Calibration

Public calibration toolbox- http://research.microsoft.com/en-us/um/people/zhang/Calib/

- http://www.vision.caltech.edu/bouguetj/calib_doc/

How can we estimate the camera parameters?

Application: camera calibration for sports video

[Farin et. Al]

images Court model

Stereo matching

Given two or more images of the same scene or object as well as their camera parameters, how to compute a representation of its shape?

What are some possible representations for shapes?• depth maps

• volumetric models

• 3D surface models

• planar (or offset) layers

Outline

Stereo matching - Traditional stereo

- Active stereo

Volumetric stereo - Visual hull

- Voxel coloring

- Space carving

Stereo matching• 11.1, 11.2,.11.3,11.5 in Sezliski book

• D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms.International Journal of Computer Vision, 47(1/2/3):7-42, April-June 2002.

Readings

Stereo

scene pointscene point

optical centeroptical center

image planeimage plane

Stereo

Basic Principle: Triangulation• Gives reconstruction as intersection of two rays• Requires

> calibration

> point correspondence

Stereo correspondence

Determine Pixel Correspondence• Pairs of points that correspond to same scene point

Epipolar Constraint• Reduces correspondence problem to 1D search along conjugate

epipolar lines• Java demo: http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html

epipolar lineepipolar lineepipolar lineepipolar lineepipolar plane

Stereo image rectification

Stereo image rectification

• reproject image planes onto a commonplane parallel to the line between optical centers

• pixel motion is horizontal after this transformation• two homographies (3x3 transform), one for each

input image reprojection C. Loop and Z. Zhang. Computing Rectifying Homographies

for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.

Rectification

Original image pairs

Rectified image pairs

Stereo matching algorithms

Match Pixels in Conjugate Epipolar Lines• Assume brightness constancy

• This is a tough problem

• Numerous approaches> A good survey and evaluation: http://www.middlebury.edu/stereo/

Your basic stereo algorithm

For each epipolar line

For each pixel in the left image• compare with every pixel on same epipolar line in right image

• pick pixel with minimum matching cost

Improvement: match windows• This should look familiar.. (normalized cross correlation or SSD)• Can use Lukas-Kanade or discrete search (latter more common)

Window size

• Smaller window+

-

• Larger window+

-

W = 3 W = 20

Effect of window size

More constraints?

We can enforce more constraints to reduce matching ambiguity - smoothness constraints: computed disparity at a pixel

should be consistent with neighbors in a surrounding window.

- uniqueness constraints: the matching needs to be bijective

- ordering constraints: e.g., computed disparity at a pixel

should not be larger than the disparity of its right neighbor pixel by

more than one pixel.

Stereo results

Ground truthScene

• Data from University of Tsukuba

• Similar results on other images without ground truth

Results with window search

Window-based matching(best window size)

Ground truth

Better methods exist...

A better methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,

International Conference on Computer Vision, September 1999.

Ground truth

More recent development

High-Quality Single-Shot Capture of Facial Geometry [siggraph 2010, project website] - capture high-fidelity facial geometry from multiple cameras

- pairwise stereo reconstruction between neighboring cameras

- hallucinate facial details

More recent development

High Resolution Passive Facial Performance Capture [siggraph 2010, project website] - capture dynamic facial geometry from multiple video cameras

- spatial stereo reconstruction for every frame

- building temporal correspondences across the entire sequence

Stereo reconstruction pipeline

Steps• Calibrate cameras

• Rectify images

• Compute disparity

• Estimate depth

• Camera calibration errors

• Poor image resolution

• Occlusions

• Violations of brightness constancy (specular reflections)

• Large motions

• Low-contrast image regions

Stereo reconstruction pipeline

Steps• Calibrate cameras

• Rectify images

• Compute disparity

• Estimate depth

What will cause errors?

Outline

Stereo matching - Traditional stereo

- Active stereo

Volumetric stereo - Visual hull

- Voxel coloring

- Space carving

Active stereo with structured light

Project “structured” light patterns onto the object• simplifies the correspondence problem

camera 2

camera 1

projector

camera 1

projector

Li Zhang’s one-shot stereo

Active stereo with structured light

Laser scanning

Optical triangulation• Project a single stripe of laser light• Scan it across the surface of the object• This is a very precise version of structured light scanning

Digital Michelangelo Projecthttp://graphics.stanford.edu/projects/mich/

Laser scanned models

The Digital Michelangelo Project, Levoy et al.

Laser scanned models

The Digital Michelangelo Project, Levoy et al.

RGBD Sensors

2008 2010 2013

Lower size/cost with better accuracy

Kinect Sensor

Kinect Sensor