103
3D Photography: Stereo Vision Kalin Kolev, Marc Pollefeys Spring 2013 http://cvg.ethz.ch/teaching/2013spring/3dphoto/

3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

  • Upload
    others

  • View
    2

  • Download
    0

Embed Size (px)

Citation preview

Page 1: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

3D Photography: Stereo Vision

Kalin Kolev, Marc Pollefeys

Spring 2013 http://cvg.ethz.ch/teaching/2013spring/3dphoto/

Page 2: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Feb 18 Introduction

Feb 25 Lecture: Geometry, Camera Model, Calibration

Mar 4 Lecture: Features, Tracking/Matching

Mar 11 Project Proposals by Students

Mar 18 Lecture: Epipolar Geometry

Mar 25 Lecture: Stereo Vision

Apr 1 Easter

Apr 8 Short lecture “SfM / SLAM” + 2 papers

Apr 15 Project Updates

Apr 22 Short lecture “Active Ranging, Structured Light” + 2 papers

Apr 29 Short lecture “Volumetric Modeling” + 2 papers

May 6 Short lecture “Mesh-based Modeling” + 2 papers

May 13 Short lecture “Shape-from-X” + 2 papers

May 20 Pentecost / White Monday

May 27 Final Demos

Schedule (tentative)

Page 3: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Stereo & Multi-View Stereo

http://cat.middlebury.edu/stereo/

Tsukuba dataset

Page 4: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Stereo

• Standard stereo geometry • Stereo matching

• Correlation • Optimization (DP, GC)

• General camera configuration • Rectifications • Plane-sweep

• Multi-view stereo

Page 5: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Stereo

Page 6: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Occlusions

(Slide from Pascal Fua)

Page 7: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Exploiting scene constraints

Page 8: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Ordering constraint

1 2 3 4,5 6 1 2,3 4 5 6

2 1 3 4,5 6 1

2,3

4 5

6

surface slice surface as a path

occlusion right

occlusion left

Page 9: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Uniqueness constraint

• In an image pair each pixel has at most one corresponding pixel • In general one corresponding pixel • In case of occlusion there is none

Page 10: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Disparity constraint

surface slice surface as a path

bounding box

use reconstructed features to determine bounding box

constant disparity surfaces

Page 11: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Stereo matching

Optimal path (dynamic programming )

Similarity measure (SSD or NCC)

Constraints • epipolar • ordering • uniqueness • disparity limit

Trade-off • Matching cost (data) • Discontinuities (prior)

Consider all paths that satisfy the constraints

pick best using dynamic programming

Page 12: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Hierarchical stereo matching D

owns

ampl

ing

(G

auss

ian

pyra

mid

)

Dis

pari

ty p

ropa

gati

on

Allows faster computation

Deals with large disparity ranges

Page 13: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Disparity map

image I(x,y) image I´(x´,y´) Disparity map D(x,y)

(x´,y´)=(x+D(x,y),y)

Page 14: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent
Page 15: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Energy minimization

(Slide from Pascal Fua)

Page 16: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Graph Cut

(Slide from Pascal Fua)

(general formulation requires multi-way cut!)

Page 17: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

(Boykov et al ICCV‘99)

(Roy and Cox ICCV‘98)

Simplified graph cut

Page 18: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent
Page 19: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Stereo matching with general camera configuration

Page 20: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Image pair rectification

Page 21: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Planar rectification

Bring two views to standard stereo setup (moves epipole to ∞) (not possible when in/close to image)

~ image size

(calibrated)

Distortion minimization (uncalibrated)

Page 22: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent
Page 23: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Polar re-parameterization around epipoles Requires only (oriented) epipolar geometry Preserve length of epipolar lines Choose ∆θ so that no pixels are compressed

original image rectified image

Polar rectification (Pollefeys et al. ICCV’99)

Works for all relative motions Guarantees minimal image size

Page 24: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

polar rectification

planar rectification

original image pair

Page 25: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Example: Béguinage of Leuven

Does not work with standard Homography-based approaches

Page 26: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Stereo camera configurations

(Slide from Pascal Fua)

Page 27: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

• Multi-baseline, multi-resolution • At each depth, baseline and resolution

selected proportional to that depth • Allows to keep depth accuracy constant!

Variable Baseline/Resolution Stereo (Gallup et al., CVPR08)

Page 28: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Variable Baseline/Resolution Stereo: comparison

Page 29: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-view depth fusion

• Compute depth for every pixel of reference image • Triangulation • Use multiple views • Up- and down sequence • Use Kalman filter

(Koch, Pollefeys and Van Gool. ECCV‘98)

Allows to compute robust texture

Page 30: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Plane-sweep multi-view matching

• Simple algorithm for multiple cameras • no rectification necessary • doesn’t deal with occlusions

Collins’96; Roy and Cox’98 (GC); Yang et al.’02/’03 (GPU)

Page 31: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Space Carving

Page 32: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

3D Reconstruction from Calibrated Images

Scene Volume V

Input Images (Calibrated)

Goal: Determine transparency, radiance of points in V

Page 33: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Discrete Formulation: Voxel Coloring

Discretized Scene Volume

Input Images (Calibrated)

Goal: Assign RGBA values to voxels in V photo-consistent with images

Page 34: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Complexity and Computability

Discretized Scene Volume

N voxels C colors

3

All Scenes (CN3) Photo-Consistent

Scenes

True Scene

Page 35: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Issues

Theoretical Questions Identify class of all photo-consistent scenes

Practical Questions How do we compute photo-consistent models?

Page 36: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

1. C=2 (silhouettes)

Volume intersection [Martin 81, Szeliski 93]

2. C unconstrained, viewpoint constraints Voxel coloring algorithm [Seitz & Dyer 97]

3. General Case Space carving [Kutulakos & Seitz 98]

Voxel Coloring Solutions

Page 37: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Reconstruction from Silhouettes (C = 2)

Binary Images

Approach: Backproject each silhouette Intersect backprojected volumes

Page 38: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Voxel Algorithm for Volume Intersection

Color voxel black if on silhouette in every image O(MN3), for M images, N3 voxels Don’t have to search 2N3 possible scenes!

Page 39: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Properties of Volume Intersection

Pros • Easy to implement, fast • Accelerated via octrees [Szeliski 1993]

Cons

• No concavities • Reconstruction is not photo-consistent • Requires identification of silhouettes

Page 40: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

1. C=2 (silhouettes) Volume intersection [Martin 81, Szeliski 93]

2. C unconstrained, viewpoint constraints Voxel coloring algorithm [Seitz & Dyer 97]

3. General Case Space carving [Kutulakos & Seitz 98]

Voxel Coloring Solutions

Page 41: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

1. Choose voxel 2. Project and correlate 3. Color if consistent

Voxel Coloring Approach

Visibility Problem: in which images is each voxel visible?

Page 42: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Layers

Depth Ordering: visit occluders first!

Scene Traversal

Condition: depth order is view-independent

Page 43: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Compatible Camera Configurations Depth-Order Constraint

Scene outside convex hull of camera centers

Outward-Looking cameras inside scene

Inward-Looking cameras above scene

Page 44: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Calibrated Image Acquisition

Calibrated Turntable 360° rotation (21

images)

Selected Dinosaur Images

Selected Flower Images

Page 45: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Voxel Coloring Results

Dinosaur Reconstruction 72 K voxels colored 7.6 M voxels tested 7 min. to compute on a 250MHz SGI

Flower Reconstruction 70 K voxels colored 7.6 M voxels tested 7 min. to compute on a 250MHz SGI

Page 46: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Limitations of Depth Ordering

A view-independent depth order may not exist

p q

Need more powerful general-case algorithms

Unconstrained camera positions Unconstrained scene geometry/topology

Page 47: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

1. C=2 (silhouettes)

Volume intersection [Martin 81, Szeliski 93]

2. C unconstrained, viewpoint constraints

Voxel coloring algorithm [Seitz & Dyer 97]

3. General Case Space carving [Kutulakos & Seitz 98]

Voxel Coloring Solutions

Page 48: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Space Carving Algorithm

Space Carving Algorithm

Image 1 Image N

…...

Initialize to a volume V containing the true scene

Repeat until convergence

Choose a voxel on the current surface

Carve if not photo-consistent Project to visible input images

Page 49: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Convergence Consistency Property

The resulting shape is photo-consistent all inconsistent points are removed

Convergence Property Carving converges to a non-empty shape

a point on the true scene is never removed

V’

V

p

Page 50: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

What is Computable?

The Photo Hull is the UNION of all photo-consistent scenes in V • It is a photo-consistent scene reconstruction • Tightest possible bound on the true scene • Computable via provable Space Carving Algorithm

True Scene

V

Photo Hull

V

Page 51: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Space Carving Algorithm

The Basic Algorithm is Unwieldy • Complex update procedure

Alternative: Multi-Pass Plane Sweep • Efficient, can use texture-mapping

hardware • Converges quickly in practice • Easy to implement

Page 52: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

True Scene Reconstruction

Page 53: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 54: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 55: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 56: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 57: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 58: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 59: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 60: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 61: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 62: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 63: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 64: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 65: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 66: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 67: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 68: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 69: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 70: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 71: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 72: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 73: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Multi-Pass Plane Sweep • Sweep plane in each of 6 principle directions • Consider cameras on only one side of plane • Repeat until convergence

Page 74: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Space Carving Results: African Violet

Input Image (1 of 45) Reconstruction

Reconstruction Reconstruction

Page 75: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Space Carving Results: Hand

Input Image (1 of 100)

Views of Reconstruction

Page 76: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Other Features Coarse-to-fine Reconstruction

• Represent scene as octree • Reconstruct low-res model first, then refine

Hardware-Acceleration • Use texture-mapping to compute voxel

projections • Process voxels an entire plane at a time

Limitations • Need to acquire calibrated images • Restriction to simple radiance models • Bias toward maximal (fat) reconstructions • Transparency not supported

Page 79: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

I Light Intensity

Object Color

N

Normal vector

L Lighting vector

V View Vector

R Reflection vector

color of the light

Diffuse color

Saturation point

0 1

1

1

Reflected Light in RGB color space

Dielectric Materials (such as plastic and glass)

C

Space-carving for specular surfaces (Yang, Pollefeys & Welch 2003)

Extended photoconsistency:

Page 80: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Experiment

Page 81: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Animated Views

Our result

Page 82: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Volumetric Graph cuts

ρ(x)

1. Outer surface 2. Inner surface (at

constant offset) 3. Discretize

middle volume 4. Assign

photoconsistency cost to voxels

Slides from [Vogiatzis et al. CVPR2005]

Page 83: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Volumetric Graph cuts

Source

Sink

Slides from [Vogiatzis et al. CVPR2005]

Page 84: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Volumetric Graph cuts

Source

Sink

Cost of a cut ≈ ∫∫ ρ(x) dS

S

S

cut ⇔ 3D Surface S

[Boykov and Kolmogorov ICCV 2001]

Slides from [Vogiatzis et al. CVPR2005]

Page 85: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Volumetric Graph cuts

Source

Sink

Minimum cut ⇔ Minimal 3D Surface under photo-consistency metric

[Boykov and Kolmogorov ICCV 2001]

Slides from [Vogiatzis et al. CVPR2005]

Page 86: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Photo-consistency

• Occlusion

1. Get nearest point on outer surface

2. Use outer surface for occlusions

Slides from [Vogiatzis et al. CVPR2005]

Page 87: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Photo-consistency

• Occlusion

Self occlusion

Slides from [Vogiatzis et al. CVPR2005]

Page 88: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Photo-consistency

• Occlusion

Self occlusion

Slides from [Vogiatzis et al. CVPR2005]

Page 89: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Photo-consistency

• Occlusion N

threshold on angle between normal and viewing direction threshold= ~60°

Slides from [Vogiatzis et al. CVPR2005]

Page 90: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Photo-consistency

• Score

Normalised cross correlation Use all remaining cameras pair wise

Slides from [Vogiatzis et al. CVPR2005]

Page 91: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Photo-consistency

• Score

Average NCC = C Voxel score ρ = 1 - exp( -tan2[π(C-1)/4] / σ2 )

0 ≤ ρ ≤ 1 σ = 0.05 in all experiments

Slides from [Vogiatzis et al. CVPR2005]

Page 92: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Example

Slides from [Vogiatzis et al. CVPR2005]

Page 93: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Example - Visual Hull

Slides from [Vogiatzis et al. CVPR2005]

Page 94: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Example - Slice

Slides from [Vogiatzis et al. CVPR2005]

Page 95: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Example - Slice with graphcut

Slides from [Vogiatzis et al. CVPR2005]

Page 96: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Example – 3D

Slides from [Vogiatzis et al. CVPR2005]

Page 97: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Shrinking Bias

• ‘Balooning’ force • favouring bigger volumes that fill the visual hull

L.D. Cohen and I. Cohen. Finite-element methods for

active contour models and balloons for 2-d and 3-d images. PAMI, 15(11):1131–1147, November 1993. Slides from [Vogiatzis et al. CVPR2005]

Page 98: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Shrinking Bias

• ‘Balooning’ force • favouring bigger volumes that fill the visual hull

L.D. Cohen and I. Cohen. Finite-element methods for active contour

models and balloons for 2-d and 3-d images. PAMI, 15(11):1131–1147, November 1993.

∫∫ ρ(x) dS - λ ∫∫∫ dV S V

Slides from [Vogiatzis et al. CVPR2005]

Page 99: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Shrinking Bias

Slides from [Vogiatzis et al. CVPR2005]

Page 100: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Shrinking Bias

Slides from [Vogiatzis et al. CVPR2005]

Page 101: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

wij

SOURCE

wb

wb

Graph

h

j i

wb = λh3

wij = 4/3πh2 * (ρi+ρj)/2

[Boykov and Kolmogorov ICCV 2001]

Slides from [Vogiatzis et al. CVPR2005]

Page 102: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

102

Address Memory and Computational Overhead

(Sinha et. al. 2007)

– Compute Photo-consistency only where it is needed – Detect Interior Pockets using Visibility

Graph-cut on Dual of Adaptive Tetrahedral Mesh

Page 103: 3D Photography: Stereo Vision · • uniqueness • disparity limit Trade-off • Matching cost (data) ... • At each depth, baseline and resolution ... • It is a photo-consistent

Interesting comparison of multi-view stereo approaches

http://vision.middlebury.edu/mview/