Discontinuity Preserving Stereo with Small Baseline

Multi-Flash Illumination

Rogerio Feris1, Ramesh Raskar2, Longbin Chen1, Karhan Tan3 and Matthew Turk1

1University of California, Santa Barbara2Mitsubishi Electric Research Labs

3Epson Palo Alto Lab

Introduction

Correspondence Problem

Stereo Near Depth Discontinuities:

- Occlusion Problem

- Perspective Distortions

- Violation of Smoothness Constraints

Passive Versus Active Methods

Introduction

Our Approach:

Small Baseline Multi-Flash Illumination

- Simple, Inexpensive

- Compact, Self-Contained

- Discontinuity Preserving

Depth Edges with Multi-Flash

Raskar, Tan, Feris, Yu, Turk – ACM SIGGRAPH 2004

Bottom Flash Top Flash Left Flash Right Flash

Ratio images and directions of epipolar traversal

Shadow-Free

Depth Edges

Shadow-Free Depth Edges

Qualitative Depth Map

Qualitative Depth

Sign of Depth Edge

- Indicates which side is the foreground and which side is the background

Shadow Width

- Encodes object relative distances

Sign of Depth Edge

+ -+-

(+) Foreground (-) Background

Original Ratio Left Ratio Right Signed Edges

Shadow Width Bottom Flash Image Ratio Image

Plot Along Scanline

Shadow Width Bottom Flash Image Ratio Image

Shadow Width Estimation:

Meanshift Segmentation algorithm applied on the ratio image

Imaging Geometry

Object

Flash

Shadow

CameraB

z1

z2

f

21

12 )(

zz

zzfBd

Shadow Width

d

Qualitative Depth

Working on this Equation …

)log()log()1log(

)11log()1log(

1

12

1

22

1

22

zzd

z

z

fB

dz

z

z

fB

dz

Log Depth DifferenceShadow Width

Gradient-Domain

Problem!

Qualitative Depth

1) Compute Sharp Depth Gradient G = (Gh,Gv)

otherwise ),()1log(

edgedepth anot is y)(x, if 0),(

yxsdyxG

hhh

Log Depth Difference Sign of depth edge

2) Compute Q’ by integrating G (Poisson Equation)3) Qualitative depth map Q = exp(Q’)

Qualitative Depth

Useful Prior Information for Stereo !

Occlusion Map

Partial Occlusion Problem

Object

Camera

Occlusion

A B

(Seen by A but not by B)

Occlusion Bounded by Shadows

Object

CameraA B

Flash

Occlusion (Seen by A but not by B)

Occlusion Bounded by Shadows

Object

CameraA B

Flash

Lower Bound Shadow

Occlusion Bounded by Shadows

Object

CameraA B

Flash

Upper Bound Shadow

Occlusion Bounded by Shadows

Object

Camera

Occlusion

A B

Average of Upper/Lower Shadow widths

Flash

Occlusion Bounded by Shadows

Occlusion Map

Left View Right View

Discontinuity Preserving Stereo

Matching

Local Stereo

Problem: Shape and size of correlation window

- Small Window Ambiguities / Noise

- Large Window Problems at Depth Discontinuities

Depth Edge Preserving Local Stereo

Object Boundary (Depth Edge)

Correlation Window

Local Stereo

Smooth Disparity

Delimited by depth edges + Occlusions

Correlation Window

Problem: Shape and size of correlation window

- Small Window Ambiguities / Noise

- Large Window Problems at Depth Discontinuities

Depth Edge Preserving Local Stereo

Local Stereo

Left View Depth Edges + Occlusion Ground Truth

Challenging Scene:

- Ambiguous patterns, textureless regions, geometrically complex object, thin structures

Local StereoConventional 9x9

Conventional 31x31

Our Approach 31x31--- Conventional

Stereo Our Approach

Global Stereo

Global Optimization – Markov Random Field (MAP-MRF)

X = {xs} Disparity of each pixel (Hidden)

Y = {ys} Matching cost at each disparity (Observed)

X3 X1 X2 X7

X4 X6

X5 X8

y1 y2

Global Stereo

Global Optimization – Markov Random Field (MAP-MRF)

s s sNt

tsstsss xxyxYXP)(

),(),()|(

X = {xs} Disparity of each pixel (Hidden)

Y = {ys} Matching cost at each disparity (Observed)

Data Term

Smoothness Term

Inference by Belief Propagation [Jian Sun et al, 2003]

Global Stereo

Qualitative Depth Map as Evidence

- Used to set the smoothness term

- Information propagation is stopped at depth edges

- Encourage disparities for neighboring pixels according to depth difference in qualitative map

Occlusion Penalty

Global Stereo

Conventional Belief Propagation Our Approach

RMS: 0.9589 RMS: 0.4590

Conclusions

Contributions

- Stereo with small baseline illumination

- Useful Feature Maps (Qualitative Depth + Occlusion Map)

- Enhanced Local and Global Stereo Algorithms

Pros / Cons

- Robust, Simple, Inexpensive and Compact

- Limited to handle outdoor scenes and motion

Website (datasets, source code)

- http://www.cs.ucsb.edu/~rferis/multi-flash-stereo

Thank you !

Multi-Flash Stereo Webpage

http://www.cs.ucsb.edu/~rferis/multi-flash-stereo

Four Eyes Lab, UCSB

http://ilab.cs.ucsb.edu

Occlusion Bounded by Shadows

Occlusion Detection by averaging length of shadows

Images taken with light sources surrounding the other camera