Mitsubishi Electric Research Labs

Progressively Refined Reflectance Fields from Natural Illumination

Wojciech Matusik

Matt Loper

Hanspeter Pfister

MotivationMotivation

• Complex natural scenes are difficult to acquire• Acquisition needs to be easy and robust• Image-based lighting offers high realism • We would like to relight image-based models at

any scale (from small objects to cities)

MotivationMotivation

• Image-based Relighting– no scene geometry – just images– no assumptions about scene reflectance properties

Previous WorkPrevious Work

• Forward Approaches– Georghiades 99, Debevec 2000, Malzbender 01,

Masselus 02, Peers 03

• Inverse Approaches– Zongker 99, Chuang 00, Wexler 02

• Pre-computed Light Transport– Sloan 02, Ng 03

Reflectance FieldReflectance Field

• 8D function:

[Debevec 2000]

),;,;,;,( rrrriiiir vuvuf

(θr,φr)

(ur,vr)

(θi,φi)

(ui,vi)

Reflectance (Weighting) FunctionReflectance (Weighting) Function

• Assumes incident illumination originates at infinity

• x,y are image space coordinates

θi

φi

),;,( iiw yxf

Light Transport ModelLight Transport Model

• A light flow in the scene can be modeled as a multiple-input / multiple-output linear system:

TLB

Scene

light transport

matrix

T

Incident Light

L

Observed Image

B

Unroll to a vector Unroll to a vector

Light Transport ModelLight Transport Model

• Solve independently for each output pixel multiple-input / single-output linear system :

LTb ii

Scene

light transport

vector

Ti

Incident Light

L

Observed Pixel

bi

RepresentationRepresentation

• Approximate Ti as a sum of 2D rectangular kernels Rk,i, each with weight wk,i.

k

ikiki RwT ,,

θi

φi

Inverse EstimationInverse Estimation

• Given input images Lj we record observed pixel values bij:

• Given matrix L and vector bi the goal is to estimate Ti

– Positions and sizes of the rectangular kernels Rk,i

– Weights wk, i

jiij LTb

,...],[,...],[ 2121 LLTbb iii

Estimating Kernel WeightsEstimating Kernel Weights

• Assume that we know sizes and positions of the kernels Rk,i and would like to compute their weights

0minarg2 iiiiw wtosubjectbwA

LTb ii

02

1minarg

i

iTi

Tiii

Ti

Tiw

wtosubject

bAwwAAw

ii wLRLRw ikk

ik

,, iiwA

•Efficient solution using quadratic programming

Estimating Kernel Positions & SizesEstimating Kernel Positions & Sizes

• Hierarchical kd-tree subdivision of the kernels input image domain

• At each level choose subdivision that reduces error the most

• Kernels are non-overlapping

2

iii bwA

Kernel SubdivisionsKernel Subdivisions

specular

refractive

1 2 3 4 5 10 20 24

subsurface

scattering

glossy

hard

shadow

Subdivisions

Spatial CorrectionSpatial Correction

• The kernels search strategy does not always work• Solution: For each output pixel:

– try kernel positions and sizes of the neighboring output pixels

– try shifted versions of the current kernels– solve for new weights– keep new kernels if the error decreases

Integration with Incident IlluminationIntegration with Incident Illumination

• Is very efficient• For each output pixel i

• The incident illumination is stored as a summed-area table to evaluate

kiikiki

kikikiii LRwLRwLTb ,,,,

iik LR ,

Data AcquisitionData Acquisition

• We have built two acquisition systems– Indoor scenes / small objects

– Outdoor scenes (city)

Acquisition System IAcquisition System I

Example Input ImagesExample Input Images

Results Results

• Refractive and specular elements

Prediction Actual

Results – New IlluminationResults – New Illumination

Results - White Vertical BarResults - White Vertical Bar

Prediction Actual

ResultsResults

Estimate Actual

• Diffuse elements, shadows

Results - White Vertical BarResults - White Vertical Bar

ResultsResults

Estimate Actual

• Subsurface Scattering

Results - White Vertical BarResults - White Vertical Bar

ResultsResults

• Glossy elements and interreflections

Estimate

Actual

Results - White Vertical BarResults - White Vertical Bar

ResultsResults

• One shifted version of the same image used as input illumination

Acquisition System IIAcquisition System II

• Two Synchronized CamerasCamera #1 Camera #2

Example Observed ImagesExample Observed Images

Results – Relighting The CityResults – Relighting The City

• White vertical bar

LessonsLessons

• Inverse approaches benefit from good kernel search strategies & more computation power

• Inverse approaches are more efficient than forward approaches

• Challenges:– Scene needs to be static– Varied set of input illumination– Illumination is not at infinity

ConclusionsConclusions

• Advantages of our algorithm:– Natural Illumination Input– All-frequency Robustness– Compact Representation– Progressive Refinement– Fast Evaluation– Simplicity

Future WorkFuture Work

• New acquisition systems– object and camera are fixed w.r.t. each other and they

rotate in a single, natural environment

• Combining representations from different viewpoints and proxy geometry

• Coarse-to-fine estimation in the observed image space– start with low resolution observed images & search

exhaustively for the best kernels– propagate the kernels to higher resolution images

AcknowledgementsAcknowledgements

• Jan Kautz• Barb Cutler• Jennifer Roderick Pfister• EGSR Reviewers