Inverse Volume Rendering with Material Dictionaries Ioannis
Gkioulekas 1 Shuang Zhao 2 Kavita Bala 2 Todd Zickler 1 Anat Levin
3 1 Harvard 3 Weizmann 2 Cornell 1
Slide 2
Most materials are translucent 2 jewelry skin architecture
Photo credit: Bei Xiao, Ted Adelson food
Slide 3
We know how to render them 3 Monte-Carlo rendering material
parameters Veach 1997, Dutr et al. 2006 ? rendered image
Slide 4
We show how to measure them 4 inverse rendering material
parameters rendered image captured photograph
Slide 5
Our contributions 5 material 1. exact inverse volume rendering
with arbitrary phase functions! 2. validation with calibration
materials known parameters 3. database of broad range of materials
thinthick non- dilutable solids
Slide 6
material sample Why is inverse rendering so hard? 6 radiative
transfer random walk of photons inside volume volume light
transport has very complex dependence material parameters thinthick
non- dilutable solids
Slide 7
thinthick non- dilutable solids Light transport approximations
7 Previous approach: single-scattering random walk of photons
inside volume single-bounce random walk Narasimhan et al. 2006
Slide 8
Light transport approximations 8 Previous approach: diffusion
Jensen et al. 2001 Papas et al. 2013 isotropic distribution of
photons parameter ambiguity material 1 material 2 random walk of
photons inside volume thinthick non- dilutable solids
Slide 9
Inverse rendering without approximations 9 random walk of
photons inside volume exact inversion of random walk thinthick non-
dilutable solids
Slide 10
Our approach 10 appearance matching ii. operator-theoretic
analysis i. material representation iii. stochastic
optimization
Slide 11
Background 11 phase function p() scattering coefficient s
extinction coefficient t m = ( t s p()) random walk of photons
inside medium
Slide 12
Papas et al. 2013 Phase function parameterization 12 not
general enough Henyey-Greenstein lobes Chen et al. 2006 Donner et
al. 2008 Fuchs et al. 2007 Goesele et al. 2004 Gu et al. 2008
Hawkins et al. 2005 Holroyd et al. 2011 McCormick et al. 1981 Pine
et al. 1990 Prahl et al. 1993 Wang et al. 2008 Gkioulekas et al.
2013 Narasimhan et al. 2006 Jensen et al. 2001 Previous approach:
single-parameter families
Slide 13
m = q q m q p = q q p q D = {m 1, m 2, , m Q } Dictionary
parameterization 13 tent phase functions D = {p 1, p 2, , p Q }
p1p1 p2p2 p3p3 p4p4 p5p5 p6p6 p7p7 p8p8 p9p9 p 10 p 11 dictionary
of arbitrary p similarly for t and s 11 22 33 44 55 66 77 88 99 10
11 D phase functions materials t = q q t,q s = q q s,q
Slide 14
Our approach 14 appearance matching ii. operator-theoretic
analysis i. material representation iii. stochastic optimization m
= q q m q
Slide 15
Operator-theoretic analysis 15 m = ( t s p()) random walk of
photons inside medium discretized random walk paths propagation
step
Slide 16
total radiance K() = q q K q Operator-theoretic analysis 16 m =
( t s p()) discretized random walk paths propagation step L(x, )
radiance at all medium points and directions L n+1 (x, ) = L n (x,
)K rendering operator R = (I - K) -1 L input L = n L n L(x, ) = R L
input (x, ) radiance after n steps radiance after n+1 steps R()= (I
- q q K q ) -1 dictionary representation: m = q q m q
Slide 17
Our approach 17 appearance matching ii. operator-theoretic
analysis i. material representation iii. stochastic optimization m
= q q m q R()= (I - q q K q ) -1
Slide 18
Stochastic optimization 18 appearance matching analytic
operator expression for gradient! R() render()single-step q
render() R()KqKq gradient descent optimization for inverse
rendering min photo - render() 2
Slide 19
Stochastic optimization 19 exact gradient descent for k = 1, ,
N, k = k - 1 - a k end N = a few hundreds several CPU hours * =
intractable exact
Slide 20
Stochastic optimization 20 Monte-Carlo rendering to compute 10
2 samples noisy + fast 10 4 samples 10 6 samples accurate +
slow
Slide 21
Stochastic optimization 21 exact gradient descent for k = 1, ,
N, k = k - 1 - a k end N = a few hundreds several CPU hours * =
intractable stochastic gradient descent for k = 1, , N, k = k - 1 -
a k end N = a few hundreds few CPU seconds * = solvable
exactnoisy
Slide 22
Theory wrap-up 22 appearance matching ii. operator-theoretic
analysis i. material representation iii. stochastic optimization m
= q q m q R()= (I - q q K q ) -1 noisy min photo - render() 2
Slide 23
Our contributions 23 material 1. exact inverse volume rendering
with arbitrary phase functions! 2. validation with calibration
materials known parameters 3. database of broad range of materials
thinthick non- dilutable solids
Acquisition setup 25 material sample frontlighting backlighting
camera
Slide 26
Acquisition setup 26 bottom rotation stage top rotation stage
material sample frontlighting backlighting material sample
frontlighting camera backlighting bottom rotation stage top
rotation stage camera
Slide 27
Validation 27 Frisvad et al. 2007 polystyrene monodispersions
aluminum oxide polydispersions very precise dispersions (NIST
Traceable Standards) calibration materials known parameters Mie
theory size % particle material medium material
Slide 28
Parameter accuracy 28 polystyrene 1polystyrene 2polystyrene
3aluminum oxide all parameters estimated within 4% error comparison
of ground-truth and measured parameters ground-truth measured
Henyey-Greenstein fit -0 p()
Slide 29
Matching novel measurements 29 captured rendered rendered with
HGprofiles polystyrene 3 comparison of captured and rendered images
images under unseen geometries predicted within 5% RMS error
ground-truth measured Henyey-Greenstein fit
Slide 30
Our contributions 30 material 1. exact inverse volume rendering
with arbitrary phase functions! 2. validation with calibration
materials known parameters 3. database of broad range of materials
thinthick non- dilutable solids
Effect of phase function 37 mixed soap measured phase function
Henyey-Greenstein fit -0 p() rendered image chromaticity measured
Henyey-Greenstein fit
Slide 38
Discussion 38 faster capture and convergence: trade-offs
between accuracy, generality, mobility, and usability more
interesting materials: more general solids, heterogeneous volumes,
fluorescing materials other setups: alternative lighting (basis,
adaptive, high- frequency), geometries, or imaging (transient
imaging)
Slide 39
Take-home messages 39 material 1. exact inverse volume
rendering with arbitrary phase functions! 2. validation with
calibration materials known parameters 3. database of broad range
of materials thinthick non- dilutable solids
Slide 40
Acknowledgements 40 Henry Sarkas (Nanophase) Wenzel Jakob
(Mitsuba) Funding: National Science Foundation European Research
Council Binational Science Foundation Feinberg Foundation Intel
Amazon http://tinyurl.com/sa2013-inverse Database of measured
materials:
Slide 41
Error surface 41 appearance matching min photo - render()
2
Slide 42
Light generation 42 MEMS light switch RGB combiner blue (480
nm) laser green (535 nm) laser red (635 nm) laser