Embed Size (px)
DESCRIPTION
Primal-Dual Coding to Probe Light Transport Matthew O'Toole, Ramesh Raskar, and Kiriakos N. Kutulakos. ACM SIGGRAPH, 2012. Abstract: We present primal-dual coding, a photography technique that enables direct fine-grain control over which light paths contribute to a photo. We achieve this by projecting a sequence of patterns onto the scene while the sensor is exposed to light. At the same time, a second sequence of patterns, derived from the first and applied in lockstep, modulates the light received at individual sensor pixels. We show that photography in this regime is equivalent to a matrix probing operation in which the elements of the scene's transport matrix are individually re-scaled and then mapped to the photo. This makes it possible to directly acquire photos in which specific light transport paths have been blocked, attenuated or enhanced. We show captured photos for several scenes with challenging light transport effects, including specular inter-reflections, caustics, diffuse inter-reflections and volumetric scattering. A key feature of primal-dual coding is that it operates almost exclusively in the optical domain: our results consist of directly-acquired, unprocessed RAW photos or differences between them.
Citation preview
Primal-Dual Codingto Probe Light Transport
Matthew O’TooleUniversity of Toronto
Ramesh RaskarMIT Media Lab
Kyros KutulakosUniversity of Toronto
conventional photography
photo records all light paths that reach camera sensor
primal-dual coding photography
photo only records a user-defined subset of light paths
conventional photography
primal-dual coding photography
new photography regime: primal-dual codingmodulate contribution of specific light path
contributions
transport probing equationgeneralizes conventional photography & confocal imaging [Wilson et al. 96]
combine illumination coding [Schechner et al. 07] & exposure coding [Hitomi et al. 11]
efficient primal-dual codes by stochastic matrix probingprovides convergence & optimality guarantees [Bekas et al. 07]
1. the result of light transport from sources to sensor pixels
2. the image recorded by the camera’s sensor
what is a photo?
transport matrix [Ng et al. 03]photo
light
conventional photography
degrees of freedom
primal-dual coding photography
“probing matrix”
degrees of freedom
multiply
primal-dual coding photography
“probing matrix”
degrees of freedom
primal-dual coding photography
“probing matrix”
degrees of freedom
primal-dual coding photography
degrees of freedom
mathematical formulation of primal-dual coding photography
one-shot primal-dual coding photography
experimental setup
SLR camera LCD mask
projector
scene
beamsplitter
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
Rademacher primal-dual codes
stochastic diagonal estimator [Bekas et al. 07]
primal codes are Rademacher random vectors: random vector in
dual codes derive from primal code:
codes converge to identity probing matrix:
variance of pixel for primal-dual codes
aperture correlation (microscopy) is a diagonal estimator [Wilson et al. 96, Levoy et al. 04]
stochastic estimator for general probingdual codes for general probing matrix :
designing probing matrices
designing probing matrices
transport matrix
designing probing matrices
transport matrix
designing probing matrices
camera
pro
jector
transport matrix
designing probing matrices
camera
pro
jector
camera
pro
jector
transport matrix
designing probing matrices
camera
pro
jector
camera
pro
jector
transport matrix
designing probing matrices
camera
pro
jector
camera
pro
jector
transport matrix
coaxial example: contrast-enhancing direct light
conventionalphoto
all light paths
coaxial example: contrast-enhancing direct light
direct +½ indirect
conventionalphoto
direct + 1/2 indirect light paths
coaxial example: contrast-enhancing direct light
direct +½ indirect
conventionalphoto
direct +¼ indirect
direct +1/16 indirect
direct + 1/16 indirect light paths
coaxial example: capturing short to long range paths
conventional
all light paths
coaxial example: capturing short to long range paths
conventional
indirect
indirect light paths
coaxial example: capturing short to long range paths
conventional
indirect mid-range indirect
medium to long range indirect light paths
coaxial example: capturing short to long range paths
conventional
indirect mid-range indirect
long-range indirect
long range indirect light paths
coaxial example: separating light transport effects
conventional
all light paths
coaxial example: separating light transport effects
conventional
indirect
indirect light paths
coaxial example: separating light transport effects
conventional
indirect direct + backscatter
direct + back-scatter light paths
coaxial example: separating light transport effects
indirect [Nayar et al. 06]
indirect direct + backscatter
direct [Nayar et al. 06]
coaxial example: separating light transport effects
indirect direct + backscatter
low-freq. indirect high-freq. indirect
conventional
all light paths
coaxial example: de-scattering [Fuchs et al. 08]
conventional de-scattered image
de-scattered diagonal
coaxial example: de-scattering [Fuchs et al. 08]
coaxial example: de-scattering [Fuchs et al. 08]
conventional
direct + 1/16 indirect
direct + 1/16 indirect light paths
de-scattered image
conventional
direct + 1/16 indirect shifted indirect
de-scattered image
coaxial example: de-scattering [Fuchs et al. 08]
shifted indirect + 1/16 light paths
non-coaxial example: making 3D regions invisible
all light paths
“privacy zone”
non-coaxial example: making 3D regions invisible
non-coaxial example: making 3D regions invisible
non-coaxial example: making 3D regions invisible
non-coaxial example: making 3D regions invisible
privacy zone
non-coaxial example: making 3D regions invisible
non-coaxial example: color coding regions of space
color coded zones
non-coaxial example: color coding regions of space
non-coaxial example: color coding regions of space
real-time PDC photographymatching 1440Hz DLP kits for illumination and attenuation
real-time PDC photography
projector
matching 1440Hz DLP kits for illumination and attenuation
real-time PDC photography
mask
camera
matching 1440Hz DLP kits for illumination and attenuation
DMD
additional details on primal-dual coding
non-negative primal-dual codesmany cases realized with strictly non-negative codesfor all others, subtract two photos computationally
codes robust to optical misalignmentsaccount for both spatial and temporal misalignments
one-shot versus multi-shotone-shot limited by projector display ratemulti-shot limited by camera read noise
concluding remarks
primal-dual coding photographyproduces new types of photos governed by probing equation
combines coded illumination and coded exposure
guarantees optimality and convergence rate of codes
can be a see-through device
future worknew possibilities for optical applications
combine with existing computational photography techniques (image-based relighting,geometry acquisition, …)
Matthew O’TooleUniversity of Toronto
Ramesh RaskarMIT Media Lab
Kyros KutulakosUniversity of Toronto
Primal-Dual Codingto Probe Light Transporthttp://www.dgp.toronto.edu/~motoole/primaldualcoding.html
64 codes
16 codes 32 codes8 codes
128 codes 256 codes
