PowerPoint Presentation
Light FieldsProperties and applicationsOutlineWhat are light fields Acquisition of light fields from a 3D scenefrom a real world sceneImage rendering from light fieldsChanging viewing angleChanging the focal plane Sampling and reconstructionDepth vs spectral supportOptimal reconstructionAnalysis of light transport
OutlineWhat are light fields Acquisition of light fields from a 3D scenefrom a real world sceneImage rendering from light fieldsChanging viewing angleChanging the focal plane Sampling and reconstructionDepth vs spectral supportOptimal reconstructionAnalysis of light transport
The Plenoptic Function
Plenus Complete, full. Optic - appearance, look.The set of things one can ever see
Light intensity as a function ofViewpoint orientation and positionTimeWavelength7D function!The 5D Plenoptic FunctionIgnoring wavelength and time We need a 5D function to describe light rays across occlusions2D orientation3D position
5The Light Field (4DAssuming no occlusions Light is constant across raysNeed only 4D to represent the space of RaysIs this assumption reasonable?In free space, i.e outside the convex hull of the scene occluders
Light field is the basic mathematical object in computational photography, graphics etc.6The Light FieldParameterizationsPoint on a Plane or curved Surface (2D) and Direction on a Hemisphere (2D)Two Points on a SphereTwo Points on two different Planes
Two Plane ParameterizationConvenient parameterization for computational photographyWhy?Similar to camera geometry (i.e. film plane vs lens plane)Linear parameterization - easy computations , no trigonometric functions, etc.
2D light fieldUsed for visualization. Assume the world is flat (2D)
The image a pinhole at (u,v) capturesAll views of a pixel (s,t)IntuitionLight Field Rendering , Levoy Hanrahan '96.10OutlineWhat are light fields Acquisition of light fields from a 3D scenefrom a real world sceneImage rendering from light fieldsChanging viewing angleChanging the focal plane Sampling and reconstructionDepth vs spectral supportOptimal reconstructionAnalysis of light transport
Acquisition of Light FieldsSynthetic 3D Scene
Discretize s,t,u,v and capture all rays intersecting the objects using a standard Ray Tracer
12Acquisition of Light Fields
Real world scenesWill be explained in more detail next week13OutlineWhat are light fields Acquisition of light fields from a 3D scenefrom a real world sceneImage rendering from light fieldsChanging viewing angleChanging the focal plane Sampling and reconstructionDepth vs spectral supportOptimal reconstructionAnalysis of light transport
Changing the View PointProblem: Computer GraphicsRender a novel view point without expensive ray tracingSolution:Sample a Synthetic light field using Ray TracingUse the Light Field to generate any point of view, no need to Ray Trace
Light Field Rendering , Levoy Hanrahan '96. view , views15Changing the View PointConceptually: Use Ray Trace from all pixels in image plane
Actually: Use Homographic mapping from XY plane to the VU and TS, and lookup resulting ray radiance.
pinholeLight Field Interpolation
NNNN + LinearLinearChanging the focal plane
Fourier Slice Photography , Ng, 05The goal, given a light field, generate views with different focal planes.18In-Camera Light Field Parameterization
We have an in camera light field parameterized by the lens plane and the sensor plane. Moving the sensor, i.e the focal plane, is equivalent to changing the image plane. The depth at which points appear focused.19The camera operatorCan define a camera as an operator on the Light Field.The conventional camera operator:
yx [Stroebel et al. 1986]A camera may be represented as an operator on the light field. For example the conventional camera. Phi is angle between the sensor plane normal and the incoming ray (x,y,u,v)20Reminder - Thin lens formulaDD1DD1const+=To focus closer - increase the sensor-to-lens distance.21Refocusing - Reparameterization
In order to get a light field with a new focal plane we reparametrize the light field which is equivalent to a shear of the light field.22Reparametrization - 4D
Refocusing - ReparameterizationRefocusReparameterization of the light fieldShearing of the Light field
Refocusing camera operatorShear and Integrate the original light field
*(cos term from conventional camera model is absorbed into L)
25Computation of Refocusing OperatorNave ApproachFor every X,Y go over all U,V and calculate the sum after reparameterization => O(n^4)
Can we do better ????
yx
26Fourier Analysis of the Camera OperatorProjection in the spatial domain Slicing in the fourier domainGiven that:
Then:
27Fourier Slice TheoremF Fourier Transform OperatorI Integral Projection OperatorS Slicing Operator
Known from CT in 2DApplicable (non trivially to N,M)Specifically for 4D , our light field28Fourier Analysis of the Camera Operator
Immediate result Faster Algorithm!29
Fourier Slice Photography , Ng, 05Fourier Slice Photography Thm More corollaries Two important results that are worth mentioning:1. Filtered Light Field Photography Thm
2. The light field dimensionality gap
Filtered Light Field Photography ThmTheorem: Filtered Light Field Photography
Note that when we do the refocusing were not really working on the actual light field, but on a filtered (sampled) representation of it.32The light field dimensionality gap
The light field is 4DIn the frequency domain The support of all the images with different focus depth is a 3D manifold
This observation was used in order to generate new views of the scene from a focal stack (Levin et al. 2010)
OutlineWhat are light fields Acquisition of light fields from a 3D scenefrom a real world sceneImage rendering from light fieldsChanging viewing angleChanging the focal plane Sampling and reconstructionDepth vs spectral supportOptimal reconstructionAnalysis of light transport
Sampling and reconstruction of light fieldsIn many cases the sampling rate is bounded due to camera limitations.We want to understand the spectrum of the light fields better in order to reconstruct betterTwo main papers go in this directionPlenoptic sampling (SIGGRAPH 2000) light field spectrum VS. scenes depthFrequency analysis of light transport (SIGGRAPH 2005) - light field spectrum VS. Path of the lightWe need to understand the spectrum of the light field in order to understand how to sample/reconstruct the acquired light field
35Light Field SamplingLight Field Acquisition DiscretizationLight Field Sampling is LimitedExample Camera Array:
u,vt,s36Sampling in frequency domainAliasing in the frequency domain
Need to analyze Light Field Spectrum
*=
ALIASING!
No Aliasing! Low pass filter . ( )37Scene Depth and Light FieldLight Field Spectrum is related to Scene DepthFrom Lambertian property each point in the scene corresponds to a line in the Light FieldLine slope is a function of the depth (z) of the point.
Plenoptic Sampling , Chai et al., 00.38Spectral Support of Light FieldConstant DepthSceneLight Field
LF SpectrumPlenoptic Sampling , Chai et al., 00.39Spectral Support of Light FieldVarying Depth
SceneLF SpectrumPlenoptic Sampling , Chai et al., 00.Spectral Support of Light Field
Plenoptic Sampling , Chai et al., 00.41Reconstruction Filters
Optimal Slope for filter:
Plenoptic Sampling , Chai et al., 00.LimitationsAssumptionsLambertian surfaces
Free Space No occlusions
Frequency Analysis of Light TransportInformally: Different features of lighting and scene causes different effects in the Frequency Content Blurry Reflections
Shadow Boundries
Low frequencyHigh frequencyA Frequency analysis of Light Transport , Durand et al. 05.Not Wave Optics!!!
Frequency Analysis of Light TransportLook at light transport as a signal processing system.Light source is the input signalInteraction are filters / transforms
SourceTransportOcclusionTransportReflection(BRDF)Local Light FieldWe study the local 4D Light Field around a central Ray during transportIn Spatial DomainIn Frequency Domain
* Local light field offers us the ability to talk about the Spectrum In a local setting
47Local Light Field (2D) Parameterization The analysis is in flatland, an extension to 4D light field is available
x-v parameterizationx- parameterizationA Frequency analysis of Light Transport , Durand et al. 05.Example Scenario
ReflectionA Frequency analysis of Light Transport , Durand et al. 05.Light Transport Spatial DomainLight Propagation Shear of the local Light FieldNo change in slope (v)Linear change in displacement (X)
Light Transport Frequency DomainShear in spatial domain is also a shear in Frequency domain
OcclusionSpatial domain:Occlusion pointwise multiplication in the spatial domainThe incoming light field is multiplied by the binary occlusion function of the occluders.
Frequency domain convolution in the frequency domain:
Occlusion example
53Reflection
* Similar analysis for curved surfaces is also presented in the paper54Reflection - cosine termSpatial domain - multiplication::
Frequency domain:
cosine term exampleIncoming Light fieldLight fieldAfter cosine term
56Reflection Mirror reparameterization
Reparameterization exampleIncoming Light fieldLight fieldAfter reparameterization
Reflection - BRDFBRDF Intuition*=directiondirectionx (space)x (space)60Reflection - BRDFBRDF example
Incoming Light fieldLight fieldAfter BRDF change
-