Upload
cameraculture-mit-media-lab
View
3.111
Download
4
Embed Size (px)
Citation preview
Slide 1
Introduction to Light Fields
1
The 5D Plenoptic FunctionQ: What is the set of all things that one can ever see?A: The Plenoptic Function [Adelson and Bergen 1991](from plenus, complete or full, and optic)
P(q, f, l, t)
The 5D Plenoptic FunctionQ: What is the set of all things that one can ever see?A: The Plenoptic Function [Adelson and Bergen 1991](from plenus, complete or full, and optic)
P(q, f, l, t)
Position-Angle Parameterization
2D position2D direction
s
q
Two-Plane Parameterization2D position2D position
s
u
Two-Plane Parameterization camera array2D position2D position
s
u
Two-Plane Parameterization SLR camera2D position2D position
s
u
Two-Plane Parameterization cell phone
s
u
Two-Plane Parameterization cell phone (pelican)
s
u
Application: Digital Image Refocusing[Ng 2005]
Kodak 16-megapixel sensor
125 square-sided microlenses
Application: 3D Displays
Parallax Panoramagram[Kanolt 1918]3DTV with Integral Imaging[Okano et al. 1999]MERL 3DTV[Matusik and Pfister 2004]
Multiple Sensors
12
Static Camera Arrays
Stanford Multi-Camera Array 125 cameras using custom hardware[Wilburn et al. 2002, Wilburn et al. 2005]
Distributed Light Field Camera64 cameras with distributed rendering[Yang et al. 2002]
Refocus
Temporal Multiplexing
15
Controlled Camera or Object MotionStanford Spherical Gantry[Levoy and Hanrahan 1996]
Relighting with 4D Incident Light Fields[Masselus et al. 2003]
Uncontrolled Camera or Object MotionUnstructured Lumigraph Rendering[Gortler et al. 1996; Buehler et al. 2001]
Spatial Multiplexing
18
Parallax Barriers (Pinhole Arrays)[Ives 1903]
sensorbarrierSpatially-multiplexed light field capture using masks (i.e., barriers):Cause severe attenuation long exposures or lower SNRImpose fixed trade-off between spatial and angular resolution (unless implemented with programmable masks, e.g. LCDs)
Integral Imaging (Flys Eye Lenslets)[Lippmann 1908]
sensorlensletf
Spatially-multiplexed light field capture using lenslets:Impose fixed trade-off between spatial and angular resolution
Light Field Photograph (Sensor)
Light Field Photograph (Decoded)[The (New) Stanford Light Field Archive]
looking uplooking to the rightSample Image
DEMO
lenstoys: light field camera
23
Modern, Digital ImplementationsDigital Light Field Photography Hand-held plenoptic camera [Ng et al. 2005] Heterodyne light field camera [Veeraraghavan et al. 2007]
Light Field = Array of (Virtual) CamerasSlide by Marc Levoy
2007 Marc LevoyTime = 25
Sub-apertureVirtual Camera = Sub-aperture ViewLight Field = Array of (Virtual) CamerasSlide by Marc Levoy
2007 Marc LevoyTime = 26
Sub-apertureVirtual Camera = Sub-aperture View
Light Field = Array of (Virtual) CamerasSlide by Marc Levoy
2007 Marc LevoyTime = 27
2007 Marc LevoyDEMO
http://lightfield.stanford.edu/aperture.swf?lightfield=data/chess_lf/preview.zip&zoom=1
The Lytro Light Field Camera
30
Stanford camera array
Microlens camera array
DigitalsensorMicrolens array (MLA)
Lytro: microlens array in camera
Main lensMicrolens array (MLA)DigitalsensorLens is not really thin, but can be treated as so.
330 330 hex array, 13.9 micron pitchOccluders are not required.14 Mpixel, square cropped to 11 MpixelsChoose pixel in same location behind each microlens
Microlens array (MLA)Main lensDigitalsensor# of sub-apertures=# of pixels behind each microlens(10 10)# of pixels persub-aperture image= # of microlenses(~ 120,000)
All rays pass through a sub-aperture
Sub-aperture captures on camera view
Choose pixel in same location behind each microlens
So why put the microlens array inside the camera?
Lytro camera has unusual shape
8x f/2 lens
Light field sensorBattery43-343 mm equivalent
Other Applications of Light Fields
36
Lens Glare Reduction [Raskar, Agrawal, Wilson, Veeraraghavan SIGGRAPH 2008]Glare/Flare due to camera lenses reduces contrast
Reducing Glare
Glare Reduced ImageAfter removing outliersConventional Photo
MERL, MIT Media LabGlare Aware Photography: 4D Ray Sampling for Reducing Glare
Raskar, Agrawal, Wilson & Veeraraghavan
Enhancing Glare
Glare Enhanced Image
Conventional Photo
MERL, MIT Media LabGlare Aware Photography: 4D Ray Sampling for Reducing Glare
Raskar, Agrawal, Wilson & Veeraraghavan
MERL, MIT Media LabGlare Aware Photography: 4D Ray Sampling for Reducing Glare
Raskar, Agrawal, Wilson & Veeraraghavan
MERL, MIT Media LabGlare Aware Photography: 4D Ray Sampling for Reducing Glare
Raskar, Agrawal, Wilson & Veeraraghavan
Glare due to Lens Inter-Reflections
a
Sensor
b
MERL, MIT Media LabGlare Aware Photography: 4D Ray Sampling for Reducing Glare
Raskar, Agrawal, Wilson & Veeraraghavan
Effects of Glare on ImageHard to model, Low Frequency in 2DBut reflection glare is outlier in 4D ray-spaceAngular Variation at pixel aLens Inter-reflections
a
Sensor
b
MERL, MIT Media LabGlare Aware Photography: 4D Ray Sampling for Reducing Glare
Raskar, Agrawal, Wilson & Veeraraghavan
43
Key IdeaLens Glare manifests as low frequency in 2D Image
But Glare is highly view dependentmanifests as outliers in 4D ray-space
Reducing Glare == Remove outliers among rays
a
Sensor
b
MERL, MIT Media LabGlare Aware Photography: 4D Ray Sampling for Reducing Glare
Raskar, Agrawal, Wilson & Veeraraghavan
Reducing Glare using a Light Field Camera
MERL, MIT Media LabGlare Aware Photography: 4D Ray Sampling for Reducing Glare
Raskar, Agrawal, Wilson & Veeraraghavan
Single Shot Light Field Cameras
Using Mask, this paper
Using Lenslets, Ng et al. 2005
MaskAdelson and Wang, 1992, Ng et al. 2005 Kanolt 1933, Veeraraghavan et al. 2007
MERL, MIT Media LabGlare Aware Photography: 4D Ray Sampling for Reducing Glare
Raskar, Agrawal, Wilson & Veeraraghavan
Put diagrams here46
Captured Photo: LED off
MERL, MIT Media LabGlare Aware Photography: 4D Ray Sampling for Reducing Glare
Raskar, Agrawal, Wilson & Veeraraghavan
Captured Photo: LED On
MERL, MIT Media LabGlare Aware Photography: 4D Ray Sampling for Reducing Glare
Raskar, Agrawal, Wilson & Veeraraghavan
Each Disk: Angular Samples at that Spatial Location
No Glare
MERL, MIT Media LabGlare Aware Photography: 4D Ray Sampling for Reducing Glare
Raskar, Agrawal, Wilson & Veeraraghavan
With Glare
MERL, MIT Media LabGlare Aware Photography: 4D Ray Sampling for Reducing Glare
Raskar, Agrawal, Wilson & Veeraraghavan
x
y
u
v
MERL, MIT Media LabGlare Aware Photography: 4D Ray Sampling for Reducing Glare
Raskar, Agrawal, Wilson & Veeraraghavan
Sub aperture views51
Sequence of Sub-Aperture Views
Average of all the Light Field viewsOne of the Light Field viewsLow Res Traditional Camera PhotoGlare Reduced Image
MERL, MIT Media LabGlare Aware Photography: 4D Ray Sampling for Reducing Glare
Raskar, Agrawal, Wilson & Veeraraghavan
Key Idea
Reducing Glare == Remove outlier among angular samples
a
Sensor
b
MERL, MIT Media LabGlare Aware Photography: 4D Ray Sampling for Reducing Glare
Raskar, Agrawal, Wilson & Veeraraghavan
Light-sensitive DisplaysDepth CamerasMulti-touch Interaction
Multi-touch Interaction with Thin Displays
54
Hover InteractionBiDi Screen: Thin, Depth-sensing LCDs Seamless transition from multi-touch to gesture Thin form factor (LCD)
55
LCD Modifications
56
DiffuserCameraLCDLightsPrototype
57