Coded Photography

Lenslet-based Light Field camera

[Adelson and Wang, 1992, Ng et al. 2005 ]

Light Field Inside a Camera

Stanford Plenoptic Camera [Ng et al 2005]

4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens

Contax medium format camera Kodak 16-megapixel sensor

Adaptive Optics microlens array 125μ square-sided microlenses

Digital Refocusing

[Ng et al 2005]

Can we achieve this with a Mask alone?

Mask based Light Field CameraMask Sensor

[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]

Mask based Light Field CameraMask Sensor

1/f0

Mask Tile

Cosine Mask Used

Captured 2D Photo

Encoding due to Mask

[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]

2D FFT

Traditional Camera Photo

Heterodyne Camera Photo

Magnitude of 2D FFT

2D FFT

Magnitude of 2D FFT

In Focus Photo

LED

2D Photo

Out of Focus Photo: Open Aperture

Out of Focus Photo: Coded Aperture

Captured Blurred Photo

[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]

Refocused on Person

Increase DoF +large aperture

Out of Focus Photo: Coded Aperture

Engineering the PSF when you cannot capture Lightfield

Digital Refocusing

Captured Blurred Photo

Digital Refocusing

Refocused Image on Person

Mask? SensorMask

SensorMask? SensorMask? Sensor

Digital Refocusing

Mask? SensorMask

SensorMask? Sensor

MaskSensor

Mask? Sensor

Heterodyne Light Field Camera

Digital Refocusing

2D Photo

4D Light Field

Computing 4D Light Field2D Sensor Photo, 1800*1800 2D Fourier Transform

2D FFT

Rearrange 2D tiles into 4D planes200*200*9*94D IFFT

4D Light Field

9*9=81 spectral copies

200*200*9*9

MaskSensor

MaskSensor

Digital Refocusing

Heterodyne Light Field Camera

Mask? Sensor

Mask = more information?

[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan], Siggraph 2007

MERL Mask-Enhanced Cameras: Heterodyned Light Fields & Coded Aperture Veeraraghavan, Raskar, Agrawal, Mohan & TumblinDifferences with Plenoptic Camera

• Micro-lens array

• Samples individual rays

• Needs alignment precision

• Some pixels wasted

• Narrowband Cosine Mask

• Samples coded comb of rays

• More flexible

• No wastage

- Half brightness, diffraction

Mask

Sensor

Microlens array

Sensor

Plenoptic Camera Heterodyne Camera

Novel Sensors

• Color– Foveon

• Dynamic Range– HDR Camera, Log sensing– Gradient sensing

• Identity– Demodulation

• 3D– ZCam, Canesta

• Motion– Line scan Camera– Flutter Shutter

Foveon: All Colors at a Single Pixel

High Dynamic Range

Fuji's SuperCCD S3 Pro

Sensor with high and low sensitivity sensors per pixel location

Gradient Camera

• Sense Pixel Intensity Difference with unknown locally adaptive gain

• Reconstruct image from 2D gradient field

Ramesh Raskar, MERLWork with Jack Tumblin, Northwestern U,

Amit Agrawal, U of Maryland

High Dynamic Range Images

Scene Intensity camera saturation map

Gradient camera saturation map

Intensity camera fail to capture rangeGradients saturate at very few isolated pixels

Natural Scene Properties

x1

105

x1

105

Intensity Gradient

Intensity Histogram Gradient Histogram

1 105 -105 105

Motion _ _

Line Scan Camera: PhotoFinish 2000 Hz

Figure 2 results

Photo with motion blur

Rectified Image to make motion lines parallel to scan lines.

Image Deblurred by solving a linear system.

Approx Cut-out

Fluttered Shutter Camera[Raskar, Agrawal, Tumblin] Siggraph2006

Coded Exposure Coded Aperture

Temporal 1-D broadband code

Spatial 2-D broadband code

Novel Sensors

• Color– Foveon

• Dynamic Range– HDR Camera, Log sensing– Gradient sensing

• Identity– Demodulation

• 3D– ZCam, Canesta

• Motion– Line scan Camera– Flutter Shutter

Perspective? Or Not?

Agrawala et al, Long Scene Panoramas, Siggraph 2006

Rademacher et al, MCOP, Siggraph 1998

Multiperspective Camera?

[ Jingyi Yu’ 2004 ]

Future ..• ‘Cloth-cam’: ‘Wallpaper-cam’

– Fusion of 4D light emission and 4D capture in the surface of a cloth…

• Human Augmentation– Cameras to replace human eyes, for blind or limited vision– Camera on the ‘back’

• More Sensors– GPS, Compass, Temperature, fingerprint recognition, face recognition– When, Where, What, How .. Why?

• Photo Sharing and Community:– Photo Clip and Scene Completion– City Scanning, Live

Light Sensitive Fabric

Bayindir, Fink 2004

Computational Photography

Novel Illumination

Novel Cameras

Scene: 8D Ray Modulator

Display

GeneralizedSensor

Generalized OpticsProcessing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

R ≈ 0.0

G ≈ 0.2

B ≈ 0.8

Fixed Color Gamut

B

G

R

“Best” primaries compromise:

Wide Gamut vs. High Power

B

G

R

Wider color gamut

λ400nm 700nm550nm

Adaptive Color Primaries

Agile Spectrum Imaging

With Ankit Mohan, Jack Tumblin [Eurographics 2008]

C

B

A

A’

B’

C’

Pinhole

Lens L1

Prism orDiffraction Grating

Lens L2Sensor

Rainbow Plane

C’’

B’’

A’’

Scene

Rainbow Plane inside Camera

Computational Photography1. Epsilon Photography

– Low-level vision: Pixels– Multi-photos by perturbing camera parameters– HDR, panorama, …– ‘Ultimate camera’

2. Coded Photography– Mid-Level Cues:

• Regions, Edges, Motion, Direct/global– Single/few snapshot

• Reversible encoding of data– Additional sensors/optics/illum– ‘Scene analysis’

3. Essence Photography– High-level understanding

• Not mimic human eye• Beyond single view/illum

– ‘New artform’