Raskar Ilp Oct08 Web

Where are the cameras?

*

We focus on creating tools to
better capture and share visual information

The goal is to create an entirely
new class of imaging platforms
that have an understanding of the world that far exceeds human ability
and produce meaningful abstractions that are well
within human comprehensibility

Ramesh Raskar

*

Raskar, Camera Culture, MIT Media Lab

Questions
What will a camera look like in 10,20 years?
*


Cameras of Tomorrow

*

MIT Media Lab

Approach
Not just USE but CHANGE cameraOptics, illumination, sensor, movementExploit wavelength, speed, depth, polarization etcProbes, actuatorsWe have exhausted bits in pixelsScene understanding is challengingBuild feature-revealing camerasProcess photonsTechnology, Applications, SocietyWe study impact of Imaging on all fronts
*

Mitsubishi Electric Research Laboratories

Raskar 2006

Spatial Augmented Reality

Curved

Planar

Non-planar

Single
Projector

Multiple
Projectors

Pocket-Proj

Objects

Computational Illumination

1998

1998

2002

2002

1999

2002

2003

1998

1997

Computational Camera and Photography

Projector

j

User : T

?

*
1.bin

Motion Blurred Photo

*

Car result
2.bin

Flutter Shutter Camera

Raskar, Agrawal, Tumblin [Siggraph2006]

LCD opacity switched
in coded sequence

*

Short Exposure

Traditional

MURA

Coded

Deblurred
Result

Captured
Single
Photo

Shutter

Banding Artifacts and some spatial frequencies are lost

Dark
and noisy

*

Low signal to noise ratio

Blurring == Convolution

Traditional Camera: Shutter is OPEN: Box Filter

PSF == Sinc Function

Sharp Photo

Blurred Photo

Fourier Transform

*

How is the blurred image formed? Its a convolution with box filter

Flutter Shutter: Shutter is OPEN and CLOSED

Sharp Photo

Blurred Photo

PSF == Broadband Function

Fourier Transform

Preserves High Spatial Frequencies

*

Coded exposure makes the filter broadband

Traditional

Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

*

Comparisons

*

License plate example: Blur = 60 pixels

Can you guess what the car make is ? How many think it is the Audi ? Actually it is a Folksvagon.

Coded Exposure

Temporal 1-D broadband code: Motion Deblurring

Coded Aperture

Spatial 2-D broadband mask: Focus Deblurring

*

Coded Aperture Camera

The aperture of a 100 mm lens is modified

Rest of the camera is unmodified

Insert a coded mask with chosen binary pattern

*

In Focus Photo

LED

*

Out of Focus Photo: Open Aperture

*

Out of Focus Photo: Coded Aperture

*

Captured Blurred Photo

*

Reversibly encode all the information in this otherwise blurred photo

Refocused on Person

*

The glint out of focus shows the unusual pattern.

Less is More

Blocking Light == More Information

Coding in Time

Coding in Space

*

Larval Trematode Worm


Photography is the Art of Exclusion

*

Difficult to argue that the worm is performing high quality deconvolution to form an image. But in our group we are setting up experiments by creating active lighting probe to understand how worms perform visual analysis.

Shielding Light

Larval Trematode Worm

Turbellarian Worm

Coded Computational Photography
Coded ExposureMotion Deblurring [2006]Coded ApertureFocus Deblurring [2007]Glare reduction [2008]Optical HeterodyningLight Field Capture [2007]Coded IlluminationMotion Capture [2007]Multi-flash: Shape Contours [2004]Coded SpectrumAgile Wavelength Profile [2008]Epsilon->Coded->Essence Photography
http://raskar.info

*


Computational Photography

Epsilon Photography
Low-level Vision: PixelsMultiphotos by bracketing (HDR, panorama)Ultimate camera
Coded Photography
Mid-Level Cues: Regions, Edges, Motion, Direct/globalSingle/few snapshotReversible encoding of dataAdditional sensors/optics/illum
Essence Photography
Not mimic human eyeBeyond single view/illumNew artform
*

Stereo-pair is a simple example of coded photography.

Many decomposition problems, direct/global, diffuse/specular,


Epsilon Photography
Dynamic rangeExposure braketing [Mann-Picard, Debevec]Wider FoV Stitching a panoramaDepth of field Fusion of photos with limited DoF [Agrawala04]NoiseFlash/no-flash image pairs [Petschnigg04, Eisemann04]Frame rateTriggering multiple cameras [Wilburn05, Shechtman02]
Many think all this is for CP

*



Epsilon Photography
Coded Photography
Essence Photography
*



3DStereo of multiple camerasHigher dimensional LFLight Field Capturelenslet array [Adelson92, Ng05], 3D lens [Georgiev05], heterodyne masks [Veeraraghavan07]Boundaries and RegionsMulti-flash camera with shadows [Raskar08]Fg/bg matting [Chuang01,Sun06]DeblurringEngineered PSFMotion: Flutter shutter[Raskar06], Camera Motion [Levin08]Defocus: Coded aperture, Wavefront coding [Cathey95] Global vs direct illuminationHigh frequency illumination [Nayar06]Glare decomposition [Talvala07, Raskar08]Coded SensorGradient camera [Tumblin05]


Epsilon Photography
Coded Photography
Essence Photography
*




Capturing the Essence of Visual Experience
Exploiting online collectionsPhoto-tourism [Snavely2006]Scene Completion [Hays2007]Multi-perspective ImagesMulti-linear Perspective [Jingyi Yu, McMillan 2004]Unwrap Mosaics [Rav-Acha et al 2008]Video texture panoramas [Agrawal et al 2005]Non-photorealistic synthesisMotion magnification [Liu05]Image PriorsLearned features and natural statisticsFace Swapping: [Bitouk et al 2008]Data-driven enhancement of facial attractiveness [Leyvand et al 2008]Deblurring [Fergus et al 2006, Jia et al 2008]


Epsilon Photography
Coded Photography
Essence Photography
*



Ramesh Raskar and
Jack TumblinBook Publishers: A K Peters

Mask?

Sensor

4D Light Field from
2D Photo:

Heterodyne Light Field Camera

Full Resolution Digital Refocusing:


Mask?

Sensor

Mask

Sensor

Mask?

Sensor

Mask

Sensor

Light Field Inside a Camera

*

Lenslet-based Light Field camera

[Adelson and Wang, 1992, Ng et al. 2005 ]

Light Field Inside a Camera

*

Stanford Plenoptic Camera [Ng et al 2005]

4000 4000 pixels 292 292 lenses = 14 14 pixels per lens

Contax medium format camera

Kodak 16-megapixel sensor

Adaptive Optics microlens array

125 square-sided microlenses

*

Digital Refocusing

[Ng et al 2005]

Can we achieve this with a Mask alone?

*

Mask based Light Field Camera

[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]

Mask

Sensor

*

How to Capture 4D Light Field with 2D Sensor ?

What should be the pattern of the mask ?

*

Optical Heterodyning

Photographic Signal
(Light Field)

Carrier

Incident Modulated Signal

Reference
Carrier

Main Lens

Object

Mask

Sensor

Recovered
Light
Field

Software Demodulation

Baseband Audio Signal

Receiver: Demodulation

High Freq Carrier 100 MHz

Reference
Carrier

Incoming Signal

99 MHz

*

1/f0

Mask Tile

Cosine Mask Used

*

Captured 2D Photo

Encoding due to Mask

*

2D FFT

Traditional Camera Photo

Heterodyne Camera Photo

Magnitude of 2D FFT

2D FFT

Magnitude of 2D FFT

*

Computing 4D Light Field

2D Sensor Photo, 1800*1800

2D Fourier Transform, 1800*1800

2D FFT

Rearrange 2D tiles into 4D planes

200*200*9*9

4D IFFT

4D Light Field

9*9=81 spectral copies

200*200*9*9

*

How to Capture 2D Light Field with 1D Sensor ?

f

fx

f0

fx0

Band-limited Light Field

Sensor Slice

Fourier Light Field Space

*

Extra sensor bandwidth cannot capture
extra dimension of the light field

f

fx

f0

fx0

Sensor Slice

Extra sensor bandwidth

Fourier Light Field Space

*

Solution: Modulation Theorem

Make spectral copies of 2D light field

f

fx

f0

fx0

Modulation Function

*

f

Modulated Light Field

fx

f0

fx0

Modulation Function

Sensor Slice captures entire Light Field

*

1/f0

Mask Tile

Cosine Mask Used

Demodulation to recover Light Field

f

fx

Reshape 1D Fourier Transform into 2D

1D Fourier Transform of Sensor Signal

*

Mask

Sensor

Where to place the Mask?

Mask Modulation Function

fx

f

Full resolution 2D image of Focused Scene Parts

Captured 2D Photo

Image of White Lambertian Plane

divide

Coding and Modulation in Camera Using Masks

Coded Aperture for Full Resolution Digital Refocusing

Heterodyne Light Field Camera

Mask?

Sensor

Mask

Sensor

Mask

Sensor

Agile Spectrum Imaging

Programmable Color Gamut for Sensor

With Ankit Mohan, Jack Tumblin [Eurographics 2008]

R 0.0

G 0.2

B 0.8

Traditional Fixed Color Gamut

B

G

R

Adaptive Color Primaries

C

B

A

A

B

C

Pinhole

Lens L1

Prism or
Diffraction Grating

Lens L2

Sensor

Rainbow Plane

C

B

A

Scene

Rainbow Plane inside Camera

Lens Flare Reduction/Enhancement using
4D Ray Sampling

Captured

Glare Reduced

Glare Enhanced

Glare = low frequency noise in 2D
But is high frequency noise in 4DRemove via simple outlier rejection
i

j

x

Sensor

u


Raskar 2006

Spatial Augmented Reality

Computational Camera and Photography

*

Dependence on incident angle

now, we can add some blockers and modulate; the design you currently see. One deficit: the cones on the receiver surface do not stay below the lens, and do not have a precise

overlap

*

Towards a 6D Display

Passive Reflectance Field Display

Martin Fuchs, Ramesh Raskar,

Hans-Peter Seidel, Hendrik P. A. Lensch

Siggraph 2008

1

2

1

1

1 MPI Informatik, Germany 2 MIT

2D

2D

2D

*

add: measurement setup for motivation; motivate shadows + lights first; reasons for artifacts//what people should see;

View dependent 4D Display

6D = light sensitive 4D display

One Pixel of a 6D Display = 4D Display

This is one of 49 single pixel blocks for the 6D display. It projects out different patterns according to the incident light illumination.

To test it separately, we feed it with a special pattern

*

MIT Media Lab

Single shot visual hull

Lanman, Raskar, Agrawal, Taubin [Siggraph Asia 2008]

On this slide, well: (1) state that we will allow a single area X-ray source to be used, (2) outline Shield Field Imaging (SFI) and how it will apply, and (3) describe the risks and IP for our SFI approach. Recently, while working on another problem, we realized that

Only solution is time-domain multiplexing so far, new approach is turning all on at same time (frequency-division multiplexing). Give analogy to telecommunications, from time division to frequency division to code division has allowed dramatic progress; were doing the same thing in the imaging domain.

*

MIT Media Lab

Single shot 3D reconstruction:
Simultaneous Projections using Masks

On this slide, well: (1) state that we will allow a single area X-ray source to be used, (2) outline Shield Field Imaging (SFI) and how it will apply, and (3) describe the risks and IP for our SFI approach. Recently, while working on another problem, we realized that

Only solution is time-domain multiplexing so far, new approach is turning all on at same time (frequency-division multiplexing). Give analogy to telecommunications, from time division to frequency division to code division has allowed dramatic progress; were doing the same thing in the imaging domain.

*

Barcode size : 3mm x 3mmDistance from camera : 5 meter
Long Distance Bar-codes

Woo, Mohan, Raskar, [2008]

*


Projects
Lightweight Medical ImagingHigh speed TomographyMuscle, blood flow activity with wearable devices for patientsFemto-second Analysis of Light TransportBuilding and modeling future ultra-high speed camerasAvoid car-crashes, analyze complex scenesProgrammable Wavelength in Thermal RangeFacial expressions, HealthcareHuman-emotion aware computing, Fast diagnosisSecond skinWearable fabric for bio-I/O via high speed optical motion captureRecord and mimic any human motion, Care for elderly, Teach a robot
Not just react, influence

*

Vicon
Motion Capture

High-speed
IR Camera

Body-worn markers

Medical Rehabilitation

Athlete Analysis

Performance Capture

Biomechanical Analysis

*


Special Effects in the Real World

Raskar 2006

Inverse
Optical
Mo-Cap

Traditional
DeviceHigh Speed Projector + Photosensing MarkersHigh Speed Camera + Reflecting/Emitting MarkersParamsLocation, Orientation, IllumLocationSettingsNatural SettingsAmbient Light Outdoors, Stage lightingImperceptible tags Hidden under wardrobeControlled LightingVisible, High contrast Markers#of TagsUnlimitedSpace LabelingUnique IdLimitedNo Unique Id Marker swappingSpeedVirtually unlimited Optical comm compsLimitedSpecial high fps cameraCostLow Open-loop projectors Current: Projector/Tag=$100High High bandwidth camera Current Camera: $10K
*


Special Effects in the Real World

Raskar 2006

Inside of Projector

The Gray code pattern

Focusing Optics

Gray code Slide

Condensing Optics

Light Source

*

Imperceptible Tags under clothing, tracked under ambient light

*

Towards Second Skin
Coded Illumination
Motion Capture Clothing
500 Hz with Id for each Marker TagCapture in Natural EnvironmentVisually imperceptible tagsPhotosensing Tag can be hidden under clothesAmbient lighting is okUnlimited Number of TagsLight sensitive fabric for dense samplingNon-imaging, complete privacyBase station and tags only a few 10s $Full body scan + actionsElderly, patients, athletes, performersBreathing, small twists, multiple segments or peopleAnimation Analysis
*

Coded Imaging

*

Coding in Time

Coding in Space (Optical Path)

Coded Illumination

Coded Wavelength

Coded Sensing

Coded Exposure for Motion Deblurring

Coded Aperture for Extended Depth of Field

Mask-based Optical Heterodyning for Light Field Capture

Multi-flash Imaging for Depth Edge Detection

Agile Spectrum Imaging

Gradient Encoding Sensor for HDR

Forerunners ..

Mask

Sensor

Mask

Sensor

*

While the single photosensor worm, has evolved into worms with multiple photosensors .. They have been evolutionary forerunners to the human eye.

So what about the cameras here .. Maybe we are thinking about these initial types in a limited way and They are forerunners to something very exciting.

Fernald, Science [Sept 2006]

Shadow

Refractive

Reflective

Tools

for

Visual Computing

*

CPUs and computers dont mimic the human brain. And robots dont mimic human activities. Should the hardware for visual computing which is cameras and capture devices, mimic the human eye? Even if we decide to use a successful biological vision system as basis, we have a range of choices. For single chambered to compounds eyes, shadow-based to refractive to reflective optics. So the goal of my group at Media Lab is to explore new designs and develop software algorithms that exploit these designs.

*

Maybe all the consumer photographer wants is a black box with big red button. No optics, sensors or flash.

If I am standing the middle of times square and I need to take a photo. Do I really need a fancy camera?

Blind Camera

Sascha Pohflepp,
U of the Art, Berlin, 2006

*

The camera can trawl on flickr and retrieve a photo that is roughly taken at the same position, at the same time of day. Maybe all the consumer wants is a blind camera.


Cameras of Tomorrow

*

Cameras of Tomorrow
Coded ExposureMotion Deblurring [2006]Coded ApertureFocus Deblurring [2007]Glare reduction [2008]Optical HeterodyningLight Field Capture [2007]Coded IlluminationMotion Capture [2007]Multi-flash: Shape Contours [2004]Coded SpectrumAgile Wavelength Profile [2008]Epsilon->Coded->Essence Photography
http://raskar.info

*

We focus on creating tools to
better capture and share visual information

The goal is to create an entirely
new class of imaging platforms
that have an understanding of the world that far exceeds human ability
and produce meaningful abstractions that are well
within human comprehensibility

Ramesh Raskarhttp://raskar.info

*

CaptureCameras EverywhereDeep pervasive sensingAnalysisComputer VisionMo-cap Personalized services, tracking in real worldAnimationSimulation of bio/chemical/physical processes at all scalesSynthesisVirtual Human, Digital Actors, Tele-avatarsExa and Zeta-scale computing: simulate every neural activity, predict weather for weeks, simulate impact of global warmingDisplayReal world ARRealistic Displays: 6D or 8D
*

Coding in Time Coding in Space (Optical Path) Coded Illumination Coded Wavelength Coded Sensing

Coded Exposure for

Motion Deblurring

Coded Aperture for

Extended Depth of Field

Mask-based Optical

Heterodyning for

Light Field Capture

Multi-flash Imaging for

Depth Edge Detection

Agile Spectrum

Imaging

Gradient Encoding

Sensor for HDR

Art & Photos

Raskar Ilp Oct08 Web