Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Light Transport and

Computational Photography Inverse problems

MIT Media LabRamesh Raskar httpraskarinfo

raskarmitedu

Raskar Camera Culture MIT Media Lab

Ramesh Raskar MIT Media Lab

After X what is neXt

How to Invent

Ramesh Raskar MIT Media Lab

Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010

Ramesh Raskar MIT Media Lab

Xd

X++

X X+Y

X

X

neXt

Ramesh Raskar httpraskarinfo

Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010

Ramesh Raskar MIT Media Lab

Simple Exercise bull Image Compression

ndash Save Bandwidth and storage

What is neXt

Ramesh Raskar MIT Media Lab

Strategy 1 Xd

bull Extend it to next (or some other) dimension

Ramesh Raskar MIT Media Lab

X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm

Ramesh Raskar MIT Media Lab

Xd

X++

X X+Y

X

X

neXt

Ramesh Raskar httpraskarinfo

Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010

Research bull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Is project worthwhile Heilmeiers Questions

bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon

bull Related workndash How is it done today and what are the limits of current practice

bull Contributionndash Whats new in your approach and why do you think it will be successful

bull Motivationndash Who caresndash If youre successful what difference will it make

bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take

bull Evaluationndash What are the midterm and final exams to check for success

bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)

httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism

Great Research Strive for Five

1 Before Five teamsBe first often let others do details

2 Beyond Five yearsWhat no one is thinking about

3 Within Five layers of lsquoHumanrsquo ImpactRelevance

4 Beyond Five minutes of descriptionDeep iterative participatory

5 Fusing Five+ ExpertiseMulti-disciplinary proactive

Ramesh Raskar httpraskarinfo

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Fernald Science [Sept 2006]

Shadow Refractive

Reflective

Tools for

Visual Computin

g

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Traditional Photography

Lens

Detector

Pixels

Image

Mimics Human Eye for a Single Snapshot

Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree

Nayar

Computational Camera + Photography

Optics Sensors and ComputationsGeneralized

Sensor

Generalized OpticsComputations

Picture

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures

Computational Photography

Novel Illumination

Computational Cameras

Scene 8D Ray Modulator

Display

GeneralizedSensor

Generalized OpticsProcessing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

Computational Photography [Raskar and Tumblin]

Resources

ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011

httpwikipediaorgcomputational_photography

httpraskarinfophoto

captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience

SynthesisAnalysis

Low Level Mid Level

HighLevel

Hyper realism

Raw

Angle spectrum

aware

Non-visual Data GPS

Metadata

Priors

Cap

ture

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Looking Around Corners

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

fgbg

DirectGlobal

Computational Photography

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Raskar Camera Culture MIT Media Lab

Ramesh Raskar MIT Media Lab

After X what is neXt

How to Invent

Ramesh Raskar MIT Media Lab

Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010

Ramesh Raskar MIT Media Lab

Xd

X++

X X+Y

X

X

neXt

Ramesh Raskar httpraskarinfo

Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010

Ramesh Raskar MIT Media Lab

Simple Exercise bull Image Compression

ndash Save Bandwidth and storage

What is neXt

Ramesh Raskar MIT Media Lab

Strategy 1 Xd

bull Extend it to next (or some other) dimension

Ramesh Raskar MIT Media Lab

X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm

Ramesh Raskar MIT Media Lab

Xd

X++

X X+Y

X

X

neXt

Ramesh Raskar httpraskarinfo

Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010

Research bull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Is project worthwhile Heilmeiers Questions

bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon

bull Related workndash How is it done today and what are the limits of current practice

bull Contributionndash Whats new in your approach and why do you think it will be successful

bull Motivationndash Who caresndash If youre successful what difference will it make

bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take

bull Evaluationndash What are the midterm and final exams to check for success

bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)

httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism

Great Research Strive for Five

1 Before Five teamsBe first often let others do details

2 Beyond Five yearsWhat no one is thinking about

3 Within Five layers of lsquoHumanrsquo ImpactRelevance

4 Beyond Five minutes of descriptionDeep iterative participatory

5 Fusing Five+ ExpertiseMulti-disciplinary proactive

Ramesh Raskar httpraskarinfo

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Fernald Science [Sept 2006]

Shadow Refractive

Reflective

Tools for

Visual Computin

g

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Traditional Photography

Lens

Detector

Pixels

Image

Mimics Human Eye for a Single Snapshot

Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree

Nayar

Computational Camera + Photography

Optics Sensors and ComputationsGeneralized

Sensor

Generalized OpticsComputations

Picture

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures

Computational Photography

Novel Illumination

Computational Cameras

Scene 8D Ray Modulator

Display

GeneralizedSensor

Generalized OpticsProcessing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

Computational Photography [Raskar and Tumblin]

Resources

ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011

httpwikipediaorgcomputational_photography

httpraskarinfophoto

captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience

SynthesisAnalysis

Low Level Mid Level

HighLevel

Hyper realism

Raw

Angle spectrum

aware

Non-visual Data GPS

Metadata

Priors

Cap

ture

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Looking Around Corners

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

fgbg

DirectGlobal

Computational Photography

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Ramesh Raskar MIT Media Lab

After X what is neXt

How to Invent

Ramesh Raskar MIT Media Lab

Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010

Ramesh Raskar MIT Media Lab

Xd

X++

X X+Y

X

X

neXt

Ramesh Raskar httpraskarinfo

Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010

Ramesh Raskar MIT Media Lab

Simple Exercise bull Image Compression

ndash Save Bandwidth and storage

What is neXt

Ramesh Raskar MIT Media Lab

Strategy 1 Xd

bull Extend it to next (or some other) dimension

Ramesh Raskar MIT Media Lab

X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm

Ramesh Raskar MIT Media Lab

Xd

X++

X X+Y

X

X

neXt

Ramesh Raskar httpraskarinfo

Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010

Research bull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Is project worthwhile Heilmeiers Questions

bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon

bull Related workndash How is it done today and what are the limits of current practice

bull Contributionndash Whats new in your approach and why do you think it will be successful

bull Motivationndash Who caresndash If youre successful what difference will it make

bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take

bull Evaluationndash What are the midterm and final exams to check for success

bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)

httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism

Great Research Strive for Five

1 Before Five teamsBe first often let others do details

2 Beyond Five yearsWhat no one is thinking about

3 Within Five layers of lsquoHumanrsquo ImpactRelevance

4 Beyond Five minutes of descriptionDeep iterative participatory

5 Fusing Five+ ExpertiseMulti-disciplinary proactive

Ramesh Raskar httpraskarinfo

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Fernald Science [Sept 2006]

Shadow Refractive

Reflective

Tools for

Visual Computin

g

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Traditional Photography

Lens

Detector

Pixels

Image

Mimics Human Eye for a Single Snapshot

Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree

Nayar

Computational Camera + Photography

Optics Sensors and ComputationsGeneralized

Sensor

Generalized OpticsComputations

Picture

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures

Computational Photography

Novel Illumination

Computational Cameras

Scene 8D Ray Modulator

Display

GeneralizedSensor

Generalized OpticsProcessing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

Computational Photography [Raskar and Tumblin]

Resources

ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011

httpwikipediaorgcomputational_photography

httpraskarinfophoto

captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience

SynthesisAnalysis

Low Level Mid Level

HighLevel

Hyper realism

Raw

Angle spectrum

aware

Non-visual Data GPS

Metadata

Priors

Cap

ture

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Looking Around Corners

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

fgbg

DirectGlobal

Computational Photography

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Ramesh Raskar MIT Media Lab

Xd

X++

X X+Y

X

X

neXt

Ramesh Raskar httpraskarinfo

Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010

Ramesh Raskar MIT Media Lab

Simple Exercise bull Image Compression

ndash Save Bandwidth and storage

What is neXt

Ramesh Raskar MIT Media Lab

Strategy 1 Xd

bull Extend it to next (or some other) dimension

Ramesh Raskar MIT Media Lab

X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm

Ramesh Raskar MIT Media Lab

Xd

X++

X X+Y

X

X

neXt

Ramesh Raskar httpraskarinfo

Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010

Research bull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Is project worthwhile Heilmeiers Questions

bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon

bull Related workndash How is it done today and what are the limits of current practice

bull Contributionndash Whats new in your approach and why do you think it will be successful

bull Motivationndash Who caresndash If youre successful what difference will it make

bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take

bull Evaluationndash What are the midterm and final exams to check for success

bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)

httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism

Great Research Strive for Five

1 Before Five teamsBe first often let others do details

2 Beyond Five yearsWhat no one is thinking about

3 Within Five layers of lsquoHumanrsquo ImpactRelevance

4 Beyond Five minutes of descriptionDeep iterative participatory

5 Fusing Five+ ExpertiseMulti-disciplinary proactive

Ramesh Raskar httpraskarinfo

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Fernald Science [Sept 2006]

Shadow Refractive

Reflective

Tools for

Visual Computin

g

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Traditional Photography

Lens

Detector

Pixels

Image

Mimics Human Eye for a Single Snapshot

Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree

Nayar

Computational Camera + Photography

Optics Sensors and ComputationsGeneralized

Sensor

Generalized OpticsComputations

Picture

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures

Computational Photography

Novel Illumination

Computational Cameras

Scene 8D Ray Modulator

Display

GeneralizedSensor

Generalized OpticsProcessing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

Computational Photography [Raskar and Tumblin]

Resources

ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011

httpwikipediaorgcomputational_photography

httpraskarinfophoto

captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience

SynthesisAnalysis

Low Level Mid Level

HighLevel

Hyper realism

Raw

Angle spectrum

aware

Non-visual Data GPS

Metadata

Priors

Cap

ture

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Looking Around Corners

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

fgbg

DirectGlobal

Computational Photography

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Ramesh Raskar MIT Media Lab

Simple Exercise bull Image Compression

ndash Save Bandwidth and storage

What is neXt

Ramesh Raskar MIT Media Lab

Strategy 1 Xd

bull Extend it to next (or some other) dimension

Ramesh Raskar MIT Media Lab

X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm

Ramesh Raskar MIT Media Lab

Xd

X++

X X+Y

X

X

neXt

Ramesh Raskar httpraskarinfo

Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010

Research bull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Is project worthwhile Heilmeiers Questions

bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon

bull Related workndash How is it done today and what are the limits of current practice

bull Contributionndash Whats new in your approach and why do you think it will be successful

bull Motivationndash Who caresndash If youre successful what difference will it make

bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take

bull Evaluationndash What are the midterm and final exams to check for success

bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)

httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism

Great Research Strive for Five

1 Before Five teamsBe first often let others do details

2 Beyond Five yearsWhat no one is thinking about

3 Within Five layers of lsquoHumanrsquo ImpactRelevance

4 Beyond Five minutes of descriptionDeep iterative participatory

5 Fusing Five+ ExpertiseMulti-disciplinary proactive

Ramesh Raskar httpraskarinfo

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Fernald Science [Sept 2006]

Shadow Refractive

Reflective

Tools for

Visual Computin

g

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Traditional Photography

Lens

Detector

Pixels

Image

Mimics Human Eye for a Single Snapshot

Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree

Nayar

Computational Camera + Photography

Optics Sensors and ComputationsGeneralized

Sensor

Generalized OpticsComputations

Picture

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures

Computational Photography

Novel Illumination

Computational Cameras

Scene 8D Ray Modulator

Display

GeneralizedSensor

Generalized OpticsProcessing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

Computational Photography [Raskar and Tumblin]

Resources

ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011

httpwikipediaorgcomputational_photography

httpraskarinfophoto

captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience

SynthesisAnalysis

Low Level Mid Level

HighLevel

Hyper realism

Raw

Angle spectrum

aware

Non-visual Data GPS

Metadata

Priors

Cap

ture

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Looking Around Corners

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

fgbg

DirectGlobal

Computational Photography

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Ramesh Raskar MIT Media Lab

Strategy 1 Xd

bull Extend it to next (or some other) dimension

Ramesh Raskar MIT Media Lab

X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm

Ramesh Raskar MIT Media Lab

Xd

X++

X X+Y

X

X

neXt

Ramesh Raskar httpraskarinfo

Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010

Research bull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Is project worthwhile Heilmeiers Questions

bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon

bull Related workndash How is it done today and what are the limits of current practice

bull Contributionndash Whats new in your approach and why do you think it will be successful

bull Motivationndash Who caresndash If youre successful what difference will it make

bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take

bull Evaluationndash What are the midterm and final exams to check for success

bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)

httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism

Great Research Strive for Five

1 Before Five teamsBe first often let others do details

2 Beyond Five yearsWhat no one is thinking about

3 Within Five layers of lsquoHumanrsquo ImpactRelevance

4 Beyond Five minutes of descriptionDeep iterative participatory

5 Fusing Five+ ExpertiseMulti-disciplinary proactive

Ramesh Raskar httpraskarinfo

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Fernald Science [Sept 2006]

Shadow Refractive

Reflective

Tools for

Visual Computin

g

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Traditional Photography

Lens

Detector

Pixels

Image

Mimics Human Eye for a Single Snapshot

Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree

Nayar

Computational Camera + Photography

Optics Sensors and ComputationsGeneralized

Sensor

Generalized OpticsComputations

Picture

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures

Computational Photography

Novel Illumination

Computational Cameras

Scene 8D Ray Modulator

Display

GeneralizedSensor

Generalized OpticsProcessing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

Computational Photography [Raskar and Tumblin]

Resources

ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011

httpwikipediaorgcomputational_photography

httpraskarinfophoto

captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience

SynthesisAnalysis

Low Level Mid Level

HighLevel

Hyper realism

Raw

Angle spectrum

aware

Non-visual Data GPS

Metadata

Priors

Cap

ture

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Looking Around Corners

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

fgbg

DirectGlobal

Computational Photography

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Ramesh Raskar MIT Media Lab

X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm

Ramesh Raskar MIT Media Lab

Xd

X++

X X+Y

X

X

neXt

Ramesh Raskar httpraskarinfo

Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010

Research bull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Is project worthwhile Heilmeiers Questions

bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon

bull Related workndash How is it done today and what are the limits of current practice

bull Contributionndash Whats new in your approach and why do you think it will be successful

bull Motivationndash Who caresndash If youre successful what difference will it make

bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take

bull Evaluationndash What are the midterm and final exams to check for success

bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)

httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism

Great Research Strive for Five

1 Before Five teamsBe first often let others do details

2 Beyond Five yearsWhat no one is thinking about

3 Within Five layers of lsquoHumanrsquo ImpactRelevance

4 Beyond Five minutes of descriptionDeep iterative participatory

5 Fusing Five+ ExpertiseMulti-disciplinary proactive

Ramesh Raskar httpraskarinfo

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Fernald Science [Sept 2006]

Shadow Refractive

Reflective

Tools for

Visual Computin

g

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Traditional Photography

Lens

Detector

Pixels

Image

Mimics Human Eye for a Single Snapshot

Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree

Nayar

Computational Camera + Photography

Optics Sensors and ComputationsGeneralized

Sensor

Generalized OpticsComputations

Picture

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures

Computational Photography

Novel Illumination

Computational Cameras

Scene 8D Ray Modulator

Display

GeneralizedSensor

Generalized OpticsProcessing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

Computational Photography [Raskar and Tumblin]

Resources

ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011

httpwikipediaorgcomputational_photography

httpraskarinfophoto

captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience

SynthesisAnalysis

Low Level Mid Level

HighLevel

Hyper realism

Raw

Angle spectrum

aware

Non-visual Data GPS

Metadata

Priors

Cap

ture

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Looking Around Corners

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

fgbg

DirectGlobal

Computational Photography

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Ramesh Raskar MIT Media Lab

Xd

X++

X X+Y

X

X

neXt

Ramesh Raskar httpraskarinfo

Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010

Research bull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Is project worthwhile Heilmeiers Questions

bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon

bull Related workndash How is it done today and what are the limits of current practice

bull Contributionndash Whats new in your approach and why do you think it will be successful

bull Motivationndash Who caresndash If youre successful what difference will it make

bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take

bull Evaluationndash What are the midterm and final exams to check for success

bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)

httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism

Great Research Strive for Five

1 Before Five teamsBe first often let others do details

2 Beyond Five yearsWhat no one is thinking about

3 Within Five layers of lsquoHumanrsquo ImpactRelevance

4 Beyond Five minutes of descriptionDeep iterative participatory

5 Fusing Five+ ExpertiseMulti-disciplinary proactive

Ramesh Raskar httpraskarinfo

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Fernald Science [Sept 2006]

Shadow Refractive

Reflective

Tools for

Visual Computin

g

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Traditional Photography

Lens

Detector

Pixels

Image

Mimics Human Eye for a Single Snapshot

Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree

Nayar

Computational Camera + Photography

Optics Sensors and ComputationsGeneralized

Sensor

Generalized OpticsComputations

Picture

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures

Computational Photography

Novel Illumination

Computational Cameras

Scene 8D Ray Modulator

Display

GeneralizedSensor

Generalized OpticsProcessing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

Computational Photography [Raskar and Tumblin]

Resources

ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011

httpwikipediaorgcomputational_photography

httpraskarinfophoto

captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience

SynthesisAnalysis

Low Level Mid Level

HighLevel

Hyper realism

Raw

Angle spectrum

aware

Non-visual Data GPS

Metadata

Priors

Cap

ture

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Looking Around Corners

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

fgbg

DirectGlobal

Computational Photography

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Research bull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Is project worthwhile Heilmeiers Questions

bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon

bull Related workndash How is it done today and what are the limits of current practice

bull Contributionndash Whats new in your approach and why do you think it will be successful

bull Motivationndash Who caresndash If youre successful what difference will it make

bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take

bull Evaluationndash What are the midterm and final exams to check for success

bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)

httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism

Great Research Strive for Five

1 Before Five teamsBe first often let others do details

2 Beyond Five yearsWhat no one is thinking about

3 Within Five layers of lsquoHumanrsquo ImpactRelevance

4 Beyond Five minutes of descriptionDeep iterative participatory

5 Fusing Five+ ExpertiseMulti-disciplinary proactive

Ramesh Raskar httpraskarinfo

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Fernald Science [Sept 2006]

Shadow Refractive

Reflective

Tools for

Visual Computin

g

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Traditional Photography

Lens

Detector

Pixels

Image

Mimics Human Eye for a Single Snapshot

Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree

Nayar

Computational Camera + Photography

Optics Sensors and ComputationsGeneralized

Sensor

Generalized OpticsComputations

Picture

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures

Computational Photography

Novel Illumination

Computational Cameras

Scene 8D Ray Modulator

Display

GeneralizedSensor

Generalized OpticsProcessing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

Computational Photography [Raskar and Tumblin]

Resources

ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011

httpwikipediaorgcomputational_photography

httpraskarinfophoto

captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience

SynthesisAnalysis

Low Level Mid Level

HighLevel

Hyper realism

Raw

Angle spectrum

aware

Non-visual Data GPS

Metadata

Priors

Cap

ture

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Looking Around Corners

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

fgbg

DirectGlobal

Computational Photography

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Is project worthwhile Heilmeiers Questions

bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon

bull Related workndash How is it done today and what are the limits of current practice

bull Contributionndash Whats new in your approach and why do you think it will be successful

bull Motivationndash Who caresndash If youre successful what difference will it make

bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take

bull Evaluationndash What are the midterm and final exams to check for success

bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)

httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism

Great Research Strive for Five

1 Before Five teamsBe first often let others do details

2 Beyond Five yearsWhat no one is thinking about

3 Within Five layers of lsquoHumanrsquo ImpactRelevance

4 Beyond Five minutes of descriptionDeep iterative participatory

5 Fusing Five+ ExpertiseMulti-disciplinary proactive

Ramesh Raskar httpraskarinfo

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Fernald Science [Sept 2006]

Shadow Refractive

Reflective

Tools for

Visual Computin

g

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Traditional Photography

Lens

Detector

Pixels

Image

Mimics Human Eye for a Single Snapshot

Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree

Nayar

Computational Camera + Photography

Optics Sensors and ComputationsGeneralized

Sensor

Generalized OpticsComputations

Picture

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures

Computational Photography

Novel Illumination

Computational Cameras

Scene 8D Ray Modulator

Display

GeneralizedSensor

Generalized OpticsProcessing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

Computational Photography [Raskar and Tumblin]

Resources

ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011

httpwikipediaorgcomputational_photography

httpraskarinfophoto

captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience

SynthesisAnalysis

Low Level Mid Level

HighLevel

Hyper realism

Raw

Angle spectrum

aware

Non-visual Data GPS

Metadata

Priors

Cap

ture

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Looking Around Corners

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

fgbg

DirectGlobal

Computational Photography

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Great Research Strive for Five

1 Before Five teamsBe first often let others do details

2 Beyond Five yearsWhat no one is thinking about

3 Within Five layers of lsquoHumanrsquo ImpactRelevance

4 Beyond Five minutes of descriptionDeep iterative participatory

5 Fusing Five+ ExpertiseMulti-disciplinary proactive

Ramesh Raskar httpraskarinfo

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Fernald Science [Sept 2006]

Shadow Refractive

Reflective

Tools for

Visual Computin

g

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Traditional Photography

Lens

Detector

Pixels

Image

Mimics Human Eye for a Single Snapshot

Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree

Nayar

Computational Camera + Photography

Optics Sensors and ComputationsGeneralized

Sensor

Generalized OpticsComputations

Picture

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures

Computational Photography

Novel Illumination

Computational Cameras

Scene 8D Ray Modulator

Display

GeneralizedSensor

Generalized OpticsProcessing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

Computational Photography [Raskar and Tumblin]

Resources

ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011

httpwikipediaorgcomputational_photography

httpraskarinfophoto

captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience

SynthesisAnalysis

Low Level Mid Level

HighLevel

Hyper realism

Raw

Angle spectrum

aware

Non-visual Data GPS

Metadata

Priors

Cap

ture

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Looking Around Corners

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

fgbg

DirectGlobal

Computational Photography

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Fernald Science [Sept 2006]

Shadow Refractive

Reflective

Tools for

Visual Computin

g

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Traditional Photography

Lens

Detector

Pixels

Image

Mimics Human Eye for a Single Snapshot

Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree

Nayar

Computational Camera + Photography

Optics Sensors and ComputationsGeneralized

Sensor

Generalized OpticsComputations

Picture

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures

Computational Photography

Novel Illumination

Computational Cameras

Scene 8D Ray Modulator

Display

GeneralizedSensor

Generalized OpticsProcessing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

Computational Photography [Raskar and Tumblin]

Resources

ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011

httpwikipediaorgcomputational_photography

httpraskarinfophoto

captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience

SynthesisAnalysis

Low Level Mid Level

HighLevel

Hyper realism

Raw

Angle spectrum

aware

Non-visual Data GPS

Metadata

Priors

Cap

ture

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Looking Around Corners

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

fgbg

DirectGlobal

Computational Photography

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Fernald Science [Sept 2006]

Shadow Refractive

Reflective

Tools for

Visual Computin

g

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Traditional Photography

Lens

Detector

Pixels

Image

Mimics Human Eye for a Single Snapshot

Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree

Nayar

Computational Camera + Photography

Optics Sensors and ComputationsGeneralized

Sensor

Generalized OpticsComputations

Picture

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures

Computational Photography

Novel Illumination

Computational Cameras

Scene 8D Ray Modulator

Display

GeneralizedSensor

Generalized OpticsProcessing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

Computational Photography [Raskar and Tumblin]

Resources

ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011

httpwikipediaorgcomputational_photography

httpraskarinfophoto

captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience

SynthesisAnalysis

Low Level Mid Level

HighLevel

Hyper realism

Raw

Angle spectrum

aware

Non-visual Data GPS

Metadata

Priors

Cap

ture

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Looking Around Corners

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

fgbg

DirectGlobal

Computational Photography

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Traditional Photography

Lens

Detector

Pixels

Image

Mimics Human Eye for a Single Snapshot

Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree

Nayar

Computational Camera + Photography

Optics Sensors and ComputationsGeneralized

Sensor

Generalized OpticsComputations

Picture

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures

Computational Photography

Novel Illumination

Computational Cameras

Scene 8D Ray Modulator

Display

GeneralizedSensor

Generalized OpticsProcessing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

Computational Photography [Raskar and Tumblin]

Resources

ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011

httpwikipediaorgcomputational_photography

httpraskarinfophoto

captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience

SynthesisAnalysis

Low Level Mid Level

HighLevel

Hyper realism

Raw

Angle spectrum

aware

Non-visual Data GPS

Metadata

Priors

Cap

ture

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Looking Around Corners

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

fgbg

DirectGlobal

Computational Photography

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Traditional Photography

Lens

Detector

Pixels

Image

Mimics Human Eye for a Single Snapshot

Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree

Nayar

Computational Camera + Photography

Optics Sensors and ComputationsGeneralized

Sensor

Generalized OpticsComputations

Picture

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures

Computational Photography

Novel Illumination

Computational Cameras

Scene 8D Ray Modulator

Display

GeneralizedSensor

Generalized OpticsProcessing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

Computational Photography [Raskar and Tumblin]

Resources

ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011

httpwikipediaorgcomputational_photography

httpraskarinfophoto

captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience

SynthesisAnalysis

Low Level Mid Level

HighLevel

Hyper realism

Raw

Angle spectrum

aware

Non-visual Data GPS

Metadata

Priors

Cap

ture

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Looking Around Corners

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

fgbg

DirectGlobal

Computational Photography

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Computational Camera + Photography

Optics Sensors and ComputationsGeneralized

Sensor

Generalized OpticsComputations

Picture

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures

Computational Photography

Novel Illumination

Computational Cameras

Scene 8D Ray Modulator

Display

GeneralizedSensor

Generalized OpticsProcessing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

Computational Photography [Raskar and Tumblin]

Resources

ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011

httpwikipediaorgcomputational_photography

httpraskarinfophoto

captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience

SynthesisAnalysis

Low Level Mid Level

HighLevel

Hyper realism

Raw

Angle spectrum

aware

Non-visual Data GPS

Metadata

Priors

Cap

ture

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Looking Around Corners

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

fgbg

DirectGlobal

Computational Photography

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Computational Photography

Novel Illumination

Computational Cameras

Scene 8D Ray Modulator

Display

GeneralizedSensor

Generalized OpticsProcessing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

Computational Photography [Raskar and Tumblin]

Resources

ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011

httpwikipediaorgcomputational_photography

httpraskarinfophoto

captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience

SynthesisAnalysis

Low Level Mid Level

HighLevel

Hyper realism

Raw

Angle spectrum

aware

Non-visual Data GPS

Metadata

Priors

Cap

ture

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Looking Around Corners

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

fgbg

DirectGlobal

Computational Photography

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Computational Photography [Raskar and Tumblin]

Resources

ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011

httpwikipediaorgcomputational_photography

httpraskarinfophoto

captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience

SynthesisAnalysis

Low Level Mid Level

HighLevel

Hyper realism

Raw

Angle spectrum

aware

Non-visual Data GPS

Metadata

Priors

Cap

ture

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Looking Around Corners

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

fgbg

DirectGlobal

Computational Photography

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

SynthesisAnalysis

Low Level Mid Level

HighLevel

Hyper realism

Raw

Angle spectrum

aware

Non-visual Data GPS

Metadata

Priors

Cap

ture

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Looking Around Corners

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

fgbg

DirectGlobal

Computational Photography

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Computational Photography

Wish List Open Research

Problems

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Oacute 2007 Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Motion Blur in Low Light

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Traditional

Deblurred Image

Blurred Photo

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Flutter Shutter Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo

Blurred PhotoPSF == Broadband Function

Fourier Transform

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Motion Blur in Low Light

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Fast periodic phenomena

4000 fps hi-speed camera

Vocal folds flapping at 404 Hz

500 fps hi-speed camera

Bottling line

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Compressive Sensing

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Periodic signals

Periodic signal with period P and band-limited to fMax = 500 Hz

Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz

4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP

t

Periodic signal x(t) with period P

P = 16ms

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

High speed camera

Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth

0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP

Nyquist Sampling of x(t)

P = 16ms Ts = 1(2 fMax)

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

P t

Strobing animation credit Wikipedia

Traditional Strobing

Use low frame-rate camera and generate beat frequencies

Low exposure to avoid blurring Low light throughputPeriod known apriori

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Random Projections Per Frame of Camera using Coded Strobing Photography

In every exposure duration observe different linear combinations of the periodic signal

Advantage of the design bull Exposure coding independent of the frequency

bull On an average light throughput is 50

tP

Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Observation Model

x at 2000fpsy at 25fps

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Signal Model

x at 2000fpsy at 25fps

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Signal amp Observation Model

A is M x N MltltN

x at 2000fpsy at 25fps

N M = 2000 25 = 80

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Recovery Sparsity

Very few non-zero elements

Structured Sparse Coefficients

y = A sMixing matrix

Basis Pursuit De-noising

Asytss min1

Observed values

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Simulation on hi-speed toothbrush

25fps normal camera 25fps coded strobing camera

Reconstructed frames 2000fps hi-speed camera

~100X speedup

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Rotating mill tool

Normal Video 25fpsReconstructed Video at 2000fps

rotating at 150Hz

Coded Strobing Video 25fps

Mill tool rotating at 50Hz

rotating at 100Hz rotating at 200Hz

Blur increases as rotational velocity increases

increasing blur

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Compressive Sensing for Images A good idea

NMNMxy where

Single Pixel Camera

compressive image measurement matrix

image

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Is Randomized Projection-based Captureapt for Natural Images

[Pandharkar Veeraraghavan Raskar 2009]

Randomized Projections

Prog

ress

ive

Pro

jecti

ons

Compression Ratio

Periodic Signals

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Ramesh Raskar Computational Illumination

Compact Programmable

Lights

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 1

Ultimate Post-capture Control

bullDigital Refocus and Motion blur

bullEmulate studio light from compact flash

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 2

Freedom from Form

bull Size Weight Power UIbull Flat camera

Bidirectional screen (BiDi)

bull Shallow DoF from tiny lens

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 3

Understand the World

bullIdentifyrecognize Materialsbull3D Awareness

bullInteract with information

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 4

Sharing Visual Experience

bullLifeLog Auto-summarybullPrivacy in public and

authentication bullHyper-real Photo Frames

bullPrint lsquomaterialrsquo

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Wish 5

Capturing Essence

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

What are the problems with lsquorealrsquo photo in conveying information

Why do we hire artists to draw what can be photographed

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Depth Discontinuities

Internal and externalShape boundaries Occluding contour Silhouettes

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Our MethodCanny

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Raskar Camera Culture MIT Media Lab

Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and

portable cameras change the social culture bull How will online photo collections

transform visual social computingbull How will movie makingnew reporting change

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Photos of tomorrow computed not recorded

httpscalarmotionwordpresscom20090315propeller-image-aliasing

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo

bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur

bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture

bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode

bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes

bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera

Computational Photography Wish ListSensor

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Every Photon has a Story

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

What isaround the corner

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Can you look around the corner

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Femto-Photography (Transient Imaging)FemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar

ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)

With M Bawendi

MIT Chemistry

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Inverting Light Transport

Multiple Scattering DirectGlobal

Dual Photography

[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos Steger 2005]

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Multi-Dimensional Light Transport

5-D Transport

Gigapan

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Collision avoidance robot navigation hellip

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Steady State 4D

Impulse Response 5D

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Scene with hidden elements

Raw Time profiles

treg

Signal Proc

Novel light transport models and inference

algorithms

Photo geometry reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Femto-PhotographyTime Resolved Multi-path Imaging

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Raskar Camera Culture MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Motion beyond line of sight

Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

hellip bronchoscopies hellip

Participating Media

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

[Nayar Krishnan Grossberg Raskar 2006]

Photo

First Bounce

Later Bounces

Direct Global

+

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

>

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Each frame = ~2ps = 06 mm of Light Travel

>

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Ripples of Waves

>

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

>

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

>

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

Inverse Problems

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Two Layer Displays

barrier

sensordisplay

lenslet

sensordisplay

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Parallax barrier

LCD display

Limitations of 3D Display

Lanman Hirsch Kim Raskar Siggraph Asia 2010

Front

Back

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

][][][ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[ik]

L[ik]

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

f[i]

g[k]

L[ik]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Implementation

Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

f[i]

g[k]

L[ik]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

0for 21 min arg 2

GF GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Lanman Hirsch Kim Raskar Siggraph Asia 2010

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

Optimization Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Optimization Iteration 10

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Optimization Iteration 20

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Optimization Iteration 30

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Optimization Iteration 40

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Optimization Iteration 50

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Optimization Iteration 60

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Optimization Iteration 70

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Optimization Iteration 80

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Optimization Iteration 90

rear mask f1[ij] front mask g1[kl]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Emitted 4D Light Field

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Conclusion

bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1

bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0for 21 min arg 2

GF GFFGL

W

F

G

L~ =

Content-Adaptive Parallax Barriers

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Parallax Barrier Np=103 pix

Hologram NH=105 pix

xH =100 patches

θp=10 pix

w

θH =1000 pix

xp=100 slits

ϕPpropwd ϕHpropλtH

Fourier Patch

Horstmeyer Oh Cuypers Barbastathis Raskar 2009

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Augmented Light Field

118

Wigner Distribution Function

Traditional Light Field

WDF

Traditional Light Field

Augmented LF

Interference amp DiffractionInteraction w optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Oh Raskar Barbastathis 2009 Augmented Light Field

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Light Fields

Referenceplane

position angle

Goal Representing propagation interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF lt WDF

Lacks phase propertiesIgnores diffraction interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherentincoherent

Radiance = PositiveNegative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

121

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Lightfield vs Hologram Displays

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

L(xθ) W(xu) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)

p Wm

Rays No Bending 1 Fresnel HG Patch

θ u

Zooming into the Light Field

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

s1

m2 m2s1

s1

(a) Parallax Barrier

(b) Hologram (c) Hybrid

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

x

u

-Transform

ltt(x+xʹ2)t(x-xʹ2)gt

Interferencexʹ

x

(a) Two Slits Coherent

t(x+xʹ2)t(x-xʹ2)W(xu)

2x

1x

Rank-1

t(x1)t(x2)

Transform-1

u R45 D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(xθ)L2(xθ)

L3(xθ)

d

z1

hH

r

z2

L1(xθ) L2(xθ) L3(xθ)

s1m2

(a)

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Three Questionsbull What are the benefits of higher dimensional imaging

bull Why is the algebraic rank of a Light Field not full

bull What makes looking around the corner possible

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

How to do Research in Imagingbull httpraskarinfo

ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar

bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are

forgotten in the long runndash Highly recommended Hamming talk at Bell Labs

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

Take home pointsbull Co-design of hwsw

ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio

communicationsbull Photons not just Pixelsbull Change the rules of the game

ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132

MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar

bull How to do Research in Imagingbull Inverse Problems Reconstruction

Rank and Sparsitybull Co-design of Optics and Computation

ndash Photons not just pixelsndash Mid-level cues

bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events

bull Limits of CS for general imagingbull Computational Light Transport

ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms

bull Apply for internshipspost-doc

Inverse ProblemsneXt

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Simple Exercise
• Strategy 1 Xd
• X =
• Slide 8
• Research
• Is project worthwhile Heilmeiers Questions
• Great Research Strive for Five
• Slide 12
• Slide 13
• Slide 14
• Traditional Photography
• Computational Camera + Photography Optics Sensors and Compu
• Computational Photography
• Computational Photography [Raskar and Tumblin]
• Slide 19
• Slide 20
• Take home points
• Slide 22
• Slide 23
• Digital Refocusing using Light Field Camera
• Slide 25
• Slide 26
• Fluttered Shutter Camera
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Is Randomized Projection-based Capture apt for Natural Images
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Convert single 2D photo into 3D
• Slide 50
• Slide 51
• Slide 52
• Slide 53
• Slide 54
• Slide 55
• Depth Edges with MultiFlash
• Slide 57
• Slide 58
• Slide 59
• Slide 60
• Slide 61
• Slide 63
• Questions
• Slide 66
• Slide 67
• Take home points (2)
• Slide 69
• Slide 70
• Slide 71
• Slide 72
• Slide 73
• Femto-Photography (Transient Imaging)
• Inverting Light Transport
• Slide 76
• Slide 77
• Slide 78
• Slide 79
• Femto-Photography Time Resolved Multi-path Imaging
• Slide 81
• Slide 82
• Slide 83
• Slide 84
• Slide 85
• Slide 86
• Slide 87
• Slide 88
• Slide 89
• Slide 90
• Slide 91
• Slide 92
• Slide 93
• Slide 94
• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 99
• Slide 100
• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107
• Slide 108
• Slide 109
• Slide 110
• Slide 111
• Slide 112
• Slide 113
• Slide 114
• Slide 115
• Slide 116
• Slide 117
• Augmented Light Field
• Light Fields
• Augmented Lightfield for Wave Optics Effects
• Slide 121
• Slide 122
• Slide 123
• Slide 124
• Slide 125
• Slide 126
• Slide 127
• Slide 128
• Three Questions
• How to do Research in Imaging
• Take home points (3)
• Slide 132