Introduction toComputational Photography

Computational Photography

Digital Camera

What isComputational Photography?

Second breakthrough by IT First : electronic image sensor (digital camera)

Digital representation of “image formed by lens” Second : Re-definition of whole camera (optics, usage)

Image is reconstructed by computation

Imageprocessing

Image

Image sensor

Optics

Whole part of camera is affected by computational photography

Film camera

Digital Camera

Light field(Light space)

€

(X,Y,Z)

€

(θ,φ)

What is camera?

Camera is a machine to record the distribution of the light in a scene

How to represent the distribution of the light in the scene? 3-D coordinate of the point where the light passing through : X, Y, Z Direction of the light : θ ， Φ Wavelength of the light (color of the light) : λ Time : t

The 7 parameters function P which represent the distribution of the light is called “Plenoptic function”

€

I = P(X,Y,Z,θ,φ,λ , t)Light source

Object

Light field (light ray) Optics (lens) Image sensor (pixel)

€

(X,Y,Z)

€

(θ,φ)

Integration of camera

Camera integrates the light for all 7 parameters Position （ range of X, Y, Z : aperture size should not be zero) Direction （ range of θ ， Φ : pixel size is not zero) Wavelength （ range of λ : No single wavelength filter) Exposure time ( range of t : shutter speed should not be too fast)

Multiple samples - θ,Φ:number of pixel, λ ： RGB ， t : burst shot So, what is multiple sampling for X, Y, Z?

€

I = P(X,Y,Z,θ,φ,λ , t)

Camera array

Measuring the distribution of the light at multiple position

ProFUSION25 (ViewPlus, Inc.)

The StanfordMulti-Camera Array(Marc Levoy @Stanford University)

Use of camera array

Free-viewpoint image

Defocus generation by synthetic aperture

3-D video (Matsuyama lab, Kyoto Univ.) Synthetic aperture(Vaish@Stanford)

Defocus control by Uncalibrated Synthetic Aperture

Natsumi Kusumoto, Shinsaku Hiura and Kosuke Sato, Uncalibrated Synthetic Aperture for Defocus Control, CVPR2009 (Jun. 2009)

€

(X,Y,Z)

€

(θ,φ)

Reviewing “integration”

Some part of information is lost by integration Sine wave which period is just as same as the integration duration

is lost

Blur of object within an exposure time Defocus by misfocus

€

I = P(X,Y,Z,θ,φ,λ , t)

€

Asin(nt)dt0

2π

∫ = 0

× = 0

Coded Exposure

Coded exposure : exposure is coded in time axis

Flutter Shutter Camera (Raskar@MERL)

This

TraditionalCoded

Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Slide by R. Raskar

Coded Exposure

Temporal 1-D broadband code: Motion Deblurring

Coded Aperture

Spatial 2-D broadband mask: Focus Deblurring

Slide by R. Raskar

Captured Blurred Photo

Slide by R. Raskar

Refocused on Person

Slide by R. Raskar

Coded Aperture

Levin@MIT(2007) Depth estimation by single

image (manual operation is necessary)

Coded Aperture

Levin@MIT(2007)

Coded Aperture

Levin@MIT(2007)

Multi-focus camera withCoded Aperture

Stabilizing the depth estimation and deblur by coded aperture

Simultaneous capture of 3 images with different focused distance

Hiura et al, CVPR(1998),SSII(1999)

Multi-Focus Range Sensorusing Coded Aperture

Invariant integration Defocus : changed according to the distance

Blur : changed according to the speed of the object

Reconstruction is not easy because the estimation of the speed or distance is necessaryIs it possible to make defocus or blur invariant to the distance or speed?

Invariant integration Defocus

Special optics : Wavefront Coding

Motion of the image sensor while exposure

Blur Reciprocal motion of the camera

CDM Optics, Inc.

Motion of the image sensorfor invariant defocus

H. Nagahara, S. Kuthirummal, C. Zhou, and S.K. Nayar, Flexible Depth of Field Photography, ECCV2008

H. Nagahara, S. Kuthirummal, C. Zhou, and S.K. Nayar, Flexible Depth of Field Photography, ECCV2008

Motion of the image sensorfor invariant defocus

Deblur by reciprocal motion of the camera

A. Levin, P. Sand, T. S. Cho, F. Durand, W. T. Freeman. Motion-Invariant Photography. SIGGRAPH2008.

Input image Deblurred image

A. Levin, P. Sand, T. S. Cho, F. Durand, W. T. Freeman. Motion-Invariant Photography. SIGGRAPH2008.

Equipment Conceptual figure forLight sources with

different speed

Deblur by reciprocal motion of the camera

More..

Resources on www Wikipedia : computational photography http://computationalphotography.org/ http://www1.cs.columbia.edu/CAVE/projects/what_is/ http://projects.csail.mit.edu/photo/

Conferences International Conference on Computational Photography SIGGRAPH, CVPR, .. Session about computational photography