Click here to load reader
Upload
cameraculture-mit-media-lab
View
1.294
Download
1
Embed Size (px)
Citation preview
Marc Levoy
The CityBlock Project
Precursor to Google Streetview Maps
Image Fusion & ReconstructionImage Fusion & Reconstruction• Single photo:Single photo: forces narrow tradeoffs: forces narrow tradeoffs:
– Focus, Exposure, aperture, time, sensitivity, noise,Focus, Exposure, aperture, time, sensitivity, noise,
– Usual result: Incomplete visual appearance.Usual result: Incomplete visual appearance.
Multiple photosMultiple photos, assorted settings , assorted settings for Optics, Sensor, Lighting, Processingfor Optics, Sensor, Lighting, Processing
• Fusion:Fusion: ‘Merge the best parts’‘Merge the best parts’
• Reconstruction:Reconstruction:Detect photo changes; Detect photo changes; compute scene invariantscompute scene invariants
High Dynamic Range ImagingHigh Dynamic Range Imaging
• Cameras have limited dynamic range
Small Exposure image, dark inside
1/500 sec
Large exposure image, saturated outside
¼ sec
Images from Raanan Fattal
High Dynamic Range ImagingHigh Dynamic Range Imaging
• Combine images at different exposures• Exposure Bracketing• [Mann and Picard 95, Debevec et al 96]
Images from Raanan Fattal
How could we put all thisinformation into oneimage ?
Tone Map 20 bit image for 8 bit DisplayTone Map 20 bit image for 8 bit Display
input smoothed(structure, large scale)
residual(texture, small scale)
Gaussian Convolution
BLUR HALOS
Naïve Approach: Gaussian Blur
Impact of Blur and Halos
• If the decomposition introduces blur and halos, the final result is corrupted.
Sample manipulation:increasing texture
(residual 3)
input smoothed(structure, large scale)
residual(texture, small scale)
edge-preserving: Bilateral Filter
Bilateral Filter: no Blur, no Halos
input
increasing texturewith Gaussian convolution
H A L O S
increasing texturewith bilateral filter
N O H A L O S
Bilateral Filter on 1D Signal
BF
p
Our Strategy
Reformulate the bilateral filter– More complex space:
Homogeneous intensity Higher-dimensional space
– Simpler expression: mainly a convolution Leads to a fast algorithm
weightsappliedto pixels
Attenuate High GradientsAttenuate High Gradients
I(x)1
105
1
Intensity
I(x)1
105
Intensity
Maintain local detail at the cost of global range
Fattal et al Siggraph 2002
Attenuate High GradientsAttenuate High Gradients
I(x)1
105
G(x)1
105
Intensity Gradient
I(x)1
105
Intensity
Maintain local detail at the cost of global range
Fattal et al Siggraph 2002
Attenuate High GradientsAttenuate High Gradients
I(x)1
105
G(x)1
105
Intensity Gradient
I(x)1
105
Intensity
Keep low gradients
Fattal et al Siggraph 2002
Gradient Compression in 1DGradient Compression in 1D
Gradient Domain CompressionGradient Domain Compression
HDR Image L Log L
Gradient Attenuation Function G
Multiply 2D Integration
Gradients Lx,Ly
Grad X
Grad Y
New Grad X
New Grad Y
2D Integration
Intensity Gradient ManipulationIntensity Gradient Manipulation
Gradient Processing
A Common Pipeline
This Section
Next Section
Grad X
Grad Y
New Grad X
New Grad Y
2D Integration
Gradient Processing
Local Illumination ChangeLocal Illumination Change
Original gradient field:
Original Image: f
*f
Modified gradient field: v
Perez et al. Poisson Image editing, SIGGRAPH 2003
Ambient FlashSelf-Reflections and Flash HotspotSelf-Reflections and Flash Hotspot
Hands
Face
Tripod
ResultAmbient
Flash
Reflection LayerReflection Layer
Hands
Face
Tripod
Intensity Gradient Vector Intensity Gradient Vector ProjectionProjection[Agrawal, Raskar, Nayar, Li SIGGRAPH 2005][Agrawal, Raskar, Nayar, Li SIGGRAPH 2005]
Intensity Gradient Vectors in Flash and Ambient ImagesIntensity Gradient Vectors in Flash and Ambient Images
Same gradient vector direction Flash Gradient Vector
Ambient Gradient Vector
Ambient Flash
No reflections
Reflection Ambient Gradient Vector
Different gradient vector direction
With reflections
Ambient Flash
Flash Gradient Vector
Residual Gradient Vector
Intensity Gradient Vector Projection
Result Gradient Vector
Result Residual
Reflection Ambient Gradient Vector
Flash Gradient Vector
Ambient Flash
FlashProjection = Result
Residual = Reflection Layer
Co-located Artifacts
Ambient
Recovering foreground layerRecovering foreground layer– Find tensor based on background image– Transform gradient field of foreground image
Foreground maskImage Difference
Dark Bldgs
Reflections on bldgs
Unknown shapes
‘Well-lit’ Bldgs
Reflections in bldgs windows
Tree, Street shapes
Background is captured from day-time scene using the same fixed camera
Night Image
Day Image
Context Enhanced Image
Mask is automatically computed from scene contrast
But, Simple Pixel Blending Creates Ugly Artifacts
Pixel Blending
Pixel Blending
Our Method:Integration of
blended Gradients
Nighttime imageNighttime image
Daytime imageDaytime image Gradient fieldGradient field
Importance Importance image Wimage W
Fina
l res
ult
Fina
l res
ult
Gradient fieldGradient field
Mixed gradient fieldMixed gradient field
GG11 GG11
GG22 GG22
xx YY
xx YY
II11
I2
GG GGxx YY
Reconstruction from Gradient FieldReconstruction from Gradient Field
• Problem: minimize errorI’ – G|• Estimate I’ so that
G = I’
• Poisson equation
I’ = div G
• Full multigridsolver
I’I’
GGXX
GGYY
Rene Magritte, ‘Empire of the Light’
Surrealism
actual photomontageset of originals perceived
Source images Brush strokes Computed labeling
Composite
Brush strokes Computed labeling
• No Flash:No Flash: Candle warmth, but high noise Candle warmth, but high noise• Flash:Flash: low noise, but no candle warmth low noise, but no candle warmth
Photography: Full of Tradeoffs...Photography: Full of Tradeoffs...
No-flash Flash
Image A: Warm, shadows, but too Noisy(too dim for a good quick photo)
No-flash
Image B: Cold, Shadow-free, Clean(flash: simple light, ALMOST no shadows)
MERGE BEST OF BOTH: apply‘Cross Bilateral’ or ‘Joint Bilateral’
(it really is much better!)
Image Fusion & ReconstructionImage Fusion & Reconstruction• Single photo:Single photo: forces narrow tradeoffs: forces narrow tradeoffs:
– Focus, Exposure, aperture, time, sensitivity, noise,Focus, Exposure, aperture, time, sensitivity, noise,
– Usual result: Incomplete visual appearance.Usual result: Incomplete visual appearance.
Multiple photosMultiple photos, assorted settings , assorted settings for Optics, Sensor, Lighting, Processingfor Optics, Sensor, Lighting, Processing
• Fusion:Fusion: ‘Merge the best parts’‘Merge the best parts’
• Reconstruction:Reconstruction:Detect photo changes; Detect photo changes; compute scene invariantscompute scene invariants
The Media Lab Camera Culture
Epsilon Photography
Capture multiple photos, each with slightly different camera parameters.
• Exposure settings• Spectrum/color settings• Focus settings• Camera position• Scene illumination
FUSION: Best-Focus DistanceFUSION: Best-Focus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
NEARNEAR
FUSION: Best-Focus DistanceFUSION: Best-Focus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
FARFAR
FUSION: Best-Focus DistanceFUSION: Best-Focus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
FUSION: Best-Focus DistanceFUSION: Best-Focus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
FUSION: Best-Focus DistanceFUSION: Best-Focus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
FUSION: Best-Focus DistanceFUSION: Best-Focus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
FUSION: Best-Focus DistanceFUSION: Best-Focus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
FUSION: Best-Focus DistanceFUSION: Best-Focus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
FUSION: Best-Focus DistanceFUSION: Best-Focus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
FUSION: Best-Focus DistanceFUSION: Best-Focus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
FUSION: Best-Focus DistanceFUSION: Best-Focus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
FUSION: Best-Focus DistanceFUSION: Best-Focus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
FUSION: Best-Focus DistanceFUSION: Best-Focus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
FUSION: Best-Focus DistanceFUSION: Best-Focus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
FUSION: Best-Focus DistanceFUSION: Best-Focus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
Source images
‘Graph Cuts’ Solution
FUSION
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
What else can we extend? What else can we extend? Film-Like Camera Parameters: Film-Like Camera Parameters: • Field of View: image stitching for panoramasField of View: image stitching for panoramas• Dynamic Range: Dynamic Range: Radiance MapsRadiance Maps• Frame Rate: Interleaved VideoFrame Rate: Interleaved Video• Resolution: ‘Super-resolution’ methodsResolution: ‘Super-resolution’ methods
Visual Appearance & Content:Visual Appearance & Content:• Tone Map:Tone Map: Detail in every shadow and highlight Detail in every shadow and highlight• Color2grey:Color2grey: Keep Keep allall color changes in grayscale color changes in grayscale • Temporal Continuity: Space-time fusionTemporal Continuity: Space-time fusion• Viewpoint Constraints: Viewpoint Constraints:
Multiple COP images Multiple COP images and more…and more…
The Media Lab Camera Culture
Epsilon Photography
Capture multiple photos, each with slightly different camera parameters.
• Exposure settings• Spectrum/color settings• Focus settings• Camera position• Scene illumination
The Media Lab Camera Culture
Project Ideas
Marc Levoy
The CityBlock Project
Precursor to Google Streetview Maps
What is ‘interesting’ here? Social voting in the real world = ‘popular’
Vein Viewer Vein Viewer (Luminetx)(Luminetx)
Near-IR camera locates subcutaneous veins and project Near-IR camera locates subcutaneous veins and project their location onto the surface of the skin.their location onto the surface of the skin.
Coaxial IR camera Coaxial IR camera + Projector+ Projector
Focus Adjustment: Sum of Bundles
http://www.mne.psu.edu/psgdl/FSSPhotoalbum/index1.htm
Varying PolarizationVarying PolarizationYoav Y. Schechner, Nir Karpel 2005Yoav Y. Schechner, Nir Karpel 2005
Best polarization state
Worst polarization state
Best polarization state
Recovered image
[Left] The raw images taken through a polarizer. [Right] White-balanced results: The recovered image is much clearer, especially at distant objects, than the raw image
Varying PolarizationVarying Polarization• Schechner, Narasimhan, NayarSchechner, Narasimhan, Nayar
• Instant dehazing Instant dehazing of images using of images using polarizationpolarization
Spatial Augmented Reality | Raskar 2011
Pamplona , Mohan, Oliveira, Raskar, Siggraph 2010
NETRA: Near Eye Tool for Refractive Assessment
EyeNetra.com
90
Confocal Microscopy Examples
Slides by Doug Lanman
Beyond Visible SpectrumBeyond Visible Spectrum
CedipRedShift
MIT Media LabMIT Media Lab
Camera CultureCamera Culture
Ramesh RaskarRamesh Raskar
MIT Media LabMIT Media Labhttp:// CameraCulture . info/http:// CameraCulture . info/
Computational Camera & Computational Camera & Photography:Photography:
http://www.flickr.com/photos/pgoyette/107849943/in/photostream/
Scheimpflug Scheimpflug principleprinciple
Ramesh Raskar, Computational Illumination
Computational Illumination
Edgerton 1930’sEdgerton 1930’s
Multi-flash Sequential Photography
Stroboscope(Electronic Flash)
Shutter Open
Flash Time
Ramesh Raskar, Karhan Tan, Rogerio Feris, Jingyi Yu, Matthew Turk
Mitsubishi Electric Research Labs (MERL), Cambridge, MAU of California at Santa Barbara
U of North Carolina at Chapel Hill
Non-photorealistic Camera: Non-photorealistic Camera: Depth Edge Detection Depth Edge Detection andand Stylized Stylized
Rendering Rendering usingusing Multi-Flash ImagingMulti-Flash Imaging
Depth Edges
Our MethodCanny
Flash MattingFlash Matting
Flash Matting, Jian Sun, Yin Li, Sing Bing Kang, and Heung-Yeung Shum, Siggraph 2006
DARPA Grand ChallengeDARPA Grand Challenge
The Media Lab Camera Culture
Epsilon Photography
Capture multiple photos, each with slightly different camera parameters.
• Exposure settings• Spectrum/color settings• Focus settings• Camera position• Scene illumination
The Media Lab Camera Culture
Lens Sensor
Camera
Static Scene
Image Destabilization[Mohan, Lanman et al. 2009]
The Media Lab Camera Culture
Static Scene
Lens Motion Sensor Motion
Camera
Image Destabilization[Mohan, Lanman et al. 2009]
MIT Media Lab Camera Culture
Our Prototype
MIT Media Lab Camera Culture
Adjusting the Focus Plane
all-in-focus pinhole image
MIT Media Lab Camera Culture
Defocus Exaggeration
destabilization simulates a reduced f-number
The Media Lab Camera Culture
Capturing Gigapixel Images[Kopf et al, 2007]
3,600,000,000 PixelsCreated from about 800 8 MegaPixel Images
The Media Lab Camera Culture
Capturing Gigapixel Images[Kopf et al, 2007]
Color Original Grayscale
New Method
Color2Gray: Color2Gray: Salience-Preserving Salience-Preserving
Color RemovalColor RemovalSIGGRAPH 2005
Gooch, Olsen, Tumblin, Gooch