View
1.468
Download
0
Category
Tags:
Preview:
DESCRIPTION
study
Citation preview
Image and Deoth from a Conventional Camera
with a Coded Apertrue
Anat Levin, Rob Fergus, Frédo Durand, William
Freeman
MIT CSAIL
study
Single input image
real objects
output #1: Depth mapoutput #2: all-infocused image
Coded Aperture
Conventional aperture and depth of field
Small aperture
Big aperture
Focal plane
Object
LensCamera sensor
Point spread
function
http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.pptFocal plane
Depth from defocus
LensObject
Camera sensor
Point spread
function
http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.pptFocal plane
Depth from defocus
Focal plane
LensCamera sensor
Point spread
function
Object
http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.ppt
Depth from defocus
Defocus as local convolution
Input defocused image
xfy k Local
observed sub-
window
Calibrated blur
kernels at depth K
Sharp sub-
window
Depth k=1
Depth k=2
Depth k=3
Introduction Estimation of depth – a branch of Computational Photography Most challenges of y = fk * x
Input Ringing with the traditional Richardson-Lucy algorithm
• Hard to de-convolve even when kernel is known
• Hard to identify correct scale:
?? Correct
scale
Smaller scale
? Larger scale
Related work – depth estimation Active methods – additional illumination sources• Structured light methods
Nayar et al. ICCV 95Zhang and Nayar, SIGGRAPH 06
Projection Defocus Analysis for Capture and Image Display, Zhang and Nayar, 06
Related work – depth estimation
Passive methods – changes of focus • Depth from defocus (DFD)
Pentland, IEEE 87Chaudhuri, Favaro et al. , 99
• Blind Deconvolution – image prior, maximum likelyhood
Kundur and Hatzinakos , IEEE 96Levin, NIPS 06
• Coded apertures for light gatheringFenimore and Cannon, Optics 78
Related work – depth estimation
• Plenoptic /light field cameraAdelson and Wang, IEEE 92Ng et al., 05
• Wavefront codingCathey & Dowski, Optics 94, 951.Rays don't converge anymore2.Image blur is the same for all
depth3.Blur spectrum does not have
too many zeros
Passive methods – changes of viewpoints
CompPhoto06/html/lecturenotes/25_LightField_6.pdf
OverviewTry deconvolving local input windows with different
scaled filters:
?? Correct
scale
Smaller scale
? Larger scale
Somehow: select best scale
yxfk
Challenges & contributions
Hard to de-convolve even when kernel is knownIDEA 1: Natural images prior
Hard to identify correct scaleIDEA 2: Coded Aperture
? =
=?
Solution 1:
Solution 2:
Deconvolution is ill posed
IDEA 1: Natural images prior
Image
gradient
put a penalty on gradients
Natural images have sparse gradients
Natural
Unnatural
What makes images special?
Deconvolution with prior
i ik xyxf )(min arg x
2 Convolution
errorDerivatives
prior
High
Low ?
?
2
2Equal convolution error
Comparing deconvolution algorithms
Input
Richardson-Lucy
Gaussian prior
“spread” gradients
Sparse prior
“localizes” gradients
8.0)( xx 2
)( xx
Statistical Model of Images
“Deconvolution using natural image priors”, Levin et. al., ETAI 07
Spatial domain
Frequency domain
Maximum a-posteriori P(x|y)
Image prior (gradient here)
Gradient operator
For Gaussian priors
For sparse priors
likelyhood
Minimize deconvolution error
Deconvolution using a Gaussian prior
Note: solved in the frequency domain in a few seconds for MB size file
Using an iterative reweighted least squares process (IRLS) [Meer 2004; Levin and Weiss to appear]
Cannot solve in frequency domain
Deconvolution using a sparse prior
Note: solved in the frequency domain around 1 hour on 2.4Ghz CPR for 2MB file
Iterative reweighted least squares process (IRLS)
Recall: OverviewTry deconvolving local input windows with different
scaled filters:
?? Correct
scale
Smaller scale
? Larger scale
Somehow: select best scale
yxfk
Challenge: smaller scale not so different than correct
IDEA 2: Coded Aperture
Mask (code) in aperture planeMake defocus patterns different from natural
images and easier to discriminate
Conventional aperture
Our coded aperture
Lens with coded aperture
Lens with coded
aperture
Camera sensor
Point spread
function
Image of a defocused point light
source
Aperture pattern
Object
Focal plane
Why coded ?
Convention
al Coded
Coded aperture- reduce uncertainty in scale identification
Correct
scale
Smaller scale
Larger scale
Why coded ?
Convention
al Coded
Coded aperture- reduce uncertainty in scale identification
Correct
scale
Smaller scale
Larger scale
Fourier transforms of 1D slide through the blur pattern
Coded aperture: Scale estimation and division by zero
?
?
=
=
0 Frequency
sp
ectr
um
0 Frequency
sp
ectr
um
0 Frequency
sp
ectr
um
0 Frequency
sp
ectr
um
0 Frequency
sp
ectr
um
Estimated image
spatial ringing
Division by zero
Estimated image
Filter, wrong scale
Filter, correct scale
Observed image
Division by zero with a conventional aperture ?
?
?
=
=
0 Frequency
sp
ectr
um
0 Frequency
sp
ectr
um
0 Frequency
sp
ectr
um
0 Frequency
sp
ectr
um
0 Frequency
sp
ectr
um
Estimated image
no spatial ringing
Tiny value at ω
Estimated image
Filter, wrong scale
Filter, correct scale
Observed image
No zero at ω !
No zero at ω !
ω is zero !
Filter Selection Criterion
The filter f has good depth discrimination - blurry image distributions Pk1(y) and Pk2(y) at depths k1 and k2 should not be similar
KL-divergence scores y
kkkkk dyyPyPyPyPyP 21121KL loglog,D
Filter Design Practical constrains
1. Binary filter to construct accurately2. Cut the filter from a single piece3. Avoid excessive radial distortion4. Avoid using the full aperture5. Diffraction impose a min size on the holes
in the file Spec.
13x13 patterns with 1 mm holesEach pattern, 8 different scales Varying between 5~15 pixels in width
Filter Design
Conventional
Conventional
Blur scale identification
Not robust at high-frequency noise
Un-normalized energy term
λk learn to minimize the scale misclassification error on a set of traning images
Ek is approximate by the reconstruction error by ML solution
x* is the deblurred image
Regularizing depth estimation
Results
Applications Digital refocusing from a single image
e.g. Synthesis an all-focus imagee.g. Post-exposure
Conclusion Pros.
All-infocus image and depth at a single shotNo loss of image resolution (compared with Plenoptic camera) Simple modificationCoded apertureConventional aperture
Cons.50 % light is blockedDepth is coarseMay need manual
correction
Recommended