Upload
caitlin-wilkinson
View
218
Download
0
Tags:
Embed Size (px)
Citation preview
Generalized Mosaics
Yoav Y. Schechner,
Shree Nayar
Department of Computer Science
Columbia University
Mosaics
Processing
Redundant Measurements
Generalized Mosaicing: Yoav Schechner and Shree Nayar
Scanning with Less Redundancy
Mantis Shrimp
TakataC
Different rows = Different optical characteristics
Generalized Mosaicing
Field of View
BrightnessDynamic Range Spectrum
Depth of Field
Polarization
camera
Spatially varying
filter
Schechner, Nayar, Generalized mosaics
Mosaic + High Dynamic Range
88 - 18,794
Fusion of Measurements
)(Mask),(),(Raw xyxIyxk
1
22 1
k kI
I
k k
k
I
III
22
Maximum-Likelihood Solution
),( ),( )(Mask
),(RawyxIyxI
x
yxkk
k
Schechner, Nayar, Generalized mosaics
8 bit camera Dynamic range as 16 bits
Generalized Mosaicing: Yoav Schechner and Shree Nayar
8 bits (almost)(pixels)
-2
-4
-6
-8 x
log M2
Log of the mask
==1 10-61 10-2
M
x
average row
frames y
frameyxx ),(Raw )(Mask
),(Mask),(),(Raw yxyxIyxframe
Mask Self-CalibrationUnknown filter (vignetting)
consistencyconstraints
0Raw)(Mask Raw)(Mask1221 xx
MaskttMaskmin AA
Consistency, Smoothness A
Schechner, Nayar, Generalized mosaics
Image Registration
2pair ingcorrespondeach
pixels all
2total DIFFDIFF
2total DIFF:Minimize
2pair ingcorrespondeach
pixels all
2total DIFFDIFF
Schechner, Nayar, Generalized mosaics
)(Mask),(),(Raw xyxIyxk
2pair ingcorrespondeach
pixels all
2total
DIFFDIFF :Minimize
Bias towards “no motion”
Schechner, Nayar, Generalized mosaics
Image Registration
)(Mask),(),(Raw xyxIyxk
),( )(Mask
),(RawyxI
x
yxk
k
Raw
Mask
Uncertainty
),( yxIk
1
22
22 1 where ,
k kk k
k
II
I
III
2
22
2
2
21
12pair pixel
III
III
DIFF
Schechner, Nayar, Generalized mosaics
Image Registration
Registration: Standard “Coarse to Fine”
5 pixels
10 pix 9 pix
18 pix 19 pix
4 pixels
2 pixelsmotion of
However, we need: multiscale UNCERTAINTIES 2
22
2
2
21
12pair pixel
III
III
DIFF
Schechner, Nayar, Generalized mosaics
Maximum-Likelihood Pyramid At each level L , for each pixel
LL II , estimate
,
),( ),(yx
yxIyxwI
,
),( yx
yxw
,
),( 2 yx
yxwI-1
),(
),(),(yxI
yxayxw
2
Schechner, Nayar, Generalized mosaics
Max-Likelihood Pyramid
Gaussian Pyramid1
22
22
)(
1
)(
)( ,
xx xII
xI
xIII
x
)(xg
)1(I )2(I )3(I )4(I )5(I
x
xIxgI )()(
x )1(I )2(I )3(I )4(I )5(I )1(I )2(I )3(I )4(I )5(I
ML estimation
x
xIxnI )()(
)(
)()(2 xI
xgx
1][ )(
xn
1
2 )(
x
xI
log I
Inte
nsit
y ra
nge
Spatial range x
Generalized Mosaic
x
log I
Inte
nsit
y ra
nge
Spatial range
frame
fram
e
Mosaic
Schechner, Nayar, Generalized mosaics
Generalized Mosaicing: Yoav Schechner and Shree Nayar
spectral
y
x
Variable Spectral Filter
MMuullttii SSppeeccttrraall Mosaic
500 700400 600 500 700400 600 500 700400 600
Generalized Mosaicing: Yoav Schechner and Shree Nayar
Generalized Mosaicing: Yoav Schechner and Shree Nayar
Rendering : Any Illumination
sunset fluorescent
Halogeny
x
Illumination at a Glance
Fluorescenty
x
Illuminant spectrum
Extra information
Spatially Varying Polarizer
Polarization Mosaicing: Yoav Schechner and Shree Nayar
0
0.2
0.4
0.6
0.8
1
-30
60
90
30
0
-60
-90
o
o
o
o
o
o
polarizance
polarizing angle
o
Polarization Mosaicing: Yoav Schechner and Shree Nayar
transmittance
Spatially Varying Polarizer
Raw images
Polarization Mosaicing: Yoav Schechner and Shree Nayar
Polarization Mosaic
Schechner Shamir Kiryati
JOSA-A 2000
reflected structure
transmitted painting
=
y
xDepth
Insert a prism behind the lens
Spatially Varying Focus
Generalized Mosaicing: Yoav Schechner and Shree Nayar
All FocusedAll Focused Mosaic
Generalized Mosaicing: Yoav Schechner and Shree Nayar
… and a Depth Map
What else?
Generalized Mosaicing: Yoav Schechner and Shree Nayar
Dynamic Range & Spectrum & Polarization etc.
Multi-dimensional Mosaics - Simultaneously
Signal undergoing LPF
“Band limited” signal
Minimizing aliasing
Nyquist sampling rate
111111111111111
1?10111001??
Sampling Criteria
Generalized Mosaicing: Yoav Schechner and Shree Nayar
Number of samples per scene point
resolution
length Lbandwidth B
aperture D
22 )/()/(
2
BLD
M2
log11 M
1011100101
1011100???
M # samples
1 11/2 21/4 3
92561
101110????
MosaicGeneralized Mosaic
Spatially varying filter
Wide field of view
SpectrumI
Depth + Focus
Dynamic range88 – 18,794
Polarization