Upload
duckleek
View
502
Download
0
Tags:
Embed Size (px)
DESCRIPTION
This is a ppt file for study meetings held in our lab, describing chapter 10 computational photography in the book of Szeliski's "Computer Vision: Algorithms and Applications."Aizawa-Yamasaki Lab. at The Univ. of Tokyo http://www.hal.t.u-tokyo.ac.jp/
Citation preview
Computational photography
Kazuhiro Sato
Computer Vision: Algorithms and Applications
Chapter 10
What is Computational Photography?
To enhance or extend the capabilities of digital photography
enhanceextend
Contents –computational photography-
1. Photometric calibration2. High dynamic range imaging3. Super-resolution and blur
removal4. Image matting and
compositing5. Texture analysis and
synthesis
1. Photometric calibration
By the way,
How does an image
come out?
imaging pipeline
First, we characterize• the mapping
functions• the amounts of noise
The mapping functions consists of
• the radiometric response function
• vignetting• the point spread function
1.1 Radiometric Response Function
fin
al bri
ghtn
ess
scene radiane
optics𝑀
𝐼sensor plane
often NOT linear
How can we determine the function?
• calibration chart
• polynomial approximation 𝐼=∑
𝑛=0
𝑁
𝑐𝑛𝑀𝑛
• least squares (explained later)
1.2 Noise Level Estimation
1. Segment the input image
2. Fit a linear function
3. Plot the standard deviation
4. Fit a lower envelope
estimated noise level
pixel value
𝜎
[Liu et al. ’08]
1.3 Vignetting
・ Radial Gradient Symmetry [Zheng et al. ’08]
∆ 𝐼 (𝑥 , 𝑦)
𝜕 𝐼𝜕𝑟
0− +¿𝜕 𝐼𝜕𝑟
Histogram (asymmetry)
𝑍 (𝑟 , 𝜃 )=𝐼 (𝑟 ,𝜃 )𝑉 (𝑟 )
Find the optimal that minimizes the asymmetry using “maximum a posteriori (MAP)”
: the image with vignetting: the vignetting-free image: the vignetting function
𝑉=argmax𝑉
𝑃 (𝑉∨𝑍 )∝argmax𝑉
𝑃 (𝑍∨𝑉 ) 𝑃 (𝑉 )
1.4 Optical Blur Estimation
・ PSF Estimation [Joshi et al. ’08]
Estimating the PSF kernel using a calibration pattern𝐾=argmin
𝐾‖𝐵−𝐷 ( 𝐼 ∗ 𝐾 )‖2
: sensed (blurred) image: predicted (sharp) image: downsampling operator
Solving a Bayesian framework
using a maximum a posteriori (MAP)
argmax𝐾
𝑃 (𝐾∨𝐵 )=argmax𝐾
𝑃 (𝐵∨𝐾 ) 𝑃 (𝐾 )𝑃 (𝐵 )
𝐿 (𝐵∨𝐾 )=‖𝑀 (𝐵 )−𝑀 ( 𝐼⊗ 𝐾 )‖2
𝜎 2
¿ argmin𝐾𝐿 (𝐵∨𝐾 )+𝐿 (𝐾 )
𝐿 (𝐾 )= λ𝛾‖𝛻𝐾‖2
・ Recovering the PSF without calibration
1. Fit step edges to the elongated ones
2. Apply the least squares for every surrounded pixel𝐾=argmin
𝐾‖𝐵−𝐷 ( 𝐼 ∗ 𝐾 )‖2
2. High Dynamic Range Imaging
Dynamic Range
0.01 1 100 10000
moonlight sunlight
Luminance[cd/m2]
indoor lighting
Human eye
CCD camera
The natural world is too bright to be captured!
Creating a properly exposed photo (High dynamic range imaging)
different exposures
properly exposed photo
create
How to create such a photo?
1. Estimate the radiometric response function
2. Estimate a radiance map
3. Tone map the resulting HDR image into a 8-bit one
1. Estimate the radiometric response function
𝑧𝑖𝑗= 𝑓 (𝐸𝑖 𝑡 𝑗 )log 𝐸 𝑖
𝑧 𝑖𝑗
𝑓 −1 (𝑧 𝑖𝑗 )=𝐸𝑖 𝑡 𝑗
th exposure ( )
𝑔 ( 𝑧𝑖𝑗 )=log 𝑓 −1 (𝑧𝑖𝑗 )=log 𝐸𝑖+ log 𝑡 𝑗
[Debevec et al. ’97]
min𝐸=∑𝑖∑𝑗
𝑤 (𝑧𝑖𝑗 ) [𝑔 (𝑧𝑖𝑗 )−log 𝐸 𝑖− log 𝑡 𝑗 ]2+¿ λ∑𝑘
𝑤 (𝑘 )𝑔′ ′ (𝑘 )2 ¿
log 𝐸 𝑖
𝑧 𝑖𝑗
h𝑤 𝑒𝑟𝑒𝑤 (𝑧 )={𝑧−𝑧𝑚𝑖𝑛 𝑧≤ (𝑧𝑚𝑖𝑛+𝑧𝑚𝑎𝑥) /2𝑧𝑚𝑎𝑥− 𝑧 𝑧>( 𝑧𝑚𝑖𝑛+ 𝑧𝑚𝑎𝑥 )/2
th exposure ( )
2. Estimate a radiance map[Mitsunaga et al. ’99]
different exposures
𝑧𝑖 ,1 𝑧𝑖 ,2
𝑧𝑖 ,1
𝑧𝑖 ,2
Merging the input images into a composite radiance map.
log 𝐸 𝑖=∑𝑗
𝑤 ( 𝑧𝑖𝑗 ) [𝑔 (𝑧 𝑖𝑗 )− log 𝑡 𝑗 ]∑𝑗
𝑤 (𝑧𝑖𝑗 )
h𝑤 𝑒𝑟𝑒𝑤 (𝑧 )=𝑔 (𝑧 )/𝑔′ (𝑧 )
different exposures
𝑧𝑖 ,1 𝑧𝑖 ,2
radiance map (grayscale)
merge
3. Tone map the resulting HDR image into a 8-bit one
bits / pixel
8-bit image
HDR image
8 bits / pixel
input HDR image gamma applied to each channel
gamma applied to luminance only
This global approach fails to preserve details in regions with widely varying exposures.
・ Global tone mapping using a transfer curve[Larson et al. ’97]
2.1 Tone mapping
・ Local tone mapping using bilateral filter[Durand et al. ’02]
This approach doesn’t create visible halos around the edges.
result withlow-pass filtering
(visible halos)
result withbilateral filtering
(no halos)
・ Gradient domain tone mapping [Fattal et al. ’02]
𝐿𝑖𝑛
𝐿𝑜𝑢𝑡
𝐶𝑜𝑢𝑡=(𝐶𝑖𝑛
𝐿𝑖𝑛)𝑠
𝐿𝑜𝑢𝑡𝑠∈ [0.4 ,0.6 ]
The new luminance is combined with the original color image.
𝐺 (𝑥 )=𝐻 ′ (𝑥 )Φ (𝑥 )
Attenuation map Tone-mapped result
2.2 Flash photography [Petschnigg et al. 04]
Combine flash and non-flash images to achieve better photos
Joint bilateral filter’s kernel
domain kernel
𝑤 (𝑖 , 𝑗 ,𝑘 , 𝑙 )=exp (− (𝑖−𝑘 )2+( 𝑗−𝑙 )2
2𝜎𝑑2 ) ∙exp(−‖𝐹 (𝑖 , 𝑗 )−𝐹 (𝑘 , 𝑙 )‖2
2𝜎𝑟2 )
range kernel
3. Super-resolution and blur removal
3.1 Color image demosaicing
Bayer RGB patternin a camera sensor
Full-color RGB
interpolate
・ Bayesian demosaicing with a two-color prior
Original Bilinear [Bennett et al. ’06]
[Bennett et al. ’06]
Two-color model [Bennett et al. ’06]
4. Image matting and compositing
Image “matting” and “compositing”?
Input OutputAlpha-matting
matting compositing
What is the problem in matting and compositing?
foreground
background
boundary
𝛼=1𝛼=0
0<𝛼<1
We need the opacity !
Failed example of matting
Alpha matte Composite Inset
Input
4.1 Blue screen matting [Smith et al. ‘96]
𝑐𝑜= [𝑅𝑜𝐺𝑜0 ]𝑐 𝑓=[𝑅 𝑓𝐺 𝑓 𝐵 𝑓 ] 𝑐𝑘=[ 00𝐵𝑘 ]
{ 𝑅𝑜=𝑅 𝑓
𝐺𝑜=𝐺 𝑓
(1−𝛼𝑜 )𝐵𝑘=𝐵 𝑓
solve
)
∴𝐶𝑜=[𝑅𝑜𝐺𝑜0𝛼𝑜 ]
¿ [𝑅𝑓 𝐺 𝑓 01−𝐵 𝑓
𝐵𝑘]
𝑐𝑜= [𝑅𝑜𝐺𝑜𝐵𝑜 ]𝑐 𝑓 1=[𝑅 𝑓 1
𝐺 𝑓 1𝐵 𝑓 1 ]
𝑐𝑘1=[00𝐵𝑘1 ]
solve
)
∴𝐶𝑜=[𝑅𝑜𝐺𝑜𝐵𝑜𝛼𝑜 ]
¿ [𝑅𝑓 1𝐺 𝑓 1
𝐵 𝑓 2𝐵𝑘1−𝐵 𝑓 1
𝐵𝑘2
𝐵𝑘1−𝐵𝑘2
1−𝐵 𝑓 1−𝐵 𝑓 2
𝐵𝑘1−𝐵𝑘2
]
・ Two-screen matting[Smith et al. ’96]
𝑐 𝑓 2=[𝑅 𝑓 2
𝐺 𝑓 2𝐵 𝑓 2 ]
𝑐𝑘2= [00𝐵𝑘2 ]
{𝐵𝑜+ (1−𝛼𝑜 )𝐵𝑘1=𝐵 𝑓 1
𝐵𝑜+(1−𝛼𝑜 )𝐵𝑘2=𝐵 𝑓 2
4.2 Natural image matting
Input Hand-drawn trimap
Alpha map
foreground composite
[Chuang et al. ’01]
・ Knockout[Berman et al. ’00]
𝐹 𝑖
𝐵𝑖
𝐶=𝛼𝐹+ (1−𝛼 ) 𝐵
1. Estimate and
𝐹=∑𝑖
𝑤 𝑖𝐹 𝑖 𝐵 ′=∑𝑖
𝑤 𝑖𝐵𝑖
2. Refine → 𝐹𝐵 ′⊥𝐵𝐵 ′ //
3. Estimate by axis-projection
𝛼𝑅=𝑓 𝑅 (𝐶 )− 𝑓 𝑅 (𝐵 )𝑓 𝑅 (𝐹 )− 𝑓 𝑅 (𝐵 )
,𝛼𝐺 ,𝛼𝐵
𝛼 𝑓𝑖𝑛𝑎𝑙= ∑𝑋=𝑅 ,𝐺 ,𝐵
[ 𝑓 𝑋 (𝐹 )− 𝑓 𝑋 (𝐵 ) ] ∙𝛼 𝑋
・ Bayesian approach[Chuang et al. ’01]
𝐹
𝐵
𝐶observation
most likely estimates
arg max𝐹 ,𝐵 ,𝛼
𝑃 (𝐹 ,𝐵 ,𝛼∨𝐶 )
𝐿 (𝐶|𝐹 ,𝐵 ,𝛼 )=−‖𝐶−𝛼𝐹− (1−𝛼 ) 𝐵‖2
𝜎𝐶2
𝐿 (𝐹 )=− (𝐹−𝐹 )𝑇 Σ𝐹−1 (𝐹−𝐹 )/2
: same as the above except : const.
・ Comparison of natural image mattings
4.3 Optimization-based matting
・ Border matting
1. Get a trimap by hard segmentation
𝑇 𝐹 (𝛼=1 ) ,𝑇 𝑢 ,𝑇 𝐵 (𝛼=0 )
[Rother et al. ’04]
2. Minimizing the energy of mapping
𝐸= ∑𝑛∈𝑇 𝑈
~𝐷𝑛 (𝛼𝑛)+∑𝑡=1
𝑇 ~𝑉 (∆𝑡 ,𝜎𝑡 ,∆𝑡+1 ,𝜎 𝑡+!)
data term smoothness term
・ Color line (closed-form) matting
1. Estimate for each
𝛼 𝑖𝑒𝑠𝑡𝑖𝑚𝑎𝑡𝑒𝑑=𝒂𝒌 ∙ (𝑪 𝒊−𝑩𝟎)=𝒂𝒌 ∙𝑪+𝑏𝑘
2. Minimize the deviations of the actual from the above 𝐸𝛼=∑
𝑘 ( ∑𝑖∈𝑊 𝑘
(𝛼𝑖−𝛼 𝑖𝑒𝑠𝑡𝑖𝑚𝑎𝑡𝑒𝑑 )2+𝜖‖𝒂𝒌‖)
data term regularization term
local window local values
[Levin et al. ’08]
3. Eliminate to get a final energy
4.4 Smoke, shadow, and flash matting
・ Smoke matting[Chuang et al. ’02]
input frameremoving the foreground
object
estimated alpha matte
insertion of new objects
・ Shadow matting[Chuang et al. ’03]
4.5 Video matting
[Chuang et al. ’02]
5. Texture analysis and synthesis
Texture synthesis is …
Small patch
Similar-looking larger patch
synthesize
Texture synthesis using non-parametric sampling
Texture synthesis using non-parametric sampling
[Efros et al. ’99]
[Efros et al. ’01]
5.1 hole filling and inpainting
inpaint
Original Inpainted result
・ Exemplar-based inpainting[Criminisi et al. ’04]
1. Compute patch priorities
𝑃 (𝒑 )=𝐶 (𝒑 )𝐷 (𝒑 )
C (𝒑 )=∑
𝒒∈Ψ 𝒑∩ ( 𝑰−𝛀 )
𝐶 (𝒒)
|Ψ 𝒑|
𝐷 (𝒑 )=|𝛻 𝐼𝑝
⊥∙𝒏𝒑|𝛼
data termconfidence term
2. Find the most similar patch to with top priorityΨ �̂�=arg min
Ψ 𝒒∈Φ𝑑 (Ψ �̂� ,Ψ 𝒒 )
3. Update confidence values𝐶 (𝒑 )=𝐶 (�̂� )∀ 𝒑∈Ψ �̂�∩Ω
onio
n p
eel
[Cri
min
isi et
al.
’04
]
original removed region
Remain the gradient alongthe region boundary
5.2 Non-photorealistic rendering
Non-photorealistic rendering using texture synthesis
・ Texture transfer・ Image analogies
・ Texture transfer
[Efros et al. ’01]
input outputtexture transfer
input outputtexture transfer
1. Find the most similar block 𝐸𝑖=𝛼𝐸𝑜𝑣𝑒𝑟𝑙𝑎𝑝+(1−𝛼 ) 𝐸 𝑙𝑢𝑚𝑖𝑛𝑎𝑛𝑐𝑒
𝐸 𝑗
𝐸𝑖
𝐵 𝑗
𝐵𝑖
( means ordinary texture synthesis. )
2. Find the minimum error boundary cut𝑒=(𝐵1
𝑜𝑣 −𝐵2𝑜𝑣 )2
𝐵1𝑜𝑣 ,𝐵2
𝑜𝑣
( the minimal cost path corresponds to the valley in the surface . )
・ Image analogies
[Hertzmann et al. ’01]
𝐴 𝐴 ′ 𝐵 𝐵 ′
?: :¿: :¿
synthesize this image
𝐴 𝐴 ′
𝐵 𝐵 ′
unfiltered example NPR-filtered example
target result
𝐴 𝐴 ′
𝐵 𝐵 ′
unfiltered example filtered example
target result
Thank you!