72
Computational photography Kazuhiro Sato Computer Vision: Algorithms and Applications Chapter 10

Chap. 10 computational photography

Embed Size (px)

DESCRIPTION

This is a ppt file for study meetings held in our lab, describing chapter 10 computational photography in the book of Szeliski's "Computer Vision: Algorithms and Applications."Aizawa-Yamasaki Lab. at The Univ. of Tokyo http://www.hal.t.u-tokyo.ac.jp/

Citation preview

Page 1: Chap. 10 computational photography

Computational photography

Kazuhiro Sato

Computer Vision: Algorithms and Applications

Chapter 10

Page 2: Chap. 10 computational photography

What is Computational Photography?

Page 3: Chap. 10 computational photography

To enhance or extend the capabilities of digital photography

enhanceextend

Page 4: Chap. 10 computational photography

Contents –computational photography-

1. Photometric calibration2. High dynamic range imaging3. Super-resolution and blur

removal4. Image matting and

compositing5. Texture analysis and

synthesis

Page 5: Chap. 10 computational photography

1. Photometric calibration

Page 6: Chap. 10 computational photography

By the way,

How does an image

come out?

Page 7: Chap. 10 computational photography

imaging pipeline

Page 8: Chap. 10 computational photography

First, we characterize• the mapping

functions• the amounts of noise

Page 9: Chap. 10 computational photography

The mapping functions consists of

• the radiometric response function

• vignetting• the point spread function

Page 10: Chap. 10 computational photography

1.1 Radiometric Response Function

fin

al bri

ghtn

ess

scene radiane

optics𝑀

𝐼sensor plane

often NOT linear

Page 11: Chap. 10 computational photography

How can we determine the function?

• calibration chart

• polynomial approximation 𝐼=∑

𝑛=0

𝑁

𝑐𝑛𝑀𝑛

• least squares (explained later)

Page 12: Chap. 10 computational photography

1.2 Noise Level Estimation

1. Segment the input image

2. Fit a linear function

3. Plot the standard deviation

4. Fit a lower envelope

estimated noise level

pixel value

𝜎

[Liu et al. ’08]

Page 13: Chap. 10 computational photography

1.3 Vignetting

・ Radial Gradient Symmetry [Zheng et al. ’08]

∆ 𝐼 (𝑥 , 𝑦)

𝜕 𝐼𝜕𝑟

0− +¿𝜕 𝐼𝜕𝑟

Histogram (asymmetry)

Page 14: Chap. 10 computational photography

𝑍 (𝑟 , 𝜃 )=𝐼 (𝑟 ,𝜃 )𝑉 (𝑟 )

Find the optimal that minimizes the asymmetry using “maximum a posteriori (MAP)”

: the image with vignetting: the vignetting-free image: the vignetting function

𝑉=argmax𝑉

𝑃 (𝑉∨𝑍 )∝argmax𝑉

𝑃 (𝑍∨𝑉 ) 𝑃 (𝑉 )

Page 15: Chap. 10 computational photography
Page 16: Chap. 10 computational photography

1.4 Optical Blur Estimation

・ PSF Estimation [Joshi et al. ’08]

Estimating the PSF kernel using a calibration pattern𝐾=argmin

𝐾‖𝐵−𝐷 ( 𝐼 ∗ 𝐾 )‖2

: sensed (blurred) image: predicted (sharp) image: downsampling operator

Page 17: Chap. 10 computational photography

Solving a Bayesian framework

using a maximum a posteriori (MAP)

argmax𝐾

𝑃 (𝐾∨𝐵 )=argmax𝐾

𝑃 (𝐵∨𝐾 ) 𝑃 (𝐾 )𝑃 (𝐵 )

𝐿 (𝐵∨𝐾 )=‖𝑀 (𝐵 )−𝑀 ( 𝐼⊗ 𝐾 )‖2

𝜎 2

¿ argmin𝐾𝐿 (𝐵∨𝐾 )+𝐿 (𝐾 )

𝐿 (𝐾 )= λ𝛾‖𝛻𝐾‖2

Page 18: Chap. 10 computational photography

・ Recovering the PSF without calibration

1. Fit step edges to the elongated ones

2. Apply the least squares for every surrounded pixel𝐾=argmin

𝐾‖𝐵−𝐷 ( 𝐼 ∗ 𝐾 )‖2

Page 19: Chap. 10 computational photography

2. High Dynamic Range Imaging

Page 20: Chap. 10 computational photography

Dynamic Range

0.01 1 100 10000

moonlight sunlight

Luminance[cd/m2]

indoor lighting

Human eye

CCD camera

Page 21: Chap. 10 computational photography

The natural world is too bright to be captured!

Page 22: Chap. 10 computational photography

Creating a properly exposed photo (High dynamic range imaging)

different exposures

properly exposed photo

create

Page 23: Chap. 10 computational photography

How to create such a photo?

1. Estimate the radiometric response function

2. Estimate a radiance map

3. Tone map the resulting HDR image into a 8-bit one

Page 24: Chap. 10 computational photography

1. Estimate the radiometric response function

𝑧𝑖𝑗= 𝑓 (𝐸𝑖 𝑡 𝑗 )log 𝐸 𝑖

𝑧 𝑖𝑗

𝑓 −1 (𝑧 𝑖𝑗 )=𝐸𝑖 𝑡 𝑗

th exposure ( )

𝑔 ( 𝑧𝑖𝑗 )=log 𝑓 −1 (𝑧𝑖𝑗 )=log 𝐸𝑖+ log 𝑡 𝑗

[Debevec et al. ’97]

Page 25: Chap. 10 computational photography

min𝐸=∑𝑖∑𝑗

𝑤 (𝑧𝑖𝑗 ) [𝑔 (𝑧𝑖𝑗 )−log 𝐸 𝑖− log 𝑡 𝑗 ]2+¿ λ∑𝑘

𝑤 (𝑘 )𝑔′ ′ (𝑘 )2 ¿

log 𝐸 𝑖

𝑧 𝑖𝑗

h𝑤 𝑒𝑟𝑒𝑤 (𝑧 )={𝑧−𝑧𝑚𝑖𝑛 𝑧≤ (𝑧𝑚𝑖𝑛+𝑧𝑚𝑎𝑥) /2𝑧𝑚𝑎𝑥− 𝑧 𝑧>( 𝑧𝑚𝑖𝑛+ 𝑧𝑚𝑎𝑥 )/2

th exposure ( )

Page 26: Chap. 10 computational photography

2. Estimate a radiance map[Mitsunaga et al. ’99]

different exposures

𝑧𝑖 ,1 𝑧𝑖 ,2

𝑧𝑖 ,1

𝑧𝑖 ,2

Merging the input images into a composite radiance map.

Page 27: Chap. 10 computational photography

log 𝐸 𝑖=∑𝑗

𝑤 ( 𝑧𝑖𝑗 ) [𝑔 (𝑧 𝑖𝑗 )− log 𝑡 𝑗 ]∑𝑗

𝑤 (𝑧𝑖𝑗 )

h𝑤 𝑒𝑟𝑒𝑤 (𝑧 )=𝑔 (𝑧 )/𝑔′ (𝑧 )

different exposures

𝑧𝑖 ,1 𝑧𝑖 ,2

radiance map (grayscale)

merge

Page 28: Chap. 10 computational photography

3. Tone map the resulting HDR image into a 8-bit one

bits / pixel

8-bit image

HDR image

8 bits / pixel

Page 29: Chap. 10 computational photography

input HDR image gamma applied to each channel

gamma applied to luminance only

This global approach fails to preserve details in regions with widely varying exposures.

・ Global tone mapping using a transfer curve[Larson et al. ’97]

2.1 Tone mapping

Page 30: Chap. 10 computational photography

・ Local tone mapping using bilateral filter[Durand et al. ’02]

Page 31: Chap. 10 computational photography

This approach doesn’t create visible halos around the edges.

result withlow-pass filtering

(visible halos)

result withbilateral filtering

(no halos)

Page 32: Chap. 10 computational photography

・ Gradient domain tone mapping [Fattal et al. ’02]

𝐿𝑖𝑛

𝐿𝑜𝑢𝑡

𝐶𝑜𝑢𝑡=(𝐶𝑖𝑛

𝐿𝑖𝑛)𝑠

𝐿𝑜𝑢𝑡𝑠∈ [0.4 ,0.6 ]

The new luminance is combined with the original color image.

𝐺 (𝑥 )=𝐻 ′ (𝑥 )Φ (𝑥 )

Page 33: Chap. 10 computational photography

Attenuation map Tone-mapped result

Page 34: Chap. 10 computational photography

2.2 Flash photography [Petschnigg et al. 04]

Combine flash and non-flash images to achieve better photos

Page 35: Chap. 10 computational photography

Joint bilateral filter’s kernel

domain kernel

𝑤 (𝑖 , 𝑗 ,𝑘 , 𝑙 )=exp (− (𝑖−𝑘 )2+( 𝑗−𝑙 )2

2𝜎𝑑2 ) ∙exp(−‖𝐹 (𝑖 , 𝑗 )−𝐹 (𝑘 , 𝑙 )‖2

2𝜎𝑟2 )

range kernel

Page 36: Chap. 10 computational photography

3. Super-resolution and blur removal

Page 37: Chap. 10 computational photography

3.1 Color image demosaicing

Bayer RGB patternin a camera sensor

Full-color RGB

interpolate

Page 38: Chap. 10 computational photography

・ Bayesian demosaicing with a two-color prior

Original Bilinear [Bennett et al. ’06]

[Bennett et al. ’06]

Page 39: Chap. 10 computational photography

Two-color model [Bennett et al. ’06]

Page 40: Chap. 10 computational photography

4. Image matting and compositing

Page 41: Chap. 10 computational photography

Image “matting” and “compositing”?

Input OutputAlpha-matting

matting compositing

Page 42: Chap. 10 computational photography

What is the problem in matting and compositing?

foreground

background

boundary

𝛼=1𝛼=0

0<𝛼<1

We need the opacity !

Page 43: Chap. 10 computational photography

Failed example of matting

Alpha matte Composite Inset

Input

Page 44: Chap. 10 computational photography

4.1 Blue screen matting [Smith et al. ‘96]

𝑐𝑜= [𝑅𝑜𝐺𝑜0 ]𝑐 𝑓=[𝑅 𝑓𝐺 𝑓 𝐵 𝑓 ] 𝑐𝑘=[ 00𝐵𝑘 ]

{ 𝑅𝑜=𝑅 𝑓

𝐺𝑜=𝐺 𝑓

(1−𝛼𝑜 )𝐵𝑘=𝐵 𝑓

solve

)

∴𝐶𝑜=[𝑅𝑜𝐺𝑜0𝛼𝑜 ]

¿ [𝑅𝑓 𝐺 𝑓 01−𝐵 𝑓

𝐵𝑘]

Page 45: Chap. 10 computational photography

𝑐𝑜= [𝑅𝑜𝐺𝑜𝐵𝑜 ]𝑐 𝑓 1=[𝑅 𝑓 1

𝐺 𝑓 1𝐵 𝑓 1 ]

𝑐𝑘1=[00𝐵𝑘1 ]

solve

)

∴𝐶𝑜=[𝑅𝑜𝐺𝑜𝐵𝑜𝛼𝑜 ]

¿ [𝑅𝑓 1𝐺 𝑓 1

𝐵 𝑓 2𝐵𝑘1−𝐵 𝑓 1

𝐵𝑘2

𝐵𝑘1−𝐵𝑘2

1−𝐵 𝑓 1−𝐵 𝑓 2

𝐵𝑘1−𝐵𝑘2

]

・ Two-screen matting[Smith et al. ’96]

𝑐 𝑓 2=[𝑅 𝑓 2

𝐺 𝑓 2𝐵 𝑓 2 ]

𝑐𝑘2= [00𝐵𝑘2 ]

{𝐵𝑜+ (1−𝛼𝑜 )𝐵𝑘1=𝐵 𝑓 1

𝐵𝑜+(1−𝛼𝑜 )𝐵𝑘2=𝐵 𝑓 2

Page 46: Chap. 10 computational photography

4.2 Natural image matting

Input Hand-drawn trimap

Alpha map

foreground composite

[Chuang et al. ’01]

Page 47: Chap. 10 computational photography

・ Knockout[Berman et al. ’00]

𝐹 𝑖

𝐵𝑖

𝐶=𝛼𝐹+ (1−𝛼 ) 𝐵

1. Estimate and

𝐹=∑𝑖

𝑤 𝑖𝐹 𝑖 𝐵 ′=∑𝑖

𝑤 𝑖𝐵𝑖

2. Refine → 𝐹𝐵 ′⊥𝐵𝐵 ′ //

3. Estimate by axis-projection

𝛼𝑅=𝑓 𝑅 (𝐶 )− 𝑓 𝑅 (𝐵 )𝑓 𝑅 (𝐹 )− 𝑓 𝑅 (𝐵 )

,𝛼𝐺 ,𝛼𝐵

𝛼 𝑓𝑖𝑛𝑎𝑙= ∑𝑋=𝑅 ,𝐺 ,𝐵

[ 𝑓 𝑋 (𝐹 )− 𝑓 𝑋 (𝐵 ) ] ∙𝛼 𝑋

Page 48: Chap. 10 computational photography

・ Bayesian approach[Chuang et al. ’01]

𝐹

𝐵

𝐶observation

most likely estimates

arg max𝐹 ,𝐵 ,𝛼

𝑃 (𝐹 ,𝐵 ,𝛼∨𝐶 )

Page 49: Chap. 10 computational photography

𝐿 (𝐶|𝐹 ,𝐵 ,𝛼 )=−‖𝐶−𝛼𝐹− (1−𝛼 ) 𝐵‖2

𝜎𝐶2

𝐿 (𝐹 )=− (𝐹−𝐹 )𝑇 Σ𝐹−1 (𝐹−𝐹 )/2

: same as the above except : const.

Page 50: Chap. 10 computational photography

・ Comparison of natural image mattings

Page 51: Chap. 10 computational photography

4.3 Optimization-based matting

・ Border matting

1. Get a trimap by hard segmentation

𝑇 𝐹 (𝛼=1 ) ,𝑇 𝑢 ,𝑇 𝐵 (𝛼=0 )

[Rother et al. ’04]

Page 52: Chap. 10 computational photography

2. Minimizing the energy of mapping

𝐸= ∑𝑛∈𝑇 𝑈

~𝐷𝑛 (𝛼𝑛)+∑𝑡=1

𝑇 ~𝑉 (∆𝑡 ,𝜎𝑡 ,∆𝑡+1 ,𝜎 𝑡+!)

data term smoothness term

Page 53: Chap. 10 computational photography

・ Color line (closed-form) matting

1. Estimate for each

𝛼 𝑖𝑒𝑠𝑡𝑖𝑚𝑎𝑡𝑒𝑑=𝒂𝒌 ∙ (𝑪 𝒊−𝑩𝟎)=𝒂𝒌 ∙𝑪+𝑏𝑘

2. Minimize the deviations of the actual from the above 𝐸𝛼=∑

𝑘 ( ∑𝑖∈𝑊 𝑘

(𝛼𝑖−𝛼 𝑖𝑒𝑠𝑡𝑖𝑚𝑎𝑡𝑒𝑑 )2+𝜖‖𝒂𝒌‖)

data term regularization term

local window local values

[Levin et al. ’08]

Page 54: Chap. 10 computational photography

3. Eliminate to get a final energy

Page 55: Chap. 10 computational photography

4.4 Smoke, shadow, and flash matting

・ Smoke matting[Chuang et al. ’02]

input frameremoving the foreground

object

estimated alpha matte

insertion of new objects

Page 56: Chap. 10 computational photography

・ Shadow matting[Chuang et al. ’03]

Page 57: Chap. 10 computational photography

4.5 Video matting

[Chuang et al. ’02]

Page 58: Chap. 10 computational photography

5. Texture analysis and synthesis

Page 59: Chap. 10 computational photography

Texture synthesis is …

Small patch

Similar-looking larger patch

synthesize

Page 60: Chap. 10 computational photography

Texture synthesis using non-parametric sampling

Texture synthesis using non-parametric sampling

[Efros et al. ’99]

[Efros et al. ’01]

Page 61: Chap. 10 computational photography

5.1 hole filling and inpainting

inpaint

Original Inpainted result

Page 62: Chap. 10 computational photography

・ Exemplar-based inpainting[Criminisi et al. ’04]

1. Compute patch priorities

𝑃 (𝒑 )=𝐶 (𝒑 )𝐷 (𝒑 )

C (𝒑 )=∑

𝒒∈Ψ 𝒑∩ ( 𝑰−𝛀 )

𝐶 (𝒒)

|Ψ 𝒑|

𝐷 (𝒑 )=|𝛻 𝐼𝑝

⊥∙𝒏𝒑|𝛼

data termconfidence term

Page 63: Chap. 10 computational photography

2. Find the most similar patch to with top priorityΨ �̂�=arg min

Ψ 𝒒∈Φ𝑑 (Ψ �̂� ,Ψ 𝒒 )

3. Update confidence values𝐶 (𝒑 )=𝐶 (�̂� )∀ 𝒑∈Ψ �̂�∩Ω

Page 64: Chap. 10 computational photography

onio

n p

eel

[Cri

min

isi et

al.

’04

]

original removed region

Remain the gradient alongthe region boundary

Page 65: Chap. 10 computational photography

5.2 Non-photorealistic rendering

Non-photorealistic rendering using texture synthesis

・ Texture transfer・ Image analogies

Page 66: Chap. 10 computational photography

・ Texture transfer

[Efros et al. ’01]

input outputtexture transfer

input outputtexture transfer

Page 67: Chap. 10 computational photography

1. Find the most similar block 𝐸𝑖=𝛼𝐸𝑜𝑣𝑒𝑟𝑙𝑎𝑝+(1−𝛼 ) 𝐸 𝑙𝑢𝑚𝑖𝑛𝑎𝑛𝑐𝑒

𝐸 𝑗

𝐸𝑖

𝐵 𝑗

𝐵𝑖

( means ordinary texture synthesis. )

Page 68: Chap. 10 computational photography

2. Find the minimum error boundary cut𝑒=(𝐵1

𝑜𝑣  −𝐵2𝑜𝑣   )2

𝐵1𝑜𝑣 ,𝐵2

𝑜𝑣

( the minimal cost path corresponds to the valley in the surface . )

Page 69: Chap. 10 computational photography

・ Image analogies

[Hertzmann et al. ’01]

𝐴 𝐴 ′ 𝐵 𝐵 ′

?: :¿: :¿

synthesize this image

Page 70: Chap. 10 computational photography

𝐴 𝐴 ′

𝐵 𝐵 ′

unfiltered example NPR-filtered example

target result

Page 71: Chap. 10 computational photography

𝐴 𝐴 ′

𝐵 𝐵 ′

unfiltered example filtered example

target result

Page 72: Chap. 10 computational photography

Thank you!