An Interactive Background Blurring Mechanism and Its Applications

NTU CSIE

互動式背景模糊

2

IntroductionPrivacy Preserving

Our systemOriginal image

Depth Map Generation

Pop-up CardPop-up Card

Perspective Box

Perspective Box

Image Defocus

Defocus BlurDefocus Blur

Camera SettingsCamera Settings

Alpha MattingAlpha Matting

Lazy SnappingLazy Snapping

Object Segmentation

Face DetectionFace Detection

Segmentations

Depth Map

Input image Shallow focus result

4

Experimental Result Defocus blur method proposed in the system

comparing with other blur filter results

Deep focus Shallow focus

5

Experimental Result Defocus blur method proposed in the system

comparing with other blur filter results

Gaussian blur Defocus blur Defocus blur + Bokeh

6

Experimental Result

7

Observation with / without depth variation in the

background

Real photograph taken by DSLR

8

Experimental Result with / without depth

variation in the background

Original photograph

Result after post-processing

Depth variation

Depth fixed

12

Applications - Partial Viewing

13

14

Applications - Image Transition at Photo Browser

15

20

END

THANK YOU

21

Method

Segment◦Lazy snapping◦Mean shift◦Alpha matting◦Face detection

Lazy Snapping. ACM Trans. On Graphics 2004.Mean Shift: A Robust Approach Toward Feature Space Analysis. PAMI 2002.A Bayesian Approach to Digital Matting. CVPR 2001.

Depth Map Generation

Pop-up CardPop-up Card

Perspective Box

Perspective Box

Image Defocus

Defocus BlurDefocus Blur

Camera SettingsCamera Settings

Alpha MattingAlpha Matting

Lazy SnappingLazy Snapping

Object Segmentation

Face DetectionFace Detection

Segmentations

Depth MapInput image Shallow focus

result

23

MethodDepth map

◦ Perspective Box Vanish point Rear wall

Tour Into the Picture: Using a spidery mesh user interface to make animation from a single image. SIGGRAPH 1997

Depth Map Generation

Pop-up CardPop-up Card

Perspective Box

Perspective Box

Image Defocus

Defocus BlurDefocus Blur

Camera SettingsCamera Settings

Alpha MattingAlpha Matting

Lazy SnappingLazy Snapping

Object Segmentation

Face DetectionFace Detection

Segmentations

Depth MapInput image Shallow focus

result

25

Method

Image defocus◦Blur circle diameter

Blur circle diameter

Segment Map

Depth Map

Camera setting

Aperture sizeFocal length of the

lensDistance of focus

u v

bd

vbub

u v

bd

vbub

(a) (b)𝑏= 𝑓𝑁ȁ�(𝑢𝑏 − 𝑢)𝑓ȁ�𝑢𝑏(𝑢− 𝑓) 1𝑢+ 1𝑣 = 1𝑓

26

MethodDefocus blur

◦bokeh

𝐼𝑑𝑒𝑓𝑜𝑐𝑢𝑠 ሺ𝑖,𝑗ሻ= σ𝐸𝑛ሺ𝑖,𝑗ሻσ𝑤𝑒𝑖𝑔ℎ𝑡𝑛(𝑖,𝑗) + 𝛼𝑛(𝑖,𝑗)

𝐸𝑛ሺ𝑖,𝑗ሻ= 𝐼ሺ𝑥𝑛,𝑦𝑛ሻ 𝑎𝑟𝑒𝑎 𝑜𝑓 𝑅𝑒𝑔𝑖𝑜𝑛(𝑥𝑛,𝑦𝑛,𝑠,𝑏)Τ

𝑤𝑒𝑖𝑔ℎ𝑡𝑛ሺ𝑖,𝑗ሻ= 1 𝑎𝑟𝑒𝑎 𝑜𝑓 𝑅𝑒𝑔𝑖𝑜𝑛(𝑥𝑛,𝑦𝑛,𝑠,𝑏)Τ

𝛼𝑛(𝑖,𝑗) =൜𝑤𝑒𝑖𝑔ℎ𝑡𝑛(𝑖,𝑗), 𝐼ሺ𝑥𝑛,𝑦𝑛ሻ< 𝑇ℎ𝑏𝑜𝑘𝑒ℎ0, 𝐼ሺ𝑥𝑛,𝑦𝑛ሻ≥ 𝑇ℎ𝑏𝑜𝑘𝑒ℎ

𝑤𝑒𝑖𝑔ℎ𝑡𝑛ሺ𝑖,𝑗ሻ= 1𝑎𝑟𝑒𝑎 𝑜𝑓 𝑅𝑒𝑔𝑖𝑜𝑛ሺ𝑥𝑛,𝑦𝑛,𝑠,𝑏ሻ× 𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛(𝑥𝑛,𝑦𝑛)𝑙𝑢𝑚𝑖𝑛𝑜𝑢𝑠(𝑥𝑛,𝑦𝑛)

……

……E1(i,j)

I(x1,y1) I(x2,y2) I(x3,y3)

I(xn,yn) I(xn+1,yn+1)

E2(i,j) E3(i,j)

En(i,j) En+1(i,j)

En+2(i,j)

I(xn+2,yn+2)

27

MethodDefocus blur

……

……E1(i,j)

I(x1,y1) I(x2,y2) I(x3,y3)

I(xn,yn) I(xn+1,yn+1)

E2(i,j) E3(i,j)

En(i,j) En+1(i,j)

En+2(i,j)

I(xn+2,yn+2)

28

Method

◦Near-by object case

Focus on the flower

Focus on the flower

29

◦Near-by object case

Without inpainting

Interpolation result

Experimental Result

30

◦Near-by object case

Method Blur circle diameter

Objects in front of the

focus

Objects behind

the focus

Defocus Blur

Text synthesis inpainting

Blur shape as alpha map

Defocus Blur

Interpolation by alpha map

Shallow focus image

31

Q&A

32

END

THANK YOU

33

Introduction Depth of field

Real light

Circle of Confusion- Readability range

Focus plane Photo planeOut of focus – blur range

Depth of field

34

Introduction

Real world

Photo plane

Shallow focus

35

IntroductionShallow focus

◦Highlight the subject by softening background diffusion

Deep focus by DC

NIKON E4300 (2003)

Shallow focus by DSLR

NIKON D90(2008)

36

Outline IntroductionMethod

◦Object Segmentation◦Depth Map Generation◦Image Defocus

Experimental ResultApplicationsConclusion