Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge...

Edge Preserving Filtering

Median Filter

Bilateral Filter

Shai Avidan

Tel-Aviv University

Slide Credits� (partial list)

• Rick Szeliski

• Steve Seitz

• Alyosha Efros

• Yacov Hel-Or

• Marc Levoy

• Bill Freeman

• Fredo Durand

• Sylvain Paris

A Gentle Introduction

to Bilateral Filtering

and its Applications

“Fixing the Gaussian Blur”:

the Bilateral Filter

Sylvain Paris – MIT CSAIL

Box Average

average

input

square neighborhood

output

normalized

box function

sum over

all pixels q

intensity at

pixel qresult at

pixel p

Equation of Box Average

��

��S

IBIBAq

qp qp )(][ �

0

Square Box Generates Defects

• Axis-aligned streaks

• Blocky results

input

output

unrelated

pixels

unrelated

pixels

related

pixels

Box Profile

pixel

position

pixel

weight

Strategy to Solve these

Problems

• Use an isotropic (i.e. circular) window.

• Use a window with a smooth falloff.

box window Gaussian window

Gaussian Blur

average

input

per-pixel multiplication

output*

input

box average

Gaussian blur

normalized

Gaussian function

Equation of Gaussian Blur

� ���

��S

IGIGBq

qp qp ||||][ �

Same idea: weighted average of pixels.

0

1

unrelated

pixels

unrelated

pixels

uncertain

pixels

uncertain

pixels

related

pixels

Gaussian Profile

pixel

position

pixel

weight��

���

��

2

2

2exp

2

1)(

����

xxG

size of the window

Spatial Parameter

� ���

��S

IGIGBq

qp qp ||||][ �

small � large �

input

limited smoothing strong smoothing

Blur Comes from

Averaging across Edges

*

*

*

input output

Same Gaussian kernel everywhere.

Bilateral Filter

No Averaging across Edges

*

*

*

input output

The kernel shape depends on the image content.

[Aurich 95, Smith 97, Tomasi 98]

space weight

not new

range weight

I

new

normalization

factor

new

Bilateral Filter Definition:

an Additional Edge Term

� � � ���

���S

IIIGGW

IBFq

qqp

p

p qp ||||||1

][rs ��

Same idea: weighted average of pixels.

Illustration a 1D Image

• 1D image = line of pixels

• Better visualized as a plot

pixel

intensity

pixel position

space

Gaussian Blur and Bilateral Filter

space rangenormalization

Gaussian blur

� � � ���

���S

IIIGGW

IBFq

qqp

p

p qp ||||||1

][rs ��

Bilateral filter[Aurich 95, Smith 97, Tomasi 98]

space

space

range

p

p

q

q

� ���

��S

IGIGBq

qp qp ||||][ �

Space and Range Parameters

• space �s : spatial extent of the kernel, size of

the considered neighborhood.

• range �r : “minimum” amplitude of an edge

� � � ���

���S

IIIGGW

IBFq

qqp

p

p qp ||||||1

][rs ��

How to Set the Parameters

Depends on the application. For instance:

• space parameter: proportional to image size

– e.g., 2% of image diagonal

• range parameter: proportional to edge

amplitude

– e.g., mean or median of image gradients

• independent of resolution and exposure

A Few

More Advanced

Remarks

Bilateral Filter Crosses Thin Lines• Bilateral filter averages across

features thinner than ~2�s

• Desirable for smoothing: more pixels = more robust

• Different from diffusion that stops at thin lines

close-up kernel

Iterating the Bilateral Filter

• Generate more piecewise-flat images

• Often not needed in computational

photo.

][ )()1( nn IBFI ��

input

1 iteration

2 iterations

4 iterations

Bilateral Filtering Color Images

� � � ���

���S

IIIGGW

IBFq

qqp

p

p qp ||||||1

][rs ��

� � � ���

���S

GGW

IBFq

qqp

p

p CCCqp ||||||||1

][rs ��

For gray-level images

For color images

intensity difference

color difference

The bilateral filter isThe bilateral filter is

extremely easy to adapt to your need.extremely easy to adapt to your need.

scalar

3D vector

(RGB, Lab)

input

output

Applications

• Image denoising

• HDR Compression

– Key idea:

– Break image into base and detail layers

– Compress base

– Recompose image

Fast Implementation: 3D Kernel

• Idea: represent image data such that the weights

depend only on the distance between points

[Paris and Durand 06]

pixel

intensity

pixel position

1D image

Plot

I = f ( x )

far in range

close in space

1st Step: Re-arranging

Symbols� � � �

� � � ��

�

�

�

���

���

S

S

IIGGW

IIIGGW

IBF

q

qpp

q

qqp

p

p

qp

qp

||||||

||||||1

][

rs

rs

��

��

� � � �

� � � � 1||||||

||||||][

rs

rs

�

�

�

�

���

���

S

S

IIGGW

IIIGGIBFW

q

qpp

q

qqppp

qp

qp

��

��

Multiply first equation by Wp

1st Step: Summary

• Similar equations

• No normalization factor anymore

• Don’t forget to divide at the end

� � � �

� � � � 1||||||

||||||][

rs

rs

�

�

�

�

���

���

S

S

IIGGW

IIIGGIBFW

q

qpp

q

qqppp

qp

qp

��

��

2nd Step: Higher-dimensional Space

pp

space

range

• “Product of two Gaussians” = higher dim.

Gaussian

2nd Step: Higher-dimensional Space

pp

space

range

• 0 almost everywhere, I at “plot location”

2nd Step: Higher-dimensional Space

pp

• 0 almost everywhere, I at “plot location”

• Weighted average at each point = Gaussian blur

2nd Step: Higher-dimensional Space

pp

• 0 almost everywhere, I at “plot location”

• Weighted average at each point = Gaussian blur

• Result is at “plot location”

�������������� �������

������� ��

������

����

������

�������

���������

������

������

�������

���������

������

New num. scheme:

• simple operations

• complex space

�������������� �������

���������� ����

������

����

�����������������

������������

����� �

������ ��

����� �

������ ��

Strategy:

downsampled

convolution

Conceptual view,

not exactly

the actual algorithm

The Algorithm

NEW IDEA : ‘Joint’ or ‘Cross’

Bilateral’ Petschnigg(2004) and

Eisemann(2004)Bilateral � two kinds of weights

NEW : get them from two kinds of images.

• Smooth image A pixels locally, but

• Limit to ‘similar regions’ of image B

Why do this? To get ‘best of both

images’

Ordinary Bilateral Filter

Bilateral � two kinds of weights, one image A :

� � � ���

���S

AAAGGW

ABFq

qqp

p

p qp ||||||1

][rs ��

cc

ss

Image A:

������������

��������

f(x)f(x)xx

‘Joint’ or ‘Cross’ Bilateral Filter

NEW: two kinds of weights, two images

� � � ���

���S

ABBGGW

ABFq

qqp

p

p qp ||||||1

][rs ��

cc

ss

A: Noisy, dim(ambient image)

cc

ss

B: Clean,strong

(Flash image)

Image A: Warm, shadows, but too Noisy(too dim for a good quick photo)(too dim for a good quick photo)

No-flash

Image B: Cold, Shadow-free, Clean(flash: simple light, ALMOST no shadows)(flash: simple light, ALMOST no shadows)

MERGE BEST OF BOTH: apply

‘Cross Bilateral’ or ‘Joint Bilateral’

(it really is much better!)

Video Enhancement Using

Per Pixel Exposures (Bennett, 06)

From this video:

ASTA: Adaptive

SSpatio-

TTemporal

Accumulation Filter

ASTA

Replace pixel difference with a general dissimilarity measure

D(x,x)=0, D(x,y)=D(y,x)

Shot noise

A new dissimilarity measure

Instead of comparing pixel intensities, look at their local

spatial neighborhood

• Raw Video Frame:

(from FIFO center)

• Histogram stretching;

(estimate gain for

each pixel)

• ‘Mostly Temporal’ Bilateral Filter:

– Average recent similar values,

– Reject outliers (avoids ‘ghosting’), spatial avg as needed

– Tone Mapping

The Process for One Frame

The Process for One Frame

• Raw Video Frame:

(from FIFO center)

• Histogram stretching;

(estimate gain for

each pixel)

• ‘Mostly Temporal’ Bilateral Filter:

– Average recent similar values,

– Reject outliers (avoids ‘ghosting’), spatial avg as needed

– Tone Mapping

The Process for One Frame

• Raw Video Frame:

(from FIFO center)

• Histogram stretching;

(estimate gain for

each pixel)

• ‘Mostly Temporal’ Bilateral Filter:

– Average recent similar values,

– Reject outliers (avoids ‘ghosting’), spatial avg as needed

– Tone Mapping

(color: # avg’ pixels)

The Process for One Frame

• Raw Video Frame:

(from FIFO center)

• Histogram stretching;

(estimate gain for

each pixel)

• ‘Mostly Temporal’ Bilateral Filter:

– Average recent similar values,

– Reject outliers (avoids ‘ghosting’), spatial avg as needed

– Tone Mapping

Bilateral Filter Variant: Mostly Temporal

• FIFO for Histogram-stretched video

– Carry gain estimate for each pixel;

– Use future as well as previous values;

• Expanded Bilateral Filter Methods:

– Static scene? Temporal-only avg. works well

– Motion? Bilateral rejects outliers: no ghosts!

• Generalize: ‘Dissimilarity’ (not just || Ip – Iq ||2)

• Voting: spatial filter de-noises motion

Multispectral Bilateral Video Fusion

(Bennett,07)

• Result:

– Produces watchable result from unwatchable input

–– VERYVERY robust; accepts almost any dark video;

– Exploits temporal coherence to emulate

Low-light HDR video, without special equipment