CSCE 641 Computer Graphics: Image Filtering & Feature Detection Jinxiang Chai

CSCE 641 Computer Graphics:Image Filtering & Feature Detection

Jinxiang Chai

Review

2D Fourier Transform

Nyquist sampling theory - Sampling Rate ≥ 2 * max frequency in the signal

Antialiasing - prefiltering

Gaussian pyramid - Mipmap

Review

2D Fourier Transform

Nyquist sampling theory - Sampling Rate ≥ 2 * max frequency in the signal

Antialiasing - prefiltering

Gaussian pyramid - Mipmap

Review: 2D Fourier Transform

Fourier Transform:

Inverse Fourier transform:

dvduevuFyxf vyuxi )(2),(),(

dydxeyxfvuF vyuxi )(2),(),(

Review: Nyquist Sampling Rate

T 2T…-2T -T… 0

x

fs(x)

f(x)

x

Fs(u)

u-fmax fmax

F(u)

u-fmax fmax

)()( uboxuFs

Inverse Fourier transform

Fourier transform

Sampling Rate ≥ 2 * max frequency in the signal!

Review: Antialiasing

Increase the sampling rate to above twice the highest frequency

Introduce an anti-aliasing filter to reduce fmax

u-fmax fmax

Fs(u)

u-fmax fmax

Review: Antialiasing

Increase the sampling rate to above twice the highest frequency

Introduce an anti-aliasing filter to reduce fmax

u-fmax fmax

Fs(u)

u-fmax fmax

- Sinc filter

- Mean filter

- Triangular filter

- Gaussian filter

Known as a Gaussian Pyramid– MipMap (Williams, 1983)

Review: Image Pyramids

Outline

Image Processing

- Median filtering

- Bilateral filtering

- Edge detection

- Corner detection

Filtering

input Gaussian filter

Median Filter

• For each neighbor in image, sliding the window

• Sort pixel values• Set the center pixel to the median

Median Filter

input Gaussian filter Median filter

Median Filter Examples

input Median 7X7


Median 11X11Median 3X3


Median 11X11Median 3X3

Straight edges kept

Sharp features lost

Median Filter Properties

Can remove outliers (peppers and salts)

Window size controls size of structure

Preserve some details but sharp corners and edges might get lost

Comparison of Mean, Gaussian, and Median

original Mean with 6 pixels


original Gaussian with 6 pixels


original Median with 6 pixels

Common Problems

Mean: blurs image, removes simple noise, no details are preserved

Common Problems


Gaussian: blurs image, preserves details only for small σ.

Common Problems



Median: preserves some details, good at removing strong noise

Common Problems




Can we find a filter that not only smooths regions but preserves sharp features such as edges?

Common Problems




Can we find a filter that not only smooths regions but preserves sharp features such as edges?

- yes, bilateral filter

Outline

Image Processing

- Median filtering


- Edge detection

- Corner detection

What Is Bilateral Filter?

Bilateral

- Affecting or undertaken by two sides equally

Property:

- Convolution filter

- Smooth image but preserve edges

- Operates in the domain and the range of image

Bilateral Filter Example

Gaussian filter Bilateral filter

Bilateral Filter Example

Gaussian filter Bilateral filter

1D Graphical ExampleCenter Sample u

It is clear that in weighting this neighborhood, we would like to preserve the step

Neighborhood

I(p)

p

)(uneighborp

The Weights

)2

)(exp()( 2

2

c

c

pupW

p

I(p)

Filtered Values

)2

)(exp()( 2

2

c

c

pupW

p

I(p)

Filtered value

Edges Are Smoothed

)2

)(exp()( 2

2

c

c

pupW

p

I(p)

Filtered value

What Causes the Problem?

)2

)(exp()( 2

2

c

c

pupW

p

I(p)

Filtered value

What Causes the Problem?

)2

)(exp()( 2

2

c

c

pupW

p

I(p)

Filtered value

Same weights for these two pixels!!

The Weights

)2

))()((exp()( 2

2

s

s

pIuIpW

)2

)(exp()( 2

2

c

c

pupW

p

I(p)

Bilateral filter

2

2 2

2

2 2

2 2

2 2

( )

'

c s

c s

I u I pu p

p N u

I u I pu p

p N u

e e I p

I u

e e

Denoise Feature preserving

Normalization

Bilateral FilteringBilateral Filtering

Kernel Properties

• Per each sample, we can define a ‘Kernel’ that averages its neighborhood

• This kernel changes from sample to sample!• The sum of the kernel entries is 1 due to the

normalization,• The center entry in the kernel is the largest,• Subject to the above, the kernel can take any

form (as opposed to filters which are monotonically decreasing).

p sc

p sc

pWpW

pIpWpWuI

)(*)(

)(*)(*)()(

Filter Parameters

As proposed by Tomasi and Manduchi, the filter is controlled by 3 parameters:

The filter can be applied for several iterations in order to further strengthen its edge-preserving smoothing

N(u) – The neighbor size of the filter support,c – The variance of the spatial distances,s – The variance of the value distances,

)(

)(

)(*)(

)(*)(*)()(

uNp sc

uNp sc

pWpW

pIpWpWuI

Bilateral Filter

input

Bilateral Filter

input

Bilateral Filter

input

Wc

Bilateral Filter

input

Wc Ws

Bilateral Filter

input

Wc Ws Ws*Wc

Bilateral Filter

input

Wc Ws Ws*Wc

Output

Bilateral Filter Results

Original


σc = 3, σs = 3


σc = 6, σs = 3


σc = 12, σs = 3


σc = 12, σs = 6


σc = 15, σs = 8

Additional Comments

The bilateral filter is a powerful filter: - Can work with any reasonable distance function

Ws and Wc,

- Easily extended to higher dimension signals, e.g. Images, video, mesh, animation data etc.

- Easily extended to vectored-signals, e.g. RGB images or RGBD images etc.

[Fleishman et al, siggraph 03]Bilateral Mesh Denoising

Readings on Image Filtering

• Section 3.3.1 in Szeliski book

• Tomasi's Bilateral Filter

• more about bilateral filter, click here

http://people.csail.mit.edu/sparis/bf_course/

Outline

Image Processing

- Median filtering


- Edge detection

- Corner detection

Edge Detection

What are edges in this image?

Edge Detection

What are edges in this image?

Edge Detection

One of the most important image processing problems

- really easy for human - really difficult for computers - fundamental for computer vision - very important for many graphics applications

How to tell a pixel is on the edge?

Edge Detection




Edge Detection




Edge Detection



How to tell a pixel is on the edge?- Related to the first-derivative.

- How to compute the first-derivatives of an image?

Gradient

A vector variable - Direction of the maximum growth of the function

- Magnitude of the growth

- Perpendicular to the edge direction

2

)/

/tan(

),(

),(

22

xf

yfa

y

f

x

fyxf

yy

fx

x

fyxf

),( yxf

Gradients

How to Calculate Gradient?

How to compute ?x

f

Intensity

Pixel column

The Good ole’ Taylor Series

Subtracting the second from the first we obtain

...)(''2

1)(')()( 2 xfhxhfxfhxf

...)(''2

1)(')()( 2 xfhxhfxfhxf

or…

)(2

)()()(' 2hO

h

hxfhxfxf

Discrete Gradient Estimation

For discrete functions, we can use the first order approximation of the gradient

where h corresponds to the step size

h

hxfhxfxf

2

)()()('


For discrete functions, we can use the first order approximation of the gradient

where h corresponds to the step size

For our purposes, h corresponds to the width of 1 pixel =>

h

hxfhxfxf

2

)()()('

2

)1,()1,(),(2

),1(),1(),(

yxIyxI

y

yxI

yxIyxI

x

yxI


So how can we compute the image gradient efficiently?

- Using our good old friend convolution!

1

0

1

*

101*

Iy

I

Ix

I

- Dropped off the “divide by 2” for speed considerations.- This only scales the gradient.

Taking the Discrete Derivative

] 1 0 1[

abs()

Basic Edge Detection Step 1

INPUT IMAGE

1) EdgeEnhancement

Horizontal [-1 0 1]

Vertical [-1 0 1]T

),( yxI

x

yxI

),(

y

yxI

),(

But isn’t edge detection susceptible to noise?

Basic Edge Detection Steps 1-2

INPUT IMAGE

2) EdgeEnhancement

Horizontal [-1 0 1]

Vertical [-1 0 1]T

),( yxI

x

yxI

),(

y

yxI

),(

1) NoiseSmoothing

16/

121

242

121

Discrete Gradient EstimationRemember that the gradient is a vector and we have calculated the coefficients in the x and y directions at each point in the image

After convolving, we get the magnitude of the gradient from at each point (pixel) from

In practice, we often sum the absolute values of the components for computational efficiency

22

),(

y

I

x

IyxG

Basic Edge Detection (cont’d)

INPUT IMAGE

1) NoiseSmoothing

2) EdgeEnhancement

Horizontal [-1 0 1]

Vertical [-1 0 1]T

),( yxI

x

yxI

),(

y

yxI

),(

2

122 ),(),(

|),(|

y

yxI

x

yxIyxI

“GRADIENT” IMAGE

16/

121

242

121


INPUT IMAGE

1) NoiseSmoothing

2) EdgeEnhancement

Horizontal [-1 0 1]

Vertical [-1 0 1]T

),( yxI

x

yxI

),(

y

yxI

),(

2

122 ),(),(

|),(|

y

yxI

x

yxIyxI


16/

121

242

121

What does the gradient image mean?


INPUT IMAGE

1) NoiseSmoothing

2) EdgeEnhancement

Horizontal [-1 0 1]

Vertical [-1 0 1]T

),( yxI

x

yxI

),(

y

yxI

),(

2

122 ),(),(

|),(|

y

yxI

x

yxIyxI


16/

121

242

121

What does the gradient image mean?

- Magnitude of intensity changes around each pixel

Discrete Gradient EstimationSo how do we segment the edges from the rest of the scene?

- Thresholding!

The Effects of ThresholdingWhile edge features are independent of illumination, the edge strength is not!

Results from threshold values of 50 and 100

Basic Edge Detection Summary

INPUT IMAGE

1) NoiseSmoothing

EDGE IMAGE

2) EdgeEnhancement

Horizontal [-1 0 1]

Vertical [-1 0 1]T

),( yxI

x

yxI

),(

y

yxI

),(

2

122 ),(),(

|),(|

y

yxI

x

yxIyxI


3)Threshold

16/

121

242

121

The effects of Filtering Noise

Threshold20

Gaussian SmoothingUnsmoothed Edges

Threshold50

Sobel Edge DetectionOne of the dominant edge detection schemes uses the Sobel

operators

Convolving each of these with the original image generates horizontal and vertical gradient images that are combined as before

101

202

101

VSobel

121

000

121

HSobel

Sobel Edge Detection

One of the dominant edge detection schemes uses the Sobel operators

101

202

101

VSobel

121

000

121

HSobel

- Can be approximated as a derivative of Gaussian

- First Gaussian smoothing and then compute derivatives

Ix

G

x

IG

)(



101

202

101

VSobel

121

000

121

HSobel



Ix

G

x

IG

)( Why?



101

202

101

VSobel

121

000

121

HSobel



- In practice we may still need to smooth for noise

Ix

G

x

IG

)(

Robert and Prewitt Edge Detectors

The Prewitt is similar to the Sobel, but uses a different kernel

Roberts was an early edge detector kernel

101

101

101

VP

111

000

111

HP

11

111R

11

112R

Sobel Edge Detector

Java Applet

http://www.stevens-tech.edu/wireless/klin/EdgeDetection/EdgeDetectionInfo.htm

Second Derivative Edge Detector

Sobel Operator can produce thick edges; ideally we are looking for infinitely thin boundaries



An alternative approach is to look for local extrema in the first derivative

A peak in the first derivative corresponds what in the second derivative?



An alternative approach is to look for local extrema in the first derivative

A peak in the first derivative corresponds what in the second derivative?

Localization with Laplacian

An equivalent measure of the second derivative in 2D is the Laplacian:

Numerically, we approximate the Laplacian using the following filtering kernel:

Zeros crossing of the filter corresponds to positions of maximum gradient.

- can be used for detecting edges - might be sensitive to noise - need to filter the image

The Laplacian of Gaussian Kernel

),()()),(( 22 yxIGyxIG

Review: Properties

Linearity:

Time shift:

Derivative:

Integration:

Convolution:

)()()()( ubGuaFxbgxaf

)()( 020 uFexxf uxi

)()(

uuFx

xdf

u

uFdxxf

)()(

)()()()( uGuFxgxf

)()()()( uGuFxgxf

The Laplacian of Gaussian Kernel

),()()),(( 22 yxIGyxIG

2

22

24

222222 2

yx

eyx

x

G

x

GG

The Laplacian of Gaussian Filter Kernel

0.1

4.1

Laplacian Edge Detector

Java applet: Click here

http://euclid.ii.metu.edu.tr/~ion528/demo/lectures/6/3/

Readings

• Section 4.2 in Szeliski book

Outline

Image Processing

- Median filtering


- Edge detection

- Corner detection

Corner Detection

Why do we need to detect corners?

- Corners correspond to points in both the world and image spaces

- Very important for image based rendering and modeling

- Very useful in computer vision

What are Corners?

Point features

What are Corners?

Point features

Where two edges come together

Where the image gradient has significant components in the x and y direction

We will establish corners from the gradient rather than the edge images

Basic Ideas

What are gradients along x and y directions?

Basic Ideas


Basic Ideas


How to measure corners based on the gradient images?

Basic Ideas


How to measure corners based on the gradient images?How to measure corners based on the gradient images?

- two major axes in the local window!

Basic Idea

• Corners: two major axes of gradients in the local window

• How to evaluate this?

Basic Idea

• Corners: two major axes of gradients in the local window

• How to evaluate this?

y

x

I

IGradient vector

Basic Idea

Ix

Iy

Basic Idea

Ix

Iy

Ix

Iy

Ix

Iy

Finding Two Major Axes

• Compute covariance matrix

• Compute two eigen values: λ1 and λ2, which indicate magnitude of two dominant axes

• If both λ1 and λ2 are large, then we have two dominant axes (corners).

1. Compute the image gradients

2. Define a neighborhood size as an area of interest around each pixel

Corner Detection Algorithm

6160531918

5855531513

5555501313

1010101111

1012121110

y

yxII

x

yxII yx

),(,

),(

3x3 neighborhood

3. For each image pixel (i,j), construct the following matrix from it and its neighborhood values

e.g.

Corner Detection Algorithm (cont’d)

6160531918

5855531513

5555501313

1010101111

1012121110

xI

2

2

),(yyx

yxx

T

y

x

y

xji III

III

I

I

I

IC

22222

2222)3,3(

5553155550

13101011]1,1[

C

Similar to covariance matrix (Ix,Iy)T!


4. For each matrix C(i,j), determine the 2 eigenvalues λ(i.j)= [λ1, λ2].

- Eigenvalues indicate magnitude of two dominant axes

- If either λ1 or λ2 is close to zero, then this is not a corner.

Simple case:



Simple case:

Isolated pixelsInterior Region Edge Corner

λ1, λ2=0 Large λ1 and small λ2 Large λ1 and large λ2 small λ1 and small λ2



- This is just a rotated version of the one on last slide

- If either λ1 or λ2 is close to zero, then this is not a corner.

General case:

Eigen-values and Corner

- λ1 is large

- λ2 is large


- λ1 is large

- λ2 is small


- λ1 is small

- λ2 is small



5. If both λ1 and λ2 are big, we have a corner (Harris also checks the ratio of λs is not too high)

ISSUE: The corners obtained will be a function of the threshold !

Image Gradients

Image Gradients

Closeup of image orientation at each pixel

The Orientation Field

Corners are detected where both λ1 and λ2 are big

The Orientation Field

Corners are detected where both λ1 and λ2 are big

Corner Detection Sample Results

Threshold=25,000 Threshold=10,000

Threshold=5,000

Results

Required Readings

• Section 4.1.1 in Szeliski book

• Harris corner detector

Homework #1

• Due by 9/23, midnight

• Submit it via csnet.

• If you do not have the csnet account, submit it to me via email.

Summary

Image Processing - Median filtering


- Edge detection

- Corner detection

Next Lecture

SIFT features

Image composition and blending

Image warping

Image morphing

Documents

CSCE 641 Computer Graphics: Image Filtering & Feature Detection Jinxiang Chai