CSCE 643 Computer Vision:Template Matching, Image Pyramids and Denoising

Jinxiang Chai

Today’s class

• Template matching

• Gaussian Pyramids

• Laplacian Pyramids

• Image denoising

Template matching• Goal: find in image

Template matching• Goal: find in image

• Main challenge: What is a good similarity or distance measure between two patches?

Template matching• Goal: find in image

• Main challenge: What is a good similarity or distance measure between two patches?– Correlation– Zero-mean correlation– Sum Square Difference– Normalized Cross

Correlation

Slide: Hoiem

Matching with filters• Goal: find in image• Method 0: filter the image with eye patch

Input Filtered Image

],[],[],[,

lnkmflkgnmhlk

What went wrong?

f = imageg = filter

Slide: Hoiem

Problem with Correlation

Problem with Correlation

Solution

Slide: Hoiem

Matching with filters• Goal: find in image• Method 1: filter the image with zero-mean eye

Input Filtered Image (scaled) Thresholded Image

],[)],[(],[,

lkgflnkmfnmhlk

True detections

False detections

mean of f

Slide: Hoiem

Matching with filters• Goal: find in image• Method 2: SSD

Input 1- sqrt(SSD) Thresholded Image

2

,

)],[],[(],[ lnkmflkgnmhlk

True detections

Relationship SSD and Correlation

Matching with filters• Goal: find in image• Method 2: SSD

Input 1- sqrt(SSD)

2

,

)],[],[(],[ lnkmflkgnmhlk

What’s the potential downside of SSD?

Slide: Hoiem

Matching with filters• Goal: find in image• Method 3: Normalized cross-correlation - subtracting the mean - dividing by the standard deviation

Matching with filters• Goal: find in image• Method 3: Normalized cross-correlation

5.0

,

2,

,

2

,,

)],[()],[(

)],[)(],[(

],[

lknm

lk

nmlk

flnkmfglkg

flnkmfglkg

nmh

Matlab: normxcorr2(template, im)

mean image patchmean template

Slide: Hoiem

Matching with filters• Goal: find in image• Method 3: Normalized cross-correlation

Input Normalized X-Correlation Thresholded Image

True detections

Matching with filters• Goal: find in image• Method 3: Normalized cross-correlation

Input Normalized X-Correlation Thresholded Image

True detections

Slide: Hoiem

Q: What is the best method to use?

A: Depends• SSD: faster, sensitive to overall intensity• Normalized cross-correlation: slower, invariant

to local average intensity and contrast

Q: What if we want to find larger or smaller eyes?

A: Image Pyramid

Gaussian Pyramid

Low-Pass Filtered ImageImage

GaussianFilter Sample

Low-Res Image

Gaussian Pyramid

filter mask

Repeat– Filter– Subsample

Until minimum resolution reached – can specify desired number of levels (e.g., 3-level pyramid)

Gaussian pyramid

Template Matching with Image Pyramids

Input: Image, Template1. Match template at current scale

2. Downsample image

3. Repeat 1-2 until image is very small

4. Take responses above some threshold, perhaps with non-maxima suppression

Laplacian pyramid

Laplacian FilterThe filtering kernel for Laplacian is

Finite difference allows us to approximate the Laplacian using the following filtering kernel.

Laplacian Pyramid

Laplacian Pyramid (subband images)- Created from Gaussian pyramid by subtraction

The Laplacian of Gaussian Kernel

),()()),(( 22 yxIGyxIG

2

22

24

222222 2

yx

eyx

x

G

x

GG

is the Laplacian operator:

The Laplacian of Gaussian Filter Kernel

0.1

4.1

Laplacian Pyramid

What happens in frequency domain?

Laplacian Pyramid

What happens in frequency domain? Bandpass filters

Can we reconstruct the original from the laplacian pyramid?

Laplacian Pyramid

What happens in frequency domain? Bandpass filters

Can we reconstruct the original from the laplacian pyramid?- Yes, laplacian pyramid is a complete image representation which encodes fine to course structure in a different level

Image representation

• Pixels: great for spatial resolution, poor access to frequency

• Fourier transform: great for frequency, not for spatial info such as locations.

• Pyramids/filter banks: balance between spatial and frequency information

Major uses of image pyramids

• Compression

• Object detection– Scale search– Features

• Detecting stable interest points

• Image blending/mosaicing

• Registration– Course-to-fine

Multi-resolution SIFT Feature Detection

- Object recognition from local scale-invariant features [pdf link], ICCV 09

- David G. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision, 60, 2 (2004), pp. 91-110

Pyramid Blending

laplacianlevel

4

laplacianlevel

2

laplacianlevel

0

left pyramid right pyramid blended pyramid

Hierarchical Image Registration1. Compute Gaussian pyramid2. Align with coarse pyramid3. Successively align with finer

pyramids– Search smaller range

Why is this faster?

- Hierarchical Model-Based Motion Estimation, ECCV 92

Denoising

Additive Gaussian Noise

Gaussian Filter

Gaussian

input Gaussian filter

Median Filter

• For each neighbor in image, sliding the window• Sort pixel values• Set the center pixel to the median

Median Filter

input Gaussian filter Median filter

Median Filter Examples

input Median 7X7

Median Filter Examples

Median 11X11Median 3X3

Median Filter Examples

Median 11X11Median 3X3

Straight edges kept

Sharp features lost

Median Filter Properties

Can remove outliers (peppers and salts)

Window size controls size of structure

Preserve some details but sharp corners and edges might get lost

Comparison of Mean, Gaussian, and Median

original Mean with 6 pixels

Comparison of Mean, Gaussian, and Median

original Gaussian with 6 pixels

Comparison of Mean, Gaussian, and Median

original Median with 6 pixels

Common Problems

Mean: blurs image, removes simple noise, no details are preserved

Common Problems

Mean: blurs image, removes simple noise, no details are preserved

Gaussian: blurs image, preserves details only for small σ.

Common Problems

Mean: blurs image, removes simple noise, no details are preserved

Gaussian: blurs image, preserves details only for small σ.

Median: preserves some details, good at removing strong noise

Common Problems

Mean: blurs image, removes simple noise, no details are preserved

Gaussian: blurs image, preserves details only for small σ.

Median: preserves some details, good at removing strong noise

Can we find a filter that not only smooths regions but preserves sharp features such as edges?

Common ProblemsMean: blurs image, removes simple noise, no details

are preservedGaussian: blurs image, preserves details only for small

σ.Median: preserves some details, good at removing

strong noise

Can we find a filter that not only smooths regions but preserves sharp features such as edges?

- yes, Tomasi's Bilateral Filter!

Bilateral Filter Example

Gaussian filter Bilateral filter

Bilateral Filter Example

Gaussian filter Bilateral filter

1D Graphical ExampleCenter Sample u

It is clear that in weighting this neighborhood, we would like to preserve the step

Neighborhood

I(p)

p

)(uneighborp

The Weights

)2

)(exp()( 2

2

c

c

pupW

p

I(p)

Filtered Values

)2

)(exp()( 2

2

c

c

pupW

p

I(p)

Filtered value

Edges Are Smoothed

)2

)(exp()( 2

2

c

c

pupW

p

I(p)

Filtered value

What Causes the Problem?

)2

)(exp()( 2

2

c

c

pupW

p

I(p)

Filtered value

What Causes the Problem?

)2

)(exp()( 2

2

c

c

pupW

p

I(p)

Filtered value

Same weights for these two pixels!!

The Weights

)2

))()((exp()( 2

2

s

s

pIuIpW

)2

)(exp()( 2

2

c

c

pupW

p

I(p)

Bilateral filter

2

2 2

2

2 2

2 2

2 2

( )

'

c s

c s

I u I pu p

p N u

I u I pu p

p N u

e e I p

I u

e e

Denoise Feature preserving

Normalization

Bilateral FilteringBilateral Filtering

Filter Parameters

As proposed by Tomasi and Manduchi, the filter is controlled by 3 parameters:

The filter can be applied for several iterations in order to further strengthen its edge-preserving smoothing

N(u) – The neighbor size of the filter support,c – The variance of the spatial distances,s – The variance of the value distances,

)(

)(

)(*)(

)(*)(*)()(

uNp sc

uNp sc

pWpW

pIpWpWuI

Bilateral Filter Results

Original

Bilateral Filter Results

σc = 3, σs = 3

Bilateral Filter Results

σc = 6, σs = 3

Bilateral Filter Results

σc = 12, σs = 3

Bilateral Filter Results

σc = 12, σs = 6

Bilateral Filter Results

σc = 15, σs = 8

Summary– Template matching (SSD or Normxcorr2)

• SSD can be done with linear filters, is sensitive to overall intensity

– Gaussian pyramid• Coarse-to-fine search, multi-scale detection

– Laplacian pyramid• More compact image representation• Can be used for compositing in graphics

– Image denoising