ECE 420- Embedded DSP LaboratoryLecture 5 – Image Processing

Thomas MoonFebruary 24, 2020

• Lab4 - pitch detection

• Lab5 - pitch synthesis• Resampling• TD-PSOLA

• Lab6 – image processing

Lab Summary So Far

2

𝑡

1-dimensional signal

𝑢 = 𝑓(𝑡)

𝑥𝑦

𝑢 = 𝑓(𝑥, 𝑦)

2-dimensional signal

à spatial-domain

Digital Image Acquisition

González, R. Callejas and Richard E. Woods. “Digital Image Processing, Third Edition.” (2009).

• Geometric Transformation

• Intensity Transforming• Spatial Filtering

Spatial Domain Process𝑔 𝑥, 𝑦 = 𝑇[𝑓 𝑥, 𝑦 ]

scaling

rotation

translation

original image

𝑥

𝑦

Intensity

255 255 255 255 … 255255255 0 0 0. 0 0 0. 255 0 0. 255 0 0

255

𝑓 𝑥, 𝑦 =

Pixel values can be• binary (0/1)• grayscale (single intensity value, e.g. 8bit intà[0,255]) • color (e.g. 3-channel RGB image)

pixel

• Change the value of pixels. Basic Intensity Transformation

𝑠 = 𝑇(𝑟)

𝑠 = 𝑟 + 𝐶

𝑠, 𝑟 ∈ [0, 𝐿 − 1]

Enhance image

𝐶 = 100 𝐶 = 100

𝑟

𝑠

𝐶

𝐿 − 1

• Change the value of pixels. Basic Intensity Transformation

𝑠 = 𝑇(𝑟)

𝑠 = 𝐿 − 1 − 𝑟

𝑠, 𝑟 ∈ [0, 𝐿 − 1]

𝐿 − 1

𝐿 − 1 𝑟

𝑠

Negative image

• Change the value of pixels. Basic Intensity Transformation

𝑠 = 𝑇(𝑟) 𝑠, 𝑟 ∈ [0, 𝐿 − 1]𝑠

𝐿 − 1

𝐿 − 1 𝑟Intensity-level Slicing

• Histogramà Represents distribution of numerical data

• Each bin denotes a particular outcome

• Number assigned to a bin is the count of observed occurrencesof values for that bin

Intensity Transformation –Histogram Processing

Histogram of 1,000 two Dice Rolls

original image: low-contrast

dark

bright

small dynamic range

low-contrast, small dynamic range

high-contrast, wide dynamic range

Histogram Equalizationàmake your histogram (pdf) uniform distribution

Histogram Equalization

𝑐𝑑𝑓(𝑠)

𝑠𝐿 − 1

1𝑐𝑑𝑓(𝑟)

𝑟𝐿 − 1

1

𝑝𝑑𝑓(𝑟)

𝑟𝐿 − 1

before𝑝𝑑𝑓(𝑠)

𝑠𝐿 − 1

1𝐿 − 1

after

Histogram Equalization

𝑐𝑑𝑓 𝑥 =:;

<

ℎ𝑖𝑠𝑡[𝑘] ℎ𝑖𝑠𝑡[𝑘] = number of pixels with value 𝑘

𝑠 = ℎ 𝑟 = 𝑟𝑜𝑢𝑛𝑑𝑐𝑑𝑓 𝑟 − 𝑐𝑑𝑓BCD𝑀𝑁 − 𝑐𝑑𝑓BCD

(𝐿 − 1)

𝑀,𝑁: width, height of image𝐿: total number of gray levels

𝑐𝑑𝑓BCD: minimum non-zero value of cdf

Spatial Filteringorigin

𝑥

𝑦

spatial domainimage 𝑓

(𝑥H, 𝑦H)

3X3 neighborhood of (𝑥H, 𝑦H)

Spatial filtering is a predefined operation that is performed on the image pixels surrounded by the neighborhood.

Intensity transformation is a special case of spatial filtering with 1X1 neighborhood size.

2D Convolution

𝑔 𝑥, 𝑦 = 𝑓 𝑥, 𝑦 ∗ 𝑤 𝑥, 𝑦 = :C

:K

𝑓 𝑖, 𝑗 𝑤(𝑥 − 𝑖, 𝑦 − 𝑗)

𝑔 𝑡 = 𝑓 𝑡 ∗ 𝑤 𝑡 = :;

𝑓 𝑘 𝑤(𝑡 − 𝑘)

• 𝑤 typically referred to as the filter kernel

• Same idea of a weighted average of elements over a sliding window

𝑤

1D convolution

2D convolution

2D Convolution - Example

0 0 0 0 0

0 0 0 0 0

0 0 1 0 0

0 0 0 0 0

0 0 0 0 0

1 2 3

4 5 6

7 8 9

𝑓 𝑥, 𝑦

𝑤 𝑥, 𝑦

𝑔 𝑥, 𝑦 = 𝑓 𝑥, 𝑦 ∗ 𝑤 𝑥, 𝑦 = :C

:K

𝑓 𝑖, 𝑗 𝑤(𝑥 − 𝑖, 𝑦 − 𝑗)

2D Convolution - Example

0 0 0 0 0

0 0 0 0 0

0 0 1 0 0

0 0 0 0 0

0 0 0 0 0

9 8 7

6 5 4

3 2 1

𝑔 𝑥, 𝑦 = 𝑓 𝑥, 𝑦 ∗ 𝑤 𝑥, 𝑦 = :C

:K

𝑓 𝑖, 𝑗 𝑤(𝑥 − 𝑖, 𝑦 − 𝑗)

0

Assume zero-pad outside

rotated kernel

2D Convolution - Example

0 0 0 0 0

0 0 0 0 0

0 0 1 0 0

0 0 0 0 0

0 0 0 0 0

9 8 7

6 5 4

3 2 1

𝑔 𝑥, 𝑦 = 𝑓 𝑥, 𝑦 ∗ 𝑤 𝑥, 𝑦 = :C

:K

𝑓 𝑖, 𝑗 𝑤(𝑥 − 𝑖, 𝑦 − 𝑗)

0 0

2D Convolution - Example

0 0 0 0 0

0 0 0 0 0

0 0 1 0 0

0 0 0 0 0

0 0 0 0 0

9 8 7

6 5 4

3 2 1

𝑔 𝑥, 𝑦 = 𝑓 𝑥, 𝑦 ∗ 𝑤 𝑥, 𝑦 = :C

:K

𝑓 𝑖, 𝑗 𝑤(𝑥 − 𝑖, 𝑦 − 𝑗)

0 0 0 0 0

0 1

2D Convolution - Example

0 0 0 0 0

0 0 0 0 0

0 0 1 0 0

0 0 0 0 0

0 0 0 0 0

9 8 7

6 5 4

3 2 1

𝑔 𝑥, 𝑦 = 𝑓 𝑥, 𝑦 ∗ 𝑤 𝑥, 𝑦 = :C

:K

𝑓 𝑖, 𝑗 𝑤(𝑥 − 𝑖, 𝑦 − 𝑗)

0 0 0 0 0

0 1 2

2D Convolution - Example

0 0 0 0 0

0 0 0 0 0

0 0 1 0 0

0 0 0 0 0

0 0 0 0 0

9 8 7

6 5 4

3 2 1

𝑔 𝑥, 𝑦 = 𝑓 𝑥, 𝑦 ∗ 𝑤 𝑥, 𝑦 = :C

:K

𝑓 𝑖, 𝑗 𝑤(𝑥 − 𝑖, 𝑦 − 𝑗)

0 0 0 0 0

0 1 2 3 0

0 4 5 6 0

0 7 8 9

2D Convolution - Example

0 0 0 0 0

0 0 0 0 0

0 0 1 0 0

0 0 0 0 0

0 0 0 0 0

9 8 7

6 5 4

3 2 1

𝑔 𝑥, 𝑦 = 𝑓 𝑥, 𝑦 ∗ 𝑤 𝑥, 𝑦 = :C

:K

𝑓 𝑖, 𝑗 𝑤(𝑥 − 𝑖, 𝑦 − 𝑗)

0 0 0 0 0

0 1 2 3 0

0 4 5 6 0

0 7 8 9 0

0 0 0 0 0

2D Correlation - Example

0 0 0 0 0

0 0 0 0 0

0 0 1 0 0

0 0 0 0 0

0 0 0 0 0

1 2 3

4 5 6

7 8 9

𝑔 𝑥, 𝑦 = 𝑓 𝑥, 𝑦 ∗ 𝑤 𝑥, 𝑦 = :C

:K

𝑓 𝑖, 𝑗 𝑤(𝑥 + 𝑖, 𝑦 + 𝑗)

0 0 0 0 0

0 9 8 7 0

0 6 5 4 0

0 3 2 1 0

0 0 0 0 0

2D-correlation

non-rotated kernel

• Different varieties of output sets for 2D convolution

1. Valid – region where w does not go outside image boundary. Output size 𝑁 − 𝐾 + 1.

2. Same – same size as input image, requires handling of elements outside of image. Output size 𝑁.

3. Full – expanded output image by sizeof w, usually assumes zeros outsideimage, useful for ‘overlap-add’ type image block processing. Output size 𝑁 + 𝐾 − 1

Convolution Output Domain

Full

Same

Valid

0 0 0 0 0

0 0 0 0 0

0 0 1 0 0

0 0 0 0 0

0 0 0 0 0

9 8 7

6 5 4

3 2 1

0 0 0 0 0

0 0 0 0 0

0 0 1 0 0

0 0 0 0 0

0 0 0 0 0

9 8 7

6 5 4

3 2 1

0 0 0 0 0

0 0 0 0 0

0 0 1 0 0

0 0 0 0 0

0 0 0 0 0

9 8 7

6 5 4

3 2 1

Same

Valid

Full

Convolution Output Domain

• “Average” the neighborhood pixels• Reduce “sharp” transitions in intensities• Similar to integration• Lowpass filter in frequency domain• Bigger size filter à More blurring

Smoothing Spatial Filters

1 1 1

1 1 1

1 1 1

PQ

X

1 2 1

2 4 2

1 2 1

PPR

X

3x3 averaging filter 3x3 weighted averaging filter 2D-Gaussian filter

original image

Gaussian filter size of 15

Gaussian filter size of 31

• Median filter• Replace the value of a pixel by the median of the

neighborhood intensity• Effective to remove the impulse noise or salt-and-

pepper noise

Smoothing Spatial Filters- Nonlinear

Original image corrupted by salt-and-pepper noise Gaussian filter Median filter

• Highlights transitions in intensity

• Similar to derivative

• Highpass filter in frequency domain

ØFirst-order derivatives – the GradientØSecond-order derivatives – the Laplacian

Sharpening Spatial Filters

3 3 3 2 1 0 0 0 0 3 3 3

0

-1

3

Intensity

𝑥

𝑥

𝑥0

-1

3

-3

𝜕𝑓𝜕𝑥

𝜕T𝑓𝜕𝑥T

ramp

step

Gradient – First Derivative

∇𝑓 ≡𝑔<𝑔W ≡

𝜕𝑓𝜕𝑥𝜕𝑓𝜕𝑦

𝑧P 𝑧T 𝑧Y

𝑧Z 𝑧[ 𝑧R

𝑧\ 𝑧] 𝑧Q

𝑔< = 𝑧\ + 2𝑧] + 𝑧Q − (𝑧P + 2𝑧T + 𝑧Y) 𝑔W = 𝑧Y + 2𝑧R + 𝑧Q − (𝑧P + 2𝑧Z + 𝑧\)

−1 −2 −1

0 0 0

1 2 1

−1 0 1

−2 0 2

−1 0 1

𝑀 𝑥, 𝑦 = 𝑔<T + 𝑔WT

𝑥

𝑦

𝑔<

𝑔W

𝑔<T + 𝑔WT

Laplacian – Second Derivative

∇T𝑓 =𝜕T𝑓𝜕𝑥T

+𝜕T𝑓𝜕𝑦T

𝜕T𝑓𝜕𝑥T

= 𝑓 𝑥 + 1, 𝑦 + 𝑓 𝑥 − 1, 𝑦 − 2𝑓(𝑥, 𝑦)

𝜕T𝑓𝜕𝑦T

= 𝑓 𝑥, 𝑦 + 1 + 𝑓 𝑥, 𝑦 − 1 − 2𝑓(𝑥, 𝑦)

= 𝑓 𝑥 + 1, 𝑦 + 𝑓 𝑥 − 1, 𝑦 + 𝑓 𝑥, 𝑦 + 1 + 𝑓 𝑥, 𝑦 − 1 − 4𝑓(𝑥, 𝑦)

0 −1 0

−1 4 −1

0 −1 0

−1 −1 −1

−1 8 −1

−1 −1 −1

Laplacian filter mask Laplacian filter maskwith diagonal directions

Laplacian:

0 1 0

1 −4 1

0 1 0

Laplacian Laplacian withdiagonal directions

• Groups of 2-3 • Groups of 3 should be larger scope

• Lead-in to the Final Project• Explore a DSP algorithm from the literature

• Implementation in Python for this stage, NOT on tablet yet• Proposal for Assigned Lab

• Overview of proposed algorithm, cite source(s)• Plan for testing and validation• Rough idea(s) for Final Project application

• Assigned Lab Report submission at end of project• Demo incorporated into Final Proposal design review

Reminder: Assigned Lab

• Lab 5: Pitch Synthesizer Quiz/Demo

• Lab 6: Image Processing (Histogram and Filtering)

• Assigned Project Lab Proposals Due 03/09 @2pm

This week