Introduction Image Enhancement and Restoration perspectives. …elec4245/sp17/lec5-2x2.pdf · 2017-05-04 · Image Restoration : Images are bad, but we want to make them look good

Image Enhancement and Restoration

Dr. Edmund Lam

Department of Electrical and Electronic EngineeringThe University of Hong Kong

ELEC4245: Digital Image Proessing(Second Semester, 2016–17)

http://www.eee.hku.hk/˜elec4245

E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2017 1 / 60

Introduction

We have been building up techniques in handling digital images: frompoint operation to filtering, with spatial and spectral domainperspectives.

We can change our viewpoints to consider the purposes of processingdigital images. Often, the digital image data are not ideal. Broadlyspeaking, we can have these three objectives:

Image Enhancement: Images may be so-so, but we want to makethem better.Image Restoration: Images are bad, but we want to make themlook good.Image Reconstruction: Captured data are not yet images, but wewant to form images from the data.

In some cases, the difference among them is not important.E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2017 2 / 60

Image enhancement

1 Image enhancement

2 Image restoration

3 Image restoration: frequency domain methods

4 Image restoration: spatial domain methods


Image enhancement

Image enhancement

Some of the techniques we studied are naturally for imageenhancement, e.g.,

Contrast enhancement (using intensity transformation)Histogram equalizationNoise suppression, also known as denoising (e.g., using spatialfiltering)Edge sharpening (using edge detection, then added to originalimage)

There can be other more “application specific” techniques, which wewould not discuss in details in this class, e.g.,

Red eye removalTransformation from grayscale to pseudo-color


Image enhancement

Image enhancement

Some common characteristics about image enhancement:

There is no new information; just different ways of presenting orvisualizing the information. In fact, some information may be lostalong the process.Often, the enhancement is for a specific purpose, and the techniqueseeks to accentuate certain features of the digital imageaccordinglyThe objective is usually subjective and difficult to quantify, andtherefore the techniques are often empirical and parametric /interactive (i.e., the user can tune parameters and providefeedback)


Image restoration

1 Image enhancement

2 Image restoration




Image restoration

Image restoration

The standard question in image restoration: Can we turn a blurry imageinto a sharp one?

−→

input output


Image restoration

Image restoration

Unlike image enhancement, image restoration can have a definitive,quantifiable goal. For example, this example image is out-of-focus; ourobjective is to compute the output which would be closest to what we can getif the image is in focus.

1 The model: why do we have out-of-focus images?2 The mathematics: how do we represent the imaging process

mathematically?3 The metric: how do we measure our “success” in restoring the

image?4 The method(s): how do we implement the algorithm(s)?


Image restoration

Image restoration: model

[imaging system][object] [image]

reality

model

+h(x, y)g(x, y)

n(x, y)

i(x, y)


Image restoration

Image restoration: model

We have added a noise term at the sensor.

Imaging is characterized by the following equation:

i(x, y) = g(x, y) ∗ h(x, y) + n(x, y), (1)

with the corresponding representation in the frequency domain:

I( fx, fy) = G( fx, fy)H( fx, fy) + N( fx, fy). (2)


Image restoration

Image restoration: out-of-focus model

Image of a point source is a circle.Radius: (simple geometry)

r0 =zi − za

zir (3)

r0

za

zi

r


Image restoration

Image restoration: out-of-focus model

For ease of derivations below, let r0 = 0.5. Therefore,

hr(r) =

{1 |r| < 1

20 otherwise.

(4)

Note the similarity with Π(x). However, hr(r) is really a 2D function!We may represent this hr(r) as Π(r) and call this the “circ”function, where the context would need to make clear whether thisis “rect” or “circ”.Alternatively, we can write circ (r) for this function, and userect (x) for the 1D function.To understand the effect of the “circ” function, we need to developthe mathematics of Fourier transform of circularly symmetricfunctions.


Image restoration

Image restoration: mathematics

Rectangular coordinates are not the only possibility for 2D. We oftenhave functions that are separable in polar coordinates:

h(r, θ) = hr(r)hθ(θ) (5)

The mathematics is a lot more involved!

However, if the function is circularly symmetric, i.e.,

h(r, θ) = hr(r), (6)

then a lot of simplification can be done.


Image restoration

Fourier-Bessel transform

We begin with the 2D continuous-time Fourier transform:

H( fx, fy) =

∞"

−∞h(x, y) e− j2π( fxx+ fy y) dx dy (7)

Apply these transformations from Cartesian to polar coordinates:

r =√

x2 + y2 θ = arctan( y

x

)(8)

x = r cosθ y = r sinθ (9)

ρ =√

f 2x + f 2

y φ = arctan(

fy

fx

)(10)

fx = ρ cosφ fy = ρ sinφ (11)

y

x

r

θ

fy

fx

ρ

φ


Image restoration


We assume a circularly symmetric function hr(r), and call theexpression in polar coordinates H0(ρ, φ):

H( fx, fy) =

∞"

−∞h(x, y)e− j2π( fxx+ fy y) dx dy

H0(ρ, φ) =

∫ 2π

0

∫ ∞

0hr(r)e− j2πrρ(cosθ cosφ+sinθ sinφ)r dr dθ (12)

=

∫ ∞

0

(∫ 2π

0e− j2πrρ cos(θ−φ) dθ

)rhr(r) dr (13)

The expression still looks complicated, but can in fact be simplified.


Image restoration


There is a “Bessel function” (zero order) defined as

J0(a) =1

2π

∫ 2π

0e− ja cos(θ−φ) dθ. (14)

This expression is independent of φ!We can pre-compute it; think about how we handle cos x in 1D.There is also a Bessel function (first order) with the relationship

∫ x

0µJ0(µ) dµ = xJ1(x) (15)

We can substitute zero-order Bessel function to Eq. (13) and obtain

H0(ρ) = 2π∫ ∞

0rhr(r)J0(2πrρ) dr.


Image restoration


We can derive the inverse transform accordingly, so that

H0(ρ) = 2π∫ ∞

0rhr(r)J0(2πrρ) dr (16)

hr(r) = 2π∫ ∞

0ρH0(ρ)J0(2πrρ) dρ. (17)

Fundamentally, this is 2D continuous-time Fourier transform.Specifically, we are relating the cross-section of circularlysymmetric functions and their 2D Fourier transform.So we can consider it a new transform relating two 1D functions:we call this the Fourier-Bessel transform, which is also known as theHankel transform of zero order.


Image restoration

Image restoration: out-of-focus mathematics

We can put in Π(r) to the Fourier-Bessel transform in Eq. (16) to obtain

H0(ρ) = 2π∫ ∞

0rΠ(r)J0(2πrρ) dr

= 2π∫ 0.5

0r(1)J0(2πrρ) dr

Let r′ = 2πrρ. We also note the identity∫ x

0 µJ0(µ) dµ = xJ1(x) to obtain

H0(ρ) =

∫ πρ

0

r′

ρJ0(r′)

dr′

2πρ=

12πρ2πρJ1(πρ) =

J1(πρ)2ρ

This is so useful we call this a “jinc” function:

H0(ρ) =J1(πρ)

2ρ, jinc (ρ) (18)


Image restoration


Π(r) and jincρ is a Fourier-Bessel transform pair:

r

Π(r)

ρ

jincρsincρ (comparison)


Image restoration


Underlying 2D Fourier transform (red = positive, green = negative):

Π(r) jinc (ρ)


Image restoration


We can gain some insights about out-of-focus images:Defocus =⇒ image frequency weighted by jinc (ρ) =⇒ attenuationof high frequencyLoss of high frequency =⇒ sharp edges become blurredThis is exactly what we see in out-of-focus images!More severe defocus =⇒ bigger (zi − za) =⇒ bigger r0

Spatial domain expansion =⇒ frequency domain compression =⇒further attenuation of high frequency =⇒ image more blurred!The mathematics allows us to quantify the effects of defocusing.When jinc (ρ) = 0, that frequency component totally disappears. Ithappens when ρ = ±1.220,±2.233, . . .For 1.220 < ρ < 2.233, we have phase reversal because jinc (ρ) < 0


Image restoration


Illustration with a “spoke” image (resolution chart):


Image restoration

Image restoration: resolution chart

There are several “standard” resolution charts, such as this (ISO 12233):

;


Image restoration

Image restoration: model and mathematics

Summary so far:

A linear space-invariant (LSI) imaging model, where

i(x, y) = g(x, y) ∗ h(x, y) + n(x, y)I( fx, fy) = G( fx, fy)H( fx, fy) + N( fx, fy)

Specifically, for defocus with a circular lens (pupil),

h(x, y) = Π(r) H( fx, fy) = jinc (ρ)

The LSI model is applicable to other situations as well. Forexample, with a linear motion blur (constant velocity in thehorizontal direction):

h(x, y) = Π(x) H( fx, fy) = sinc ( fx)


Image restoration

Image restoration: metric

The imaging equation:

i(x, y) = g(x, y) ∗ h(x, y) + n(x, y)

We assume that h(x, y) is known (e.g., measured beforehand). What weget: i(x, y). It’s a noisy, blurred image.

n(x, y) is big =⇒ use a lowpass filter to suppress the noise.Problem: image blurredi(x, y) is blurred =⇒ add highpass signal to sharpen imageProblem: noise magnified

These extreme cases suggest engineering tradeoffs.

The question in algorithm development: Is there an optimal point inbetween?


Image restoration


Assume we know the “ideal” M ×N image g(x, y) and the “restored”image g(x, y). We can calculate the mean-squared error (MSE):

MSE(g, g) =1

MN

∑

x,y

(g(x, y) − g(x, y)

)2(19)

When MSE = 0, we know g(x, y) = g(x, y).Big MSE usually means g(x, y) is very different from g(x, y).BUT: such difference may not matter much visually, e.g., if g(x, y)equals to g(x, y) shifting by one pixel, we hardly notice, but theMSE will be significant.

A less common metric for image processing is the root-mean-square(RMS) error:

RMS(g, g) =

√1

MN

∑

x,y

(g(x, y) − g(x, y)

)2(20)


Image restoration


We can also define the mean absolute error (MAE):

MAE(g, g) =1

MN

∑

x,y

∣∣∣∣g(x, y) − g(x, y)∣∣∣∣ (21)

It shares some common characteristics with MSE.If we want to produce g(x, y) by minimizing the MSE, we tend todistribute the error, resulting in smaller errors across more pixels,because we are penalizing the differences quadraticallyOn the other hand, MAE is penalizing the differences linearly, andtends to allow the restoration of sharp edges


Image restoration


A metric related to MSE is the signal-to-noise ratio (SNR):

SNR(g, g) = 10 log10

∑x,y

(g(x, y)

)2

∑x,y

(g(x, y) − g(x, y)

)2

(22)

The unit is called decibel (dB). Roughly speaking, as a rule-of-thumb,restored images at 30 dB or so are quite good; below 20 dB or so are quite bad.

We may also use the peak signal-to-noise ratio (PSNR):

PSNR(g, g) = 10 log10

MN maxx,y(g(x, y)

)2

∑x,y

(g(x, y) − g(x, y)

)2

(23)


Image restoration: frequency domain methods

1 Image enhancement

2 Image restoration





Image restoration: inverse filter

We begin with the Fourier domain:

I( fx, fy) = G( fx, fy)H( fx, fy) + N( fx, fy).

Let’s ignore N( fx, fy) first. We will see that this is often a bad habit!

Restoration problem can be solved trivially (G = restored):

G( fx, fy) =I( fx, fy)H( fx, fy)

=

(1

H( fx, fy)

)I( fx, fy) (24)

If you are a bit careful, you may worry that sometimesH( fx, fy) = 0. So we modify the restoration filter to have

R( fx, fy) =

{ 1H( fx, fy) H( fx, fy) , 0

0 otherwise.(25)

This is called the inverse filter.E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2017 30 / 60



Example: Frequency domain view of out-of-focus blur:

ρ

1jincρjincρ




Image restoration example:

−→

input output




Image restoration example: Added Gaussian noise with σ = 0.01 × 255

−→

input output




With noise,

G( fx, fy) =

(1

H( fx, fy)

)I( fx, fy)

=

(1

H( fx, fy)

) (G( fx, fy)H( fx, fy) + N( fx, fy)

)

= G( fx, fy) +N( fx, fy)H( fx, fy)

When N is non-zero, for small H, the quantityNH

can be big!



Image restoration: truncated inverse filter

For large H, G =IH

is a good approximation. So, we modify Eq. (25) as

R( fx, fy) =

{ 1H( fx, fy) |H( fx, fy)| > T

0 otherwise.(26)

Several design questions:How does it behave for different values of T?How to pick T for a particular noise level?It’s an improvement. But is it optimal in any sense? (i.e. can we dobetter?)



Image restoration: Wiener filter

Idea: not use a hard threshold. Instead, “trust” the inverse filtersolution depending on the ratio of the signal content vs noise content. Itinvolves slightly more difficult math, but can be shown to be optimalin a statistical sense.

Signal (image) is a random quantity: we are no longer trying torestore one image, but a collection of possible images with the samestatistics that may appear in a particular imaging system (e.g.,imaging cells in a microscope).Noise is also random.Both have some statistical characteristics, e.g., mean, variance.

Note that it is a very different philosophy from treating images asunknown but deterministic quantity!



Image restoration: filter model

Consider that we apply a linear filter to the observation:

+h(x, y)g(x, y)

n(x, y)

i(x, y)w(x, y)

g(x, y)

We want to find the “best” w(x, y) that works for many differentrealizations of signal and noise.

It’s a linear filter design problem, not a (non-linear) estimationproblem!

Goal: Find g(x, y) such that E[(

g(x, y) − g(x, y))2]

is minimum.

Filtering is easy to implement.




We can analytically find the optimal filter w(x, y).In frequency domain,

W( fx, fy) =H∗( fx, fy)

|H( fx, fy)|2 +Φn( fx, fy)Φg( fx, fy)

. (27)

Φg( fx, fy) is called the power spectral density of the image.Φn( fx, fy) is called the power spectral density of the noise.Φg (or Φn) is large means signal (or noise) content at thatfrequency is large.The filter is called the Wiener filter.




Interpretation: It’s the ratio of the signal-to-noise (SNR) content ateach frequency that is important!

Large SNR:ΦnΦg→ 0 =⇒W ≈ H∗

|H|2 = 1H

i.e. approaches an inverse filter

Small SNR:ΦnΦg� {|H|2,H∗} =⇒W ≈ 0

i.e. cut off that frequency



Image restoration: pseudoinverse filter

The quantityΦn( fx, fy)Φg( fx, fy)

can vary with each frequency.

Sometimes, we can only estimate the SNR for the entire image.Let α be the inverse of the SNR, i.e., the noise-to-signal ratio.Then, we can let


|H( fx, fy)|2 + α(28)

which is also known as the pseudoinvere filter. (Some peoplemay still choose to call this the Wiener filter, though.)




Example: Frequency domain view of out-of-focus blur:

ρ

α = 0.01

α = 0.1




Image restoration example: Added Gaussian noise with σ = 0.01 × 255

−→

input output




Wiener filter is analytically very nice, but often not practical.Requires H( fx, fy), Φn( fx, fy), Φg( fx, fy). We often only estimate thelatter two.More commonly, we estimate the noise level, and use pseudoinverse filter.We can also experimentally try different values of α and pick therestored image that looks best.

Other issues:Human eyes do not judge by mean square error. Even if it isoptimal in the analytical sense, it is not optimal as far as humansight is concerned.It cannot handle space-variant blur.It assumes the image is a stationary signal (so statistics at an imagecorner is the same as another), but that’s not true in practice.




In the following few slides, we will derive the Wiener filter usingfundamental linear time-invariant systems theory for stochasticsystems. You should be able to understand the main concepts;nevertheless, it is fine even if you don’t fully comprehend thederivation, as the derivation is not an integral part of our presentationof image processing.

If you have troubles with these derivations, you can consult referencebooks such as Probability and Random Processes for Electrical Engineers byLeon-Garcia.




A quick review of random processes:X(t) is a wide sense stationary (WSS) random process.Mean mX(t) is: mX(t) = E[X(t)].Its autocorrection function RX(τ) is: RX(τ) = E[X(t + τ)X(t)].The power spectral density is: SX( f ) = F {RX(τ)}.Cross-correlation between two WSS random processes is:RX,Y(τ) = E[X(t + τ)Y(t)].Cross-power spectral density is: SX,Y( f ) = F {RX,Y(τ)}.




h(t)X(t) Y(t)

Cross-correlation between input and output:

SX,Y( f ) = S∗Y,X( f ) = H∗( f )SX( f ).

Auto-correlation of output:

SY( f ) = |H( f )|2SX( f ).

Note that the phase information is lost in the transfer function.




To find the optimal solution:Assume signal is zero-mean (ok; can subtract by its mean)Assume noise is uncorrelated with signal (often not true inpractice, e.g. quantization noise)There are many ways to derive the optimal filter. Many of themethods are rather involved; the simplest is to invoke a criterionknown as the orthogonality condition:

error ⊥ observation (29)

Philosophy: if error is along the observation, we could have“observed” it and taken it away. So the minimum error has to beorthogonal to the observation.




In our case: (use 1D for simplicity)Error: g(x) − g(x)Signal: i(x)

So, by the orthogonality condition,

E[ {

g(x) − g(x)} {i(x + τ)}

]= 0. (30)

That means,E[g(x)i(x + τ)

]= E

[g(x)i(x + τ)

]. (31)

And now we are ready to take the Fourier transform.




+h(x, y)g(x, y)

n(x, y)

i(x, y)w(x, y)

g(x, y)

With reference to this diagram, we know:

F {E(g(x)i(x + τ)} = W∗( f )SI( f )F {E(g(x)i(x + τ)} = H( f )SG( f )

SI( f ) = |H( f )|2SG( f ) + SN( f )

The last two equations make use of the assumption that g and n areuncorrelated.




Therefore, if we take Eq. (31) to the Fourier domain, we have

H( f )SG( f ) = W∗( f )SI( f )

= W∗( f )(|H( f )|2SG( f ) + SN( f )

)

W( f ) =H∗( f )

|H( f )|2 +SN( f )SG( f )

.

Using 2D notataion,


|H( fx, fy)|2 +Φn( fx, fy)Φg( fx, fy)

.


Image restoration: spatial domain methods

1 Image enhancement

2 Image restoration





Norm metric

We have already come across MSE and MAE as two metrics for error.If we define

ε(x, y) = g(x, y) − g(x, y), (32)

Then we can define the `2-norm and `1-norm of the errors as

`2-norm: ‖ε(x, y)‖2 =

√∑

x,yε2(x, y) (33)

`1-norm: ‖ε(x, y)‖1 =∑

x,y

∣∣∣∣ε(x, y)∣∣∣∣ (34)

`2-norm is related to MSE as `1-norm is related to MAE.



Norm metric

Often, without the subscript designation, ‖ · ‖means the `2-norm.Furthermore, we often use

‖ε(x, y)‖2 =∑

x,yε2(x, y) (35)

to refer to the square of the `2-norm.

Furthermore, Parseval’s theorem essentially states that the square ofthe `2-norm of a signal and its Fourier transform are the same (exceptperhaps a scaling factor):

‖g(x, y)‖2 = ‖G( fx, fy)‖2 (36)



Image restoration: optimization

Let’s see if spatial domain formulation gives us different insights onthe problem. We seek to minimize

g = arg ming‖h ∗ g − i‖2 (37)

Imagine g as a series of “knobs” to turn! This is an inverse problem.

Direct approach would not work:Small changes in i causes great changes in the solution g.Basically, noise at frequencies where H is small would cause bigproblems.Mathematically, we say that the image restoration problem is ill-posed(or, ill-conditioned).




Ill-posedness is very common for inverse problems.There exists a rich theory on solving inverse problems, and isuseful to many disciplines, e.g. physics.

The “solution” we know from these other works: we do not apply anunconstrained optimization. Instead, we add a constraint (bound) onthe “energy”:

‖g‖2 ≤ E2. (38)

Interpretation: we make sure that noise cannot be multiplied so greatlyand have a large energy. Note the similarity with the “clipping” in Wienerfilter above.




(Calculus) We can turn the problem to unconstrained minimization byLagrange multipliers:

g(µ) = arg ming

{‖h ∗ g − i‖2 + µ‖g‖2

}, (39)

(µ > 0) is the Lagrange multiplier.

Note:µ ≈ 0⇒ “effectively unconstrained”µ→∞⇒ flat image (hence smallest energy)µ therefore controls the tradeoff between fidelity to observed datavs prior knowledge / smoothness.




By Parseval’s theorem, we can find the minimization in the frequencydomain. So, we seek G( fx, fy) such that

‖HG − I‖2 + µ‖G‖2 = 0. (40)

Although not mathematically precise in the derivation, we arebasically seeking

∂∂G

{(HG − I)∗ (HG − I) + µG∗G

}= 0

(H∗G∗ − I∗) H + µG∗ = 0(H∗H + µ

)G∗ = I∗H

G =H∗

‖H‖2 + µI




Therefore, from the space domain, the restoration filter is (again)

R( fx, fy) =H∗( fx, fy)

|H( fx, fy)|2 + µ. (41)

We again arrive at the pseudoinverse filter! Our Lagrange multiplier isequivalent to the noise-to-signal ratio.




If we replace Eq. (38) by a filtered version of g:

‖c ∗ g‖2 ≤ E2. (42)

We can go through a similar analysis and find out that

G( fx, fy) =H∗( fx, fy)

|H( fx, fy)|2 + µ|C( fx, fy)|2 I( fx, fy) (43)

So, we can even attain the complete Wiener solution with

µ|C( fx, fy)|2 =Φn( fx, fy)Φg( fx, fy)

(44)




This informs us that the Wiener filter has multiple interpretations:In time domain, the extra parameter controls the amount ofregularization (energy constraint).In frequency domain, the extra parameter depends on thesignal-to-noise ratio.

If we regularize on a filtered version (using C( fx, fy)) of g, we canembed in C the signal-to-noise information. In low frequency, signalshould dominate; in high frequency, noise dominates. So, C( fx, fy) is ahigh-pass filter


Documents

Introduction Image Enhancement and Restoration perspectives. …elec4245/sp17/lec5-2x2.pdf · 2017-05-04 · Image Restoration : Images are bad, but we want to make them look good