Digital Image Processing - Πανεπιστήμιο...

University of Ioannina - Department of Computer Science and Engineering

Image Restoration and Reconstruction

(Linear Restoration Methods)

Christophoros Nikou

cnikou@cs.uoi.gr

Digital Image Processing

C. Nikou – Digital Image Processing (E12)

Contents

In this lecture we will look at linear image

restoration techniques

– Differentiation of matrices and vectors

– Linear space invariant degradation

– Restoration in absence of noise

• Inverse filter

• Pseudo-inverse filter

– Restoration in presence of noise

• Inverse filter

• Wiener filter

• Constrained least squares filter

Differentiation of Matrices and

Vectors

Notation:

A is a MxN matrix with elements aij.

x is a Nx1 vector with elements xi.

f(x) is a scalar function of vector x.

g(x) is a Mx1 vector valued function of vector x.

Vectors (cont...)

Scalar derivative of a matrix.

Vectors (cont...)

Vector derivative of a function (gradient).

f(x) is a scalar function of vector x.

f f ff

Vectors (cont...)

Vector derivative of a vector (Jacobian):

g(x) is a Mx1 vector valued function of vector x.

Vectors (cont...)

Some useful derivatives.

b is a Nx1 vector with elements bi.

It is the derivative of the scalar valued function

bTx with respect to vector x.

Vectors (cont...)

A is a NxN matrix with elements aij.

If A is symmetric:

x Ax A A x

x Ax Ax

Vectors (cont...)

b is a Mx1 vector with elements bi.

It may be proved using the previous properties.

Ax b A Ax b

Linear, Position-Invariant

Degradation

We now consider a degraded image to be

modelled by:

where h(x, y) is the impulse response of the

degradation function ( i.e. point spread function

blurring the image).

The convolution implies that the degradation

mechanism is linear and position invariant (it

depends only on image values and not on

location).

( , ) ( , ) ( , ) ( , )g x y h x y f x y x y

Degradation (cont...)

In the Fourier domain:

where multiplication is element-wise.

In matrix-vector form:

where H is a doubly block circulant matrix and

f,g, and η are vectors (lexicographic ordering).

( , ) ( , ) ( , ) ( , )G k l H k l F k l N k l

g = Hf + η

Degradation (cont...)

If the degradation function is unknown the

problem of simultaneously recovering f(x,y)

and h(x,y) is called blind deconvolution.

g = Hf + η

( , ) ( , ) ( , ) ( , )g x y h x y f x y x y

( , ) ( , ) ( , ) ( , )G k l H k l F k l N k l

Estimating the point spread function

• In what follows, we consider that the degradation

function is known.

• If the psf is not known, some basic methods to

estimate it are:

– By observation

• Apply sharpening filters to a sub-image gs(m,n) where the

signal is strong (there is almost no noise) and obtain a visually

pleasant result fs(m,n). The psf may be approximated by

Hs(k,l)= Gs(k,l)/ Fs(k,l).

• The task needs trial and error and may be tedious.

• Used in special circumstances (e.g. Restoration of old

photographs)

(cont.)

estimate it are:

– By experimentation

• If the acquisition equipment or a similar one is available an

image similar to the degraded may be obtained by varying the

system settings.

• Then obtain the image of an impulse (small dot of light) using

the same settings.

(cont.)

estimate it are:

– By modeling: atmospheric turbulence:

5/62 2( , ) expH u v k u v

(cont.)

• By modeling: planar motion

– x0(t) and y0(t) are the time varying components of

motion at each pixel.

– The total exposure at any pixel is obtained by

integrating the instantaneous exposure over the time

the shutter is open.

– Assumption: the shutter opening and closing is

instantaneous.

– If T is the duration of the exposure, the recorded image

is expressed by:

( , ) [ ( ), ( )]T

g x y f x x t y y t dt

(cont.)

2 ( ) ( )

( , ) ( , )

[ ( ), ( )]

j ux vy

Tj ux vy

T j ux t vy t

j ux t vy t

G u v g x y e dxdy

f x x t y y t dt e dxdy

f x x t y y t e dxdy dt

F u v e dt

F u v e

2 ( ) ( )

0( , )

T j ux t vy t

H u v e dt

(cont.)

• Considering uniform linear motion:

0 0( ) , ( )t t

x t a y t bT T

• The psf becomes:

0( , )

T j uat vbt T

j ua vb

H u v e dt

Tua vb e

(cont.)

• Result of blurring with:

0 0( ) , ( ) , 0.1, 1t t

x t a y t b a b TT T

Linear Restoration

Using the imaging system

we want to estimate the true image from the

degraded observation with known

degradation H.

A linear method applies an operator (a

matrix) P to the observation g to estimate the

unobserved noise-free image f:

f̂ = Pg

g = Hf + η

Restoration in Absence of Noise

The Inverse Filter

When there is no noise:

an obvious solution would be to use the

inverse filter:

yielding

g = Hf

-1P = H

ˆ -1 -1f = Pg = H g = H Hf = f

The Inverse Filter (cont...)

For a NxN image, H is a N2xN2 matrix!

To tackle the problem we transform it to the

Fourier domain.

H is doubly block circulant and therefore it

may be diagonalized by the 2D DFT matrix W:

ˆ -1f = H g

-1H = W ΛW

Therefore:

-1H = W ΛW

{ (1,1),..., ( ,1), (1,2),... ( , )}diag H H N H H N NΛ

f̂ = Pg ˆ -1f = H g ˆ -1 -1f = (W ΛW) g

ˆ -1 -1f = W Λ Wg ˆ -1 -1

Wf = WW Λ Wg

ˆ -1F = Λ G

This is the vectorized form of the DFT of the

image:

Take the inverse DFT and obtain f(m,n).

Problem: what happens if H(k,l) has zero

values?

Cannot perform inverse filtering!

ˆ -1F = Λ G( , )ˆ ( , )( , )

G k lF k l

H k l =

The Pseudo-inverse Filter

A solution is to set:

which is a type of pseudo-inversion.

Notice that the signal cannot be restored at

locations where H(k,l)=0.

( , ), ( , ) 0

ˆ ( , )( , )

0 , ( , ) 0

G k lH k l

H k lF k l

The Pseudo-inverse Filter (cont...)

A pseudo-inverse filter also arises by the

unconstrained least squares approach.

Find the image f, that, when it is blurred by

H, it will provide an observation as close as

possible to g, i.e. It minimizes the distance

between Hf and g.

The Pseudo-inverse Filter (cont...)

This distance is expressed by the norm:

min ( )Jf

Hf - g

2 0T H Hf -g 2 2T T H Hf H g

f H H H g

2( )J f Hf - g

Restoration in Presence of Noise

The Inverse Filter

Recall the imaging model with spatially

invariant degradation and noise

g = Hf + η

( , ) ( , ) ( , ) ( , )g x y h x y f x y x y

( , ) ( , ) ( , ) ( , )G k l H k l F k l N k l

Applying the inverse filter in the Fourier

domain:

Even if we know H(k,l) we cannot recover F(k,l)

due to the second term.

If H(k,l) has small values the second term

dominates (it goes to infinity if H(k,l)=0!).

( , ) ( , ) ( , ) ( , )G k l H k l F k l N k l

( , )ˆ ( , ) ( , ) ( , )

N k lF k l F k l

• One approach to get around the problem is

to limit the ratio G(k,l) / H(k,l) to

frequencies near the origin that have lower

probability of being zero.

• We know that H(0,0) is usually the highest

value of the DFT.

• Thus, by limiting the analysis to

frequencies near the origin we reduce the

probability of encountering zero values.

Blurring degradation

The Inverse Filter (cont...)Inverse filter with cut-off

Wiener Filter

• So far we assumed nothing about the statistical

properties of the image and noise.

• We now consider image and noise as random

variables and the objective is to find an estimate

of the uncorrupted image f such that the mean

square error between the estimate and the image

is minimized:

where E[x] is the expected value of vector x.

ˆmin ( )E f

Wiener Filter (cont...)

• Recall also the definition of the correlation

matrix between two vectors x and y:

• We assume that the image and the noise

are uncorrelated:

1 1 1 2 1

N N N N

E x y E x y E x y

xyR xy

ηf fηR R 0

• We are looking for the best estimate:

• Let’s confine our estimate to be obtainable

by a linear operator on the observation:

and the goal is to find the best matrix P.

ˆmin ( )E f

ˆ f Pg

Denoting by the n-th row of P:

2 22ˆ ˆ ˆ( ) ( )J E E E

f f f f f f Pg

2 2ˆ( ) T T

n n n n

J E E f f p g f p g

ˆ( )T

n n n n

J E f f p g f p g

T T T T T T

n n n n n n n n

E f f p gf f g p p gg p

T T T T T T

n n n n n n n n

E E E E f f p gf f g p p gg p

2n n n

f f gf ggR p R p R p

• We can now minimize the sum with respect

to each term:

2 0n n n

f f gf ggR p R p R p

2 2 0n n gf ggR R p

gg gfp R R

f g ggp R R

• Assembling the rows together:

• We have to compute the two matrices:

1 fg ggP R R

( )( )T TE E ggR gg Hf η Hf η

T T T T T TE Hff H Hfη ηf H η η

T T ff fη ηf ηηHR H HR R H R

• Assumming noise is uncorrelated with image:

• Also,

• Finally the matrix we are looking for is

( ) ...T T TE E fg ffR fg f Hf η R H

T gg ff ηηR HR H R

fg gg ff ff ηη

P R R R H HR H R

• The estimated uncorrupted image is

which may be also expressed as

• This result is known as the Wiener filter or

the minimum mean square error (MMSE)

filter.

ˆ f Pg 1ˆ T T

ff ff ηη

f R H HR H R g

1 1 1ˆ T T

ηη ff ηη

f H R H R H R g

• Special cases:

–No blur (H=I, g=f+η):

–No noise (Rηη=0, g=Ηf):

•This is the inverse filter.

–No blur, no noise (H=I, Rηη=0):

•Do nothing on the observation.

ff ff ηη

f R R R g

1ˆ f Η g

ˆ f g

• The size of the matrix to be inverted poses

difficulties and Wiener filter is implemented in the

Fourier domain.

• This occurs when H is doubly block circulant

(represents convolution) and the image f and

noise η are wide-sense stationary (w.s.s).

Definition of a w.s.s. signal:

1) E[f (m,n)]=μ, independent of m,n.

2) E[f(m,n) f(k,l)]=r(m-k,n-l), independent of location.

Reminder: the inverse DFT complex exponential

matrix diagonalizes any circulant matrix:

The columns of W-1 are the eigenvectors of any

circulant matrix H.

The corresponding eigenvalues are the DFT values

of the signal producing the circulant matrix.

Remember also that

and that

1 HH W Λ W

We will employ the following relations:

If H is real:

1 HH W Λ W

H W Λ W WΛ W

1 * * 1T T H H

H H WΛ W W Λ W

1 * 1 *1( )N

H HW Λ W W Λ W

The Wiener solution is now transformed to

the Fourier domain:

Notice that the matrices are diagonal.

1ˆ T T

ff ff ηη

f R H HR H R g

11 1 * 1 1 1 * 1( )( ) ( )( )( ) ( )

ff H H ff H ηη

W Λ W W Λ W W Λ W W Λ W W Λ W W Λ W g

11 * 1 *( )

ff H H ff H ηη

W Λ Λ W W Λ Λ Λ Λ W g

* * 1ˆ ( ) ff H H ff H ηηWf Λ Λ Λ Λ Λ Λ Wg

1 * * 1( ) ff H H ff H ηηW Λ Λ Λ Λ Λ Λ Wg

Sff (k,l)=DFT(Rff (m,n)) is the power spectrum

of the image f (m,n).

Sηη(k,l)=DFT(Rηη (m,n)) is the power spectrum

of the noise η(m,n).

* * 1ˆ ( ) ff H H ff H ηηWf Λ Λ Λ Λ Λ Λ Wg

( , ) ( , )( , ) ( , )

( , ) ( , ) ( , )

S k l H k lF k l G k l

S k l H k l S k l

If Sff (k,l) is not zero we may define the Signal

to Noise Ratio in the frequency domain:

and the Wiener filter becomes:

( , )SNR( , )

ffS k lk l

( , )( , ) ( , )

( , ) SNR ( , )

H k lF k l G k l

H k l k l

A well known estimate of Sff (k,l) is the

periodogram (the ML estimate of Rff when f is

assumed Gaussian):

In practice, as F(k,l) is unknown, we use

21( , ) ( , )ffS k l F k l

21ˆ ( , ) ( , )ffS k l G k lMN

Wiener Filter (cont...)Inverse WienerDegraded image

Noise variance one order

of magnitude less.

Noise variance ten orders

of magnitude less.

Constrained Least Squares Filter

• When we do not have information on the power

spectra the Wiener filter is not optimal.

• Another idea is to introduce a smoothness term in

our criterion.

• We define smoothness by the quantity

– Q is a high pass filter operator, e.g. the Laplacian,

– Q is a doubly block circulant matrix representing

convolution.

• We look for smooth solutions minimizing

Constrained Least Squares Filter (cont...)

We have the following constrained least

squares (CLS) optimization problem:

Minimize subject to

yielding the Lagrange multiplier minimization of

the function:

2 2( , )J f Hf g Qf

Smoothness termData fidelity term

2Qf Hf g

2 ( ) 2 0T T H Hf g Q Qf( , ) 0J

f H H Q Q H g

Parameter λ controls the degree of smothness:

0, T T

f H H H g pseudo-inverse, ultra rough solution

, 0 f ultra smooth solution

In the Fourier domain, the constrained least

squares filter becomes:

Keep always in mind to zero-pad the images

properly.

( , )( , ) ( , )

( , ) ( , )

H k lF k l G k l

H k l Q k l

Low noise: Wiener and CLS generate equal results.

High noise: CLS outperforms Wiener if λ is properly selected.

It is easier to select the scalar value for λ than to approximate the SNR

which is seldom constant.

Restoration Performance Measures

Mean square error (MSE):

1 ˆMSE ( , ) ( , )M N

f m n f m nMN

Original image f and restored image f̂

Restoration Performance Measures

(cont...)

The Signal to Noise Ratio (SNR) considers the

difference between the two images as noise:

ˆ ( , )

SNRˆ( , ) ( , )

f m n f m n

Digital Image Processing - Πανεπιστήμιο...

Documents

digital image processing 3 - cs.uoi.grpkarvel/courses/dip/Theory/digital_image_processing... · Digital Image Processing Intensity Transformations Πέτρος Καρβέλης pkarvelis@gmail.com

Michelle Nikou Ed Kit - Heide Museum of Modern Art Nikou Ed Kit.pdf · What mad pursuit? What struggle to escape? What pipes and timbrels? What wild ecstasy? Materials Grey lead pencils

Digital Image Processingcnikou/Courses/Digital_Image_Processing/2012-2013/... · Digital Image Processing Morphological Image Processing 2 C. Nikou –Digital Image Processing Morphological

Christophoros Nikou cnikou@cs.uoi

Wavelets and Multiresolution Processing (Background) › ~cnikou › Courses › Digital_Image... · Wavelets and Multiresolution Processing (Background) Christophoros Nikou cnikou@cs.uoi.gr

Digital Image Processing Basic Methods for Image Segmentationcs.uoi.gr/~cnikou/Courses/Digital_Image_Processing/... · 2013-12-06 · 2 C. Nikou –Digital Image Processing Image

Wavelets and Multiresolution Processing (Background)lkon/imageprocessing/Chapter_07a... · C. Nikou – Digital Image Processing (E12) Wavelets and Multiresolution Processing All

27/4/2011 Color Image Processing - University of Ioanninacnikou/Courses/Digital_Image_Processing/... · 2011. 4. 27. · 27/4/2011 3 13 Color Fundamentals (cont…) Distinguishing

Image Analysis Segmentation by Clustering2 C. Nikou –Image Analysis (T-14) Segmentation by Clustering •Grouping and fitting •The Gestalt school of psychology •K-means •Mean

Digital Image Processing - University of Ioanninacnikou/Courses/Digital_Image... · C. Nikou – Digital Image Processing (E12) The ith row of a NxN Haar transformation matrix contains

Digital Image Processing - cs.uoi.grpkarvel/courses/dip/Theory/digital_image_processing... · Digital Image Processing Αποκαάαη ικόνας ... R. Gonzalez and R. Woods

Digital Image Processing Contentscs.uoi.gr/~cnikou/Courses/Digital_Image_Processing/...Image Processing (2002) A B C. Nikou – Digital Image Processing (E12) Images taken from Gonzalez

Digital Image Processingcnikou/Courses/Digital_Image_Processing/... · 2011. 3. 8. · 2 3 Image Histograms The histogram of an image shows us the distribution of ggy grey levels

Digital Image Processing - University of Ioanninacnikou/Courses/Digital_Image_Processing/2011-201… · – Key stages in digital image processing. 2/17/2012 4 7 What is a Digital

University of Ioannina - Department of Computer Science Intensity Transformations (Point Processing) Christophoros Nikou cnikou@cs.uoi.gr Digital Image

ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina

University of Ioannina - Department of Computer Science Spatial Filtering Christophoros Nikou cnikou@cs.uoi.gr Digital Image Processing

University of Ioannina - Department of Computer Science Digital Imaging Fundamentals Christophoros Nikou cnikou@cs.uoi.gr Digital Image Processing Images

people.kth.sedimos/pdfs/CDC_2017_timed_abstractions.pdf · Decentralized Abstractions and Timed Constrained Planning of a General Class of Coupled Multi-Agent Systems Alexandros Nikou,

Digital Image Processing - University of Ioanninacs.uoi.gr/~cnikou/Courses/Digital_Image_Processing/... · C. Nikou –Digital Image Processing (E12) Filtering in the Frequency Domain