DISP Lab, Graduate Institute of Communication Engineering, NTU

1

Image Enhancement [DVT final project]

Speaker: Yu-Hsiang WangAdvisor: Prof. Jian-Jung Ding

Digital Image and Signal Processing LabGraduate Institute of Communication Engineering

National Taiwan University

2

OutlineTargetPossible enhancement methodsInterpolation

◦Adaptive osculatory rational interpolationEnhancement

◦Bilateral Enhancers◦Non Local Means

ConclusionReference

3

TargetScale image from SD (Standard-

definition) to HD (High-definition).Given: 720x480, YCbCr, 4:2:2, 8

bits per channel.Target: 1920x1080, YCbCr, 4:4:4,

10 bits per channel.

4

Possible enhancement methodsSharpnessNoise reductionEdge smoothnessSkin-tone enhancementTexture enhancementSuper resolution (SR)

Multi-Image SR [1]

Single-Image Multi-Patch SR [1]

5

Interpolation: Bilinear1. Interpolate R1:2. Interpolate R2:3. Interpolate P:

2 11 11 21

2 1 2 1

x x x xf R f Q f Qx x x x

2 12 12 22

2 1 2 1

x x x xf R f Q f Qx x x x

2 11 2

2 1 2 1

y y y yf P f R f Ry y y y

[2]

6

Interpolation: Adaptive osculatory rational interpolationThe interpolation kernel function

of adaptive osculatory rational interpolation (AORI) is more accurately approximate to the ideal interpolation not only in space domain but also in frequency domain.

Apply 4 points to interpolate 1 point in one direction.

7

Interpolation: Adaptive osculatory rational interpolationThe interpolation function

◦where r(x) is the interpolated value, g(xk) are the sample values, RI(x) is the interpolation

kernel.

3

0

,k kk

r x g x RI x x

8

Interpolation: Adaptive osculatory rational interpolationThe interpolation kernel

[M. Hu and J. Q. Tan, ”Adaptive osculatory rational interpolation for image processing,” Journal of Computational and Applied Mathematics, vol. 195, pp. 46-53, 2006.]

2

2

2

2

0.168 0.9129 1.08081,

0.8319 1.0808

0.1953 0.5858 0.39051 2,

2.4402 1.7676

0 2 .

x xx

x x

x xRI x x

x x

x

9

Interpolation: Adaptive osculatory rational interpolationThe original 1920x1080 image:

10

Interpolation: Adaptive osculatory rational interpolationThe interpolated image by AORI:

11

Interpolation: Adaptive osculatory rational interpolationThe original 1920x1080 image:

12

Interpolation: Adaptive osculatory rational interpolationThe interpolated image by AORI:

13

Enhancement: Bilateral EnhancersExtended from the bilateral filter

(BF).BF:

◦A nonlinear filter adopts a low-pass Gaussian filter for both the domain filter and the range filter.

◦Smooth the noise while preserving edges.

14

Enhancement: Bilateral EnhancersBilateral filter (BF)

◦with the normalization factor

where r is the input image, h is the output image, Ω(x) is a subset of the input image r with

x and ξ are the pixels coordinates. (row, column)

[C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in Proc. ICCV, pp. 839–846, 1998.]

1

( ), , ,c s d

x

h x k x r x r r x

( )

, , ,c s d

xk x x r r x

5 5

15

Enhancement: Bilateral EnhancersBilateral filter (BF)

◦with the normalization factor

The function c operates on the spatial domain designed as

The s operates on the range domain designed as

1

( ), , ,c s d

x

h x k x r x r r x

( )

, , ,c s d

xk x x r r x

2, exp2 c

c

xx

2

2, exp2 s

s

r x rr r x

16

Enhancement: Bilateral EnhancersProblem of Bilateral filter:

◦Only edge-preserving and de-noising.

◦Do not enhance the sharpness of an image.

17

Enhancement: Bilateral EnhancersBilateral enhancers (sharpness +

smoothness)

◦The c function is set as

the same as in BF. (Gaussian function on spatial domain)

[C. Gatta and P. Radeva, “Bilateral enhancers,” IEEE ICIP, pp. 3161-3164, 2009.]

( )

, , ,c p d

xj x r x x r x r

1

( ), , ,c s d

x

h x k x r x r r xvs

2

1, exp22 cc

c

xx

18

Enhancement: Bilateral EnhancersBilateral enhancers (sharpness +

smoothness)

◦The p composes of two parts (p = ps + pe): the edge-preserving smoothing (ps)

the selective sharpening (pe)

( )

, , ,c p d

xj x r x x r x r

1

( ), , ,c s d

x

h x k x r x r r xvs

2

2, ,2s s

s

p exp

r x rr r x

2

2, 1 ,2e e

s

p exp

r x rr x r

19

Enhancement: Bilateral EnhancersThe edge-preserving smoothing

(ps)

◦ηs: adjusts the intensity of the blurring.

◦σs: controls how strong should be an edge to be preserved from the blurring.

If the intensity difference is small, Gaussian smoothing is performed.

2

2, ,2s s

s

p exp

r x rr r x

20

Enhancement: Bilateral EnhancersThe selective sharpening (pe)

◦ηe and σe have similar meaning as for the edge-preserving smoothing.

If the intensity difference is small, no enhancement is performed.

2

2, 1 ,2e e

s

p exp

r x rr x r

21

Enhancement: Bilateral EnhancersThe interpolated image by AORI:

22

Enhancement: Bilateral EnhancersEnhanced the previous page’s

image by BE.

23

Enhancement: Bilateral EnhancersThe original 1920x1080 image:

24

Enhancement: Bilateral EnhancersThe interpolated image by AORI:

25

Enhancement: Bilateral EnhancersEnhanced the previous page’s

image by BE.

26

Framework of System

Adaptive osculatory rational interpolation

Input 720 480 image I

Determine property of I by

DCT

Non local means for denoising

Bilateral enhancers for sharpness

1920 1080 image r

Restored 1920 1080 image

uniformnon-uniform

27

Enhancement: Non Local MeansPurpose: Noise reduction.Given an interpolated imageThe estimated value

◦W is a search window of fixed size (we choose 5x5 here) centered at pixel i.

◦The weights depend on the similarity between the pixel i and j.

[A. Buades, B. Coll, and J.-M. Morel, “A non-local algorithm for image denoising,” IEEE CVPR, Jun. 2005.]

.r r i i I

, ,j W

NL r i i j r j

,j

i j

28

Enhancement: Non Local MeansThe weighted Euclidean distance

◦Nk denotes a square neighborhood of fixed size (we choose 3x3) and centered at a pixel k.

◦α is the standard deviation of the Gaussian kernel.

◦G(N) is the Gaussian kernel of the same size as Nk.

22

*i j i jr N r N r N r N G N

29

Enhancement: Non Local MeansThe weights are defined as

◦Z(i) is the normalizing constant

◦h denotes a degree of filtering. (It controls the decay of the weights function of the Euclidean distances.) (h = 2.5)

2

2

1, exp ,i jr N r N

i jZ i h

2

2expi j

j

r N r NZ i

h

30

Enhancement: Non Local MeansScheme of NL-means strategy.

[6]

31

Enhancement: Non Local MeansAdvantage:

◦NL-means compares the gray level of pixels.

◦Compare the geometrical configuration in a whole neighborhood.

Disadvantage:◦Do not perform sharpness.◦Blur some edges.

32

Enhancement: Non Local MeansThe interpolated image by AORI:

33

Enhancement: Non Local MeansDe-noise the previous image by NL-

means.

34

Enhancement: Non Local MeansThe interpolated image by AORI:

35

Enhancement: Non Local MeansDe-noise the previous image by NL-

means.

36

Enhancement: Non Local MeansEnhanced the previous page’s

image by BE.

37

ConclusionAdaptive osculatory rational

interpolation is more accurately approximate to the ideal interpolation.

Bilateral enhancers performs well at sharpness and smoothness.

Non local means is mainly used for noise reduction.

38

Reference [1] D. Glasner, S. Bagon, and M. Irani, “Super-resolution

from a single image,” ICCV, pp. 349-356, Sep. 2009. [2]http://en.wikipedia.org/wiki/Bilinear_interpolation [3] M. Hu and J. Q. Tan, ”Adaptive osculatory rational

interpolation for image processing,” Journal of Computational and Applied Mathematics, vol. 195, pp. 46-53, 2006.

[4] C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in Proc. ICCV, pp. 839–846, 1998.

[5] C. Gatta and P. Radeva, “Bilateral enhancers,” IEEE ICIP, pp. 3161-3164, 2009.

[6] A. Buades, B. Coll, and J.-M. Morel, “A non-local algorithm for image denoising,” IEEE CVPR, Jun. 2005.