20100508 Color Filter Array Demosaicking Using High-Order Interpolation Techniques With a Weighted Median Filter for Sharp Color Edge Preservation

8/7/2019 20100508 Color Filter Array Demosaicking Using High-Order Interpolation Techniques With a Weighted Median Filter for Sharp Color Edge Preservation

http://slidepdf.com/reader/full/20100508-color-filter-array-demosaicking-using-high-order-interpolation-techniques 1/44

School of Electrical Engineering and Computer Science

Kyungpook National Univ.

Color Filter Array Demosaicking Using High-

Order Interpolation Techniques with a

Weighted Median Filter for Sharp Color Edge

Preservation

IEEE Transactions on Image Processing ,

Vol. 18, No. 9, 2009

Jim S, Jimmy Li, and Sharmil Randhawa

Presented by In-Yong Song



Abstract

Proposed method – Demosaicking

• Use of single-sensor digital camera for color image capture

• Estimation process to determine missing color value

– Application of Taylor series and cubic spline

interpolation

• Estimation of four opposite direction value

• Preservation of edge

– Weighed median filter

• Determination of weight by using classifier based on edge

orientation map

2 / 44



Introduction

Color filter array demosaicking – Obtaining full color images from images captured by

single image sensor

– Edge-directed interpolation• Use in most demosaicking method

• Determination of preferred direction by gradient around

pixel

Fig. 1. An 8 x 8 window of the Bayer pattern.

3 / 44



Common assumption in demosaicking – Locally constant color difference or hue based on

high interchannel correlation within object of image

– Problem of common assumption

• Only use within boundary of object

– Some interpolation method

• Use of weighted average of neighboring pixel

– Determining weight by gradient

• Use of gradient value

– Selection of preferred direction for interpolation

4 / 44



– Disadvantege of previous method• Production of color artifact

• Limitation of performance by accuracy of estimation

algorithm

5 / 44



Proposed method – Proposal of noniterative demosaicking algorithm

• Determination of missing color value

– Performing differently for various type of image

– Production of minimal artifact

– Outperforming previous demosaicking algorithm for

most type of image

6 / 44



– Two stage operation• Interpolation algorithm in first stage

– Determining four estimates of missing color value for four

different direction

– Application of Taylor series and cubic spline

• Output stage in second stage

– Use of classifier based on edge orientation map

» Assigning weight for weighted median filter

– Determination of output by using edge-preserving weighted

median filter

7 / 44



High-Order Extrapolation(HOX)

Description of extrapolation – Use of only pixels on same side of edge

– Avoiding extrapolation across edge as smooth hue

assumption

Fig. 2. One-dimensional extrapolation example.

8 / 44



High-Order Extrapolation(HOX) – Determination of four directional estimates of

missing color pixel

• Obtaining each of directional estimate by exploiting spectral

correlation among neighboring pixel along particulardirection

– Inclusion of higher-order term

• Getting more high accuracy

9 / 44



– Approximation by using Taylor series

– Estimate of green value at position in Fig. 3.

( )

0

( )( ) ( )

!

m

m

m

g ag x x a

m

∞

=

= −∑

( ) x x G

Fig. 3. 1-D Bayer pattern and edge boundaryon the RHS of .

x B

(1)

10 / 44



– Application of Taylor series• Extrapolating green value pixel at position along edge

from pixel on left-hand side

– First order approximation

– Central difference approximation

x

' '' ( )1 1

( ) ( 1) ( 1) ( 1) ( 1)2! !

ng x g x g x g x g xn

= − + − + − + ⋅⋅ ⋅ − + ⋅⋅ ⋅ (2)

where is the value of a green pixel at location .( )g x

x

( )

( 1) 0 2ng x for n− = ≥ (3)

_ _

' 2( ) ( 2)

( 1)2 2

x xg x g x G G

g x −− − −− = = (4)

where and are the missing green pixel values at positionand , respectively

_

xG

_

2 xG

−x

2 x −

11 / 44



– Color difference approximation

– From (5)

– Substituting (6) into (4)

– Hence, by (2),(3) and (7)

_ _ _

1 1 2 2 x x x x x xG B G B G B

− − − −− ≈ − ≈ − (5)

_ _

2 2 x x x xG G B B

− −− = − (6)

' 2( 1)2

x x B B

g x −−

− = (7)

^

1 2

1( )

2 x x x x

G G B B− −

= + − (8)

where is the estimated green value at position^

xG x

12 / 44



– Improvement of accuracy by including next higherorder team

• Second-order approximation

• Neglect of third and higher order derivatives

– Then (2) approximation

– Central difference approximation for

( )

( 1) 0 3ng x for n− = ≥ (9)

^' ''

1

1( 1) ( 1)

2 x x

G G g x g x−

= + − + − (10)

''

( 1)g x −

' '

''( ) ( 2)

( 1)2

g x g xg x

− −− = (11)

13 / 44



– Hence, (10) modification

– Using central difference approximation for

– By (9)

^' ' '

1

1( 1) ( ( ) ( 2))

4 x x

G G g x g x g x−

= + − + − − (12)

'' ''

(3 )

1 3( ) ( )

2 2( 1)1 3

( ) ( )2 2

g x g xg x

x x

− − −

− =− − −

(3 )

( 1)g x −

(13)

'' ''1 3( ) ( ) 02 2

g x g x− − − = (14)

14 / 44



– Using central difference approximation forand

– Substituting (16) into (12)

– Applying central difference approximation

– And

'' 1( )

2

g x −

'' 3( )

2g x −

' ' ' '

( ( ) ( 1)) ( ( 1) ( 2)) 0g x g x g x g x− − − − − − = (15)' ' '

( ) 2 ( 1) ( 2)g x g x g x∴ = − − − (16)

^' '

1

3 1( 1) ( 2)

2 2 x x

G G g x g x−

= + − − − (17)

_ _

' 2( ) ( 2)

( 1)2 2

x xg x g x G G

g x

−− − −

− = = (18)

' 1 3( 1) ( 3)

( 2)2 2

x xg x g x G G

g x − −− − − −

− = = (19)

15 / 44



– Substituting (18) and (19) into (17)

– Hence, using (6), (20)

– Obtaining extrapolated values for other three

directions, right, top, and bottom directions similarly

_ _^

1 2 1 3

3 1( ) ( )

4 4 x x x x x x

G G G G G G− − − −

= + − − − (20)

^

1 2 1 3

3 1( ) ( )

4 4 x x x x x x

G G B B G G− − − −

= + − − − (21)

16 / 44



– Determination of missing green value in Fig. 1._

45G

^

45 44 45 43 44 42

^

45 46 45 47 46 48

^

45 35 45 25 35 1 5

^

45 55 45 65 55 75

3 1( ) ( )

4 4

3 1( ) ( )

4 4

3 1( ) ( )4 4

3 1( ) ( )

4 4

L

R

T

B

G G B B G G

G G B B G G

G G B B G G

G G B B G G

= + − − −

= + − − −

= + − − −

= + − − −

where L,R,T and B indicate the left, right, top and bottom directions

(22)

17 / 44



High-Order Interpolation (HOI)

Interpolation method – Variation to previous method (HOX)

• Four directional estimates of missing color pixel

– Difference between HOI and HOX

• Including adjacent green value on the other side of themissing sample in interpolation process

– Use of presumption for nearest known sample

• Containing the most accurate information about missing

sample

18 / 44



– Obtaining more accurate estimate by interpolating

across center

– Using central difference approximation and

– For second-order approximation, by substituting (23)

and (24) into (11)

' ( 1) ( 1)( )

2

g x g xg x

+ − −= (23)

' ( 1) ( 3)( 2)

2

g x g xg x

− − −− = (24)

'

( )g x '

( 2)g x −

^'

1

1( 1) ( ( 1) 2 ( 1) ( 3))

8 x x

G G g x g x g x g x−

= + − + + − − + − (25)

19 / 44



– Combining(25) with (7)

– Obtaining extrapolated values for the other three

directions, right, top, and bottom directions similarly

– Difference between (26) and (21)

^

1 2 1 1 3

1 1( ) ( 2 )

2 8 x x x x x x x

G G B B G G G− − + − −

= + − + − + (26)

^

1 2 1 1 31 1( ) ( 2 )2 8

x x x x x x xG G B B G G G

− − + − −= + − + − + (26)

^

1 2 1 3

3 1( ) ( )

4 4 x x x x x x

G G B B G G− − − −

= + − − − (21)

20 / 44



– Using same convention as in (22), new equations for

interpolation w.r.t (26)

^

45 44 45 43 46 44 42

^

45 46 45 47 44 46 48

^

45 35 45 25 55 35 15

^

45 55 45 65 35 55 75

1 1( ) ( 2 )

2 8

1 1

( ) ( 2 )2 8

1 1( ) ( 2 )

2 8

1 1( ) ( 2 )

2 8

L

R

T

B

G G B B G G G

G G B B G G G

G G B B G G G

G G B B G G G

= + − + − +

= + − + − +

= + − + − +

= + − + − +

(27)

21 / 44



Cubic Spline Interpolation (CSI)

Interpolation method – Generation of smooth curve to fit set of data point

– Piecewise continuous curve with continuous first and

second-order derivative

– Denoting third degree polynomial as cubic spline3 2

( )i i i i i

S x a x b x c x d = + + +

( )i

S x

(28)

where is the index of the splinei

Fig. 4. 1-D Bayer pattern along the x-axis.

22 / 44



– Three continuity condition for cubic spline

interpolation

3 2

0 0 0 0 0

( ) [0, 2]S x a x b x c x d for x= + + + ∈

3 2

1 1 1 1 1( ) [2, 4]S x a x b x c x d for x= + + + ∈

3 2

2 2 2 2 2( ) [4, 6]S x a x b x c x d for x= + + + ∈

(29)(30)

(31)

11 : ( ) ( ) 1, 2

i i i ist condition S x S x for i

−= =

' '

12 : ( ) ( ) 1, 2

i i i ind condition S x S x for i

−= =

'' ''

13 : ( ) ( ) 1, 2

i i i ird condition S x S x for i

−= =

(32)

(33)

(34)

23 / 44



– By the first continuity condition (32)

– The first derivatives of splines

0 0 0(0)S G d = =

0 2 0 0 0 0(2) 8 4 2S G a b c d = = + + +

1 2 1 1 1 1(2) 8 4 2S G a b c d = = + + +

1 4 1 1 1 1(4) 64 16 4S G a b c d = = + + +

2 4 2 2 2 2(4) 64 16 4S G a b c d = = + + +

2 6 2 2 2 2(6) 216 36 6S G a b c d = = + + +

(35)

(36)

(37)

(38)

(39)

(40)

' 2

0 0 0 0

( ) 3 2S x a x b x c= + +' 2

1 1 1 1( ) 3 2S x a x b x c= + +

' 2

2 2 2 2( ) 3 2S x a x b x c= + +

(41)

(42)

(43)

24 / 44



– By the second continuity condition (33)

– The second derivatives of splines

' '

0 1(2) (2)S S=

' '

1 2(4) (4)S S=

0 0 0 1 1 112 4 12 4 0a b c a b c∴ + + − − − =

1 1 1 2 2 248 8 48 8 0a b c a b c∴ + + − − − =

(44)

(45)

(46)

(47)

''

0 0 0( ) 6 2S x a x b= +

''

1 1 1( ) 6 2S x a x b= +

''

2 2 2( ) 6 2S x a x b= +

(48)

(49)

(50)

25 / 44



– By the third continuity condition (34)

– Estimates of green values at positions = 1, 3, and

5

'' ''

0 1(2) (2)S S=

'' ''

1 2(4) (4)S S=

0 0 1 112 2 12 2 0a b a b∴ + − − =

1 1 2 224 2 24 2 0a b a b∴ + − − =

(51)

(52)

(53)

(54)

x

^

1 0 0 0 0 0(1)G S a b c d = = + + +

^

3 1 1 1 1 1(3) 27 9 3G S a b c d = = + + +

^

5 2 2 2 2 2(5) 125 25 5G S a b c d = = + + +

(55)

(56)

(57)

26 / 44



– Obtaining two extra condition by using hue

assumption

– By (6)

– Hence, by (55) and (56)

– Similarly

– Hence, by (56) and (57)

_ _

3 1 3 1 B B G G− = − (58)

_ _

5 3 5 3 B B G G− = − (60)

3 1 1 1 1 1 0 0 0 027 9 3 B B a b c d a b c d − = + + + − − − − (59)

5 3 2 2 2 2 1 1 1 1125 25 5 27 9 3 B B a b c d a b c d − = + + + − − − − (61)

27 / 44



– Solving 12 simultaneous equation, namely (35)-

(40),(45),(47),(52),(54,),(59), and (61)

MC V = (62)

where M, C and V are given by (63)-(65)

0 0 0 1 0 0 0 0 0 0 0 0

8 4 2 1 0 0 0 0 0 0 0 0

0 0 0 0 8 4 2 1 0 0 0 0

0 0 0 0 6 41 64 1 0 0 0 0

0 0 0 0 0 0 0 0 6 41 64 1

0 0 0 0 0 0 0 0 216 36 6 1

12 4 1 0 12 4 1 0 0 0 0 0

0 0 0 0 4 88 1 0 4 8 8 1 0

12 2 0 0 12 2 0 0 0 0 0 0

0 0 0 0 2 4 2 0 0 2 4 2 0 0

1 1 1 1 2 7 9 3 1 0 0 0 0

0 0 0 0 27 9 3 1 125 25 5 1

LetM

= − − −

− − −

− −

− −

− − − −

− − − −

(63)

28 / 44



0

0

0

0

1

1

1

1

2

2

2

2

a

b

c

d

a

bC

c

d

a

b

c

d

=

0

2

2

4

4

6

3 1

5 3

0

0

0

0

G

G

G

G

G

GV

B B

B B

= −

−

(64)(65)

1C M V −∴ = (66)

29 / 44



– Estimate on the left hand side of missing value (x=5)

– In general

– Determining other three estimates(R,T and B)

similarly – Comparing CSI method [22]

^

5 0 2 4 6 3 1 5 3

1{ 23 23 8( ) 40( )}

48G G G G G B B B B= + + + + − + − (67)

^

1 1 3 5 2 4 2

1{ 23 23 8( ) 40( )}

48 x x x x x x x x x

G G G G G B B B B+ − − − − − −

= + + + + − + − (68)

^

3 1 1 3 2 2

1{ 23 23 8( ) 8( )}

48 x x x x x x x x x

G G G G G B B B B+ + − − − +

= + + + + − − − (69)

30 / 44



Interpolation for The Red and BluePlanes

Interpolation method – Obtainment of higher-order approximation for green

plane only

– Use of first-order approximation for red and blue

plane

– Red pixel value at blue and green position^ ^ ^

45 34 45 34

^ ^ ^

45 36 45 36

^ ^ ^

45 54 45 54

^ ^ ^

45 56 45 56

( )

( )

( )

( )

TL

TR

BL

BR

R R G G

R R G G

R R G G

R R G G

= + −

= + −= + −

= + −

^ ^

44 43 44 43

^ ^

44 45 44 45

^ ^

44 34 44 34

^ ^

44 54 44 54

( )

( )

( )

( )

L

R

T

B

R R G G

R R G G

R R G G

R R G G

= + −

= + −= + −

= + −

(70) (71)

31 / 44



Edge Orientation Classifier and TheWeighted Median Filter

Edge orientation map – Determination of filter weight for weighted median

filter

• Preservation of sharp edge

– Indicating possible orientation of edge

• Use of CFA image input for every pixel

– Interpolating horizontal and vertical gradient for pixel

• Determination by known neighboring pixel in CFA input

32 / 44



– Vertical gradient and horizontal gradient at red/blue

pixel

– Vertical gradient and horizontal gradient at green

pixel

– Use of logical function

• Production of edge orientation map for whole color filter

array image

45( ) B

35 55 44 46,V G G H G G= − = − (72)

44( )G

34 54 43 45,V R R H B B= − = − (73)

1,( )

0,

if V H f V H

otherwise

<< =

(74)

33 / 44



– Estimation of red/blue pixel at blue/red pixel location

• Use of edge orientation map for diagonal edge

• Known red and blue pixel location in Bayer pattern

• For example at

– Use of diagonal gradient and instead of

and in (72), respectively

– Use of 2-D standard median filter for each edge

orientation map

• Removal of any outlier

45 B

44 46G G−

36 54 R R−

35 55G G−

44 46G G−

34 / 44



Weighted median filter – Robustness and edge pre-serving capability

– Offering great flexibility in design specification

– Control of filtering behavior

– Discrete-time vector , output Y of WMF

of width N

• Filter weight

• Is given by

_ 1 2[ , , , ]

N X X X X = ⋅ ⋅ ⋅

_ 1 2[ , , , ]

N W W W W = ⋅ ⋅ ⋅ (75)

1 2 2[ , ,..., ]

N N Y MEDIAN W X W X W X = ◊ ◊ ◊ (76)

where denotes the median operation and denotes theduplication operator

[ ]. MEDIAN ◊

35 / 44



– Adjusting weight for each of four estimates by

utilizing edge information in orientation map

• Use of WMF with window width of 4

,..., ( )K X X X K times◊ = (77)

where K is the duplication number

Fig. 5. Density functions of weighted median filters of width 4 with threepossible sets of weights and 4-point mean for uniformly distributed samples.

36 / 44



– Output for vertical edge indicated in edge orientation

map

– Output for horizontal edge indicated in edge

orientation map

^ ^ ^ ^ ^

, , 2 , 2 L R T B

G MEDIAN G G G G = ◊ ◊

(78)

^ ^ ^ ^ ^

2 , 2 , , L R T B

G MEDIAN G G G G = ◊ ◊

(79)

37 / 44



Proposed Demosaicking Algorithm

Block diagram of proposed algorithm

Fig. 6. Block diagram of the proposed algorithm for the green plane.

38 / 44



Experimental Results

Assessing performance of our proposedmethod

Fig. 7. Images used for performance evaluation.

39 / 44



– Analysis of various demosaicking method

quantitatively

Table 1. Image Quality Performance measures – PSNR dB.

40 / 44



Table 2. Image Quality Performance Measures – NCD .3

( 1 0 )−×

41 / 44



– Assessment of performance on color edge

preservation of our demosaicking method

Fig. 8. Picket fence region of (a) the original Lighthouse image and the demosaiced outputimages using (b) Bilinear interpolation, (c) Freeman, (d) Kimmel, (e) Hanmilton, (f) Lu&Tan,(g) Gunturk, (h) Plataniotis, (i) Hirakawa, (j) XLi, (k) M-HOX, (l) WM-HOX, (m) M-HOI, (n)WM-HOI, (o) CSI and (p) WM_CSI.

42 / 44



– Comparison of proposed method

43 / 44

Table 3. Image Quality Performance Measures for The Proposed InterpolationMethod-PSNR dB AND NCD .

3( 1 0 )

−×



Conclusion

Proposed interpolation technique – Application of Taylor series and cubic spline for CFA

demosaicking with high accuracy

• Deriving by using new interpolation equation

– Weighted median filter• Use of classifier based on edge orientation map

Experiment of proposed method

– Assessing performance of proposed method

• Comparison of previous method and proposed method

– Analysis of various demosaicking method quantitatively

• Measuring image quality performance for proposed method

Documents

20100508 Color Filter Array Demosaicking Using High-Order Interpolation Techniques With a Weighted Median Filter for Sharp Color Edge Preservation