45
A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

Embed Size (px)

Citation preview

Page 1: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

A frequency domain approach for Intra-coding pictures

Presenter: Andy C. Yu

forMicrosoft Research Cambridge

16th May 2006

Page 2: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

Outline:

• Introduction - H.264/AVC Intra coding v. Motion JPEG-2000 coding.

• The proposed Fintra algorithm – based on frequency domain

• Simulation results – comparison with state-of-the-art algorithms

• Conclusions

Page 3: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

Introduction

• H.264/AVC is standardised by ITU-T and MPEG ISO/IEC in Dec. 2001 for commercial video coding.

• It represents a major step forward in the development of video standards*.

• It typically outperforms all existing standards by a factor of two.

• Another important fact is that H.264/AVC is a public and open standard*.

* R. Schäfer, T. Wiegand and H. Schwarz, “The emerging H.264/AVC standard”, EBU Technical Review, Jan 2003

Page 4: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

Conventional Intra-coding structure

VideoSequence 8x8 DCT

Quantization Process(Lossy)

EntropyCoding

8

Decomposition

8

Page 5: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

H.264/AVC Intra-coding structure

VideoSequence

Integer 4x4 DCT &Quantisation Process

EntropyCoding

Decomposition

4

4

Predictedblock

Originalblock

Residue Data

Mode Selection

Page 6: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

Intra-Prediction in H.264/AVC

A B C D E F G HQIJKLMNOP

a b c de f g hi j k l

m n o p

A : Neighbouring sample that are already reconstructed at the encoder and decoder sides.

a : Samples to be predicted.

Page 7: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

Picture from I. Richardson, H.264 and MPEG-4 Video Compression, Willey Publisher, 1st edition, 2003.

Page 8: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

Picture from I. Richardson, H.264 and MPEG-4 Video Compression, Willey Publisher, 1st edition, 2003.

Page 9: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

Lagrangian Evaluation• Final mode decision is selected to minimise

Lagrangian cost.

• Where D: measure of the distortion

R: the number of bits.

λ: Lagrange parameter.

Qp: Quantisation factor.

)Qp|mode(.)Qp|mode(Qp)|mode( RDJ

Page 10: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

0 10 20 30 40 500

1000

2000

3000

4000

5000

6000

Quantisation factor, Qp

Lag

rang

ian

pa

ram

ete

r

Lagrangian evaluation (cont.)

3/)12Qp(285.0

Page 11: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

H.264 Intra-coding v. Motion JPEG2000 coding

Picture from J. Ostermann, J. Bormans et al., “Video Coding with H.264/AVC: Tools, Performance, and Complexity,” IEEE Circuits and Systems Magazine, first quarter, 2004

Page 12: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

The Proposed Fast Intra-coding Algorithm (Fintra)

Page 13: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

H.264/AVC Intra-coding structure

VideoSequence

Integer 4x4 DCT &Quantisation Process

EntropyCoding

Decomposition

4

4

Predictedblock

Originalblock

Residue Data

Mode Selection

Page 14: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

H.264/AVC Intra-coding structure

VideoSequence

Integer 4x4 DCT &Quantisation Process

EntropyCoding

Decomposition

4

4

Predictedblock

Originalblock

Residue Data

Mode Selection

Page 15: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

The search strategy for the proposed Fintra algorithm

• The proposed algorithm selects fewer modes to undergo full Lagrangian cost evaluation.

• The entire selection process operates in the discrete cosine transform (DCT) domain.

• Generally, larger residue block energy is produced by intra-coding.

• It is observed that the modes that provide the least residue energy will result in minimum R and hence minimise the Lagrangian cost.

Page 16: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

Algorithm formulation• The selection criterion can be measured

from the SAD of residue block in Discrete Cosine Transform domain:

The definition of 2D-DCT is:

3

0

3

0modee)DCT(residu )},(),({SAD

m n

nmnm PBT

1

0

1

0 2

)12(cos

2

)12(cos),()()(

22 N

n

M

muv N

vn

M

umnmvcuc

NMF

B

Page 17: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

• Thus, the selection is the least residue energy produced by the mode.

3

0

3

0mode

mode),(),(minarg

m n

nmnm PBT

15

1mode|mode|

mode)(AC)(ACDCDCminarg

i

ii PBPB

• We separate the low frequency spectrum, DC, from other AC frequencies.

Page 18: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

• DC and low-frequency AC coefficients contain more energy than the high-frequency coefficients.

15

mode|mode|mode

)(AC)(ACDCDCminargi

ii PBPB

Task: Develop efficient approach to calculate DCP|mode and ACP|mode(i)

Page 19: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

A B C D E F G HQIJKLMNOP

a b c de f g hi j k lm n o p

A

Page 20: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

ABC…

……

..

OPQ

17x1

Page 21: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

ABC…

……

..

OPQ

ω1,0 ω1,1

ω2,0

….

ω8,0

ω9,0

…. ….

…. ….ω9,1

ω1,17

ω2,17…

.

ω8,17

ω9,17

ω1,16

ω9,16

….

….

….

….

17x1

.

9x17

Page 22: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

ABC…

……

..

OPQ

ω1,0 ω1,1

ω2,0

….

ω8,0

ω9,0

…. ….

…. ….ω9,1

ω1,17

ω2,17…

.

ω8,17

ω9,17

ω1,16

ω9,16

….

….

….

….

17x19x17

DCP|mode0

…...

DCP|mode1

DCP|mode7

DCP|mode8

9x1

= .

Page 23: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

ABC…

……

..

OPQ

ω1,0 ω1,1

ω2,0

….

ω8,0

ω9,0

…. ….

…. ….ω9,1

ω1,17

ω2,17…

.

ω8,17

ω9,17

ω1,16

ω9,16

….

….

….

….

17x19x17

DCP|mode0

…...

DCP|mode1

DCP|mode7

DCP|mode8

9x1

= .PDC DC

~ N

Page 24: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

N DCP~

DC

Ni i )AC(P

~)AC(

Page 25: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

15mode|mode|

mode

3

0

3

0mode

mode

)(AC)(ACDCDCminarg

),(),(minarg

i

m n

ii

nmnm

PBPB

PBT

Thus, the selection process,

15

)AC(DCrow

~)(AC

~DCminarg

iii NΩINΩI BB

becomes

Page 26: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

1.3 0.5 -1.3-0.5 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0.1 0.10.2 0 -0.1 -0.2 -0.1 0 0.1 0.2 0.1 0 -0.1 -0.2 -0.1 0

A B DC E F G H I J K L M N O P Q

0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0

0.2 -0.5 -0.2-0.7 0 0 0 0 0.4 0.2 0.1 0 0 0 0 0 0.5

0.1 0.5 -0.20.5 -0.7 -0.4 0.1 0 0 0 0.1 0.1 0 0 0 0 0

0 0 0.10.1 0 0 0 0 0.1 0.2 0.1 0 -0.2 -0.3 -0.1 0 0

-0.3 -0.4 -0.1-0.3 0 0 0 0 0.3 0.4 0.3 0.1 0 0 0 0 0

-0.2 -0.2 0-0.1 0 0 0 0 0 0.1 0.3 0.2 0 0 0 0 -0.1

Mode 0

Mode 1

Mode 2

Mode 3

Mode 4

Mode 5

Mode 6

Mode 7

Mode 8

AC(1,0)

~

Page 27: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

Compared with 2D-DCT ?

1

0

1

0 2

)12(cos

2

)12(cos),()()(

22 N

n

M

muv N

vn

M

umnmvcuc

NMF

B

General practice: apply 1D-DCT twice, row-wise and column-wise ?

Page 28: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

1.3 0.5 -1.3-0.5 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0.1 0.10.2 0 -0.1 -0.2 -0.1 0 0.1 0.2 0.1 0 -0.1 -0.2 -0.1 0

A B DC E F G H I J K L M N O P Q

0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0

0.2 -0.5 -0.2-0.7 0 0 0 0 0.4 0.2 0.1 0 0 0 0 0 0.5

0.1 0.5 -0.20.5 -0.7 -0.4 0.1 0 0 0 0.1 0.1 0 0 0 0 0

0 0 0.10.1 0 0 0 0 0.1 0.2 0.1 0 -0.2 -0.3 -0.1 0 0

-0.3 -0.4 -0.1-0.3 0 0 0 0 0.3 0.4 0.3 0.1 0 0 0 0 0

-0.2 -0.2 0-0.1 0 0 0 0 0 0.1 0.3 0.2 0 0 0 0 -0.1

Mode 0

Mode 1

Mode 2

Mode 3

Mode 4

Mode 5

Mode 6

Mode 7

Mode 8

AC(1,0)

~

General Practice: 0.1 X (B + D + J + L – F – H – N - P) + 0.2 X (C + K - G - O) 2 Multiplication + 11 Addition operators

Page 29: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

Features of conversion matrix Ω

• Uniquely existed at each position in frequency domain.

• Independent of the video sequences.

• Low computational demand.

• The coefficients can be calculated and stored in advance.

Page 30: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

We may select at least 2 modes to undergo the full Lagrangian

process.

Match Percentage between the least residue energy and the least rate-distortion cost

25

35

45

55

65

75

85

95

1 2 3 4Number of candidates

Mat

ch P

erce

nta

ge

(%)

Akiyo (Class A)

Foreman (Class B)

Mobile (Class C)

Page 31: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

Simulations and Results• Comparison of the results with the two best

performing algorithms detailed in the literatures.

• Simulation Settings– JM6.1e version in Microsoft Visual C++– Various values of M are selected.

• Measurements:picture quality (dB), compression ratio (Kbits/sec) and entire encoding time (sec).

Page 32: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

Simulation 1:

Compared with

C. Kim, H. Shih, and C. Kuo, “Fast H.264 Intra-prediction mode selection using joint spatial and transform domain features”, Journal of Visual Communication and Image Representation, vol. 17, pp. 291-310, 2006.

by selecting M=3

Page 33: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

Foreman QCIF 30Hz 300 frames All Intra-coding

0 200 400 600 800 1000 1200 1400 1600 180026

28

30

32

34

36

38

40

42

44

Rate (Kbits/sec)

PS

NR

(dB

)

JM6.1eProposedResult in [6]

Page 34: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

0 200 400 600 800 1000 1200 1400 1600 180026

28

30

32

34

36

38

40

42

44

Rate (Kbits/sec)

PS

NR

(dB

)

JM6.1eProposedResult in [6]

Perception sensitivity

area

Foreman QCIF 30Hz 300 frames All Intra-coding

Page 35: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

0 200 400 600 800 1000 1200 1400 1600 180026

28

30

32

34

36

38

40

42

44

Rate (Kbits/sec)

PS

NR

(dB

)

JM6.1eProposedResult in [6]

Foreman QCIF 30Hz 300 frames All Intra-coding

Page 36: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

300 320 340 360 380 400 420

30

30.5

31

31.5

Rate (Kbits/sec)

Foreman QCIF 30Hz 300 frames All Intra-coding

JM6.1eProposedResult in [6]

Page 37: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

300 320 340 360 380 400 420

30

30.5

31

31.5

Rate (Kbits/sec)

Δ1≈0.1dB

Δ2≈0.3dB

Foreman QCIF 30Hz 300 frames All Intra-coding

JM6.1eProposedResult in [6]

Page 38: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

Simulation 2:

F. Pan, X. Lin, et al., “Efficient prediction mode selection 4x4 blocks in H.264,”JVT-013, ISO/IEC JTC1/SC29/ WG11 and ITU-T SG16 Q.6, Mar. 2003, Pattaya, Thailand.

F. Pan, X. Lin, et al., “Fast mode decision algorithm for Intraprediction in H.264/AVC video coding,” IEEE transaction on Circuits and Systems for Video Technology, pp. 813-823, Jul. 2005.

by selecting M=2

Page 39: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

SequencesPSNR difference Bit rate difference Speed up c.f. JM6.1e

Pan et al Proposed Pan et al Proposed Pan et al Proposed

Bus (CIF) -0.22 dB -0.12 dB 3.85 % 1.46 % 58.12 % 75.22 %

Coastguard -0.11 dB -0.06 dB 2.36 % 1.68 % 55.03 % 72.48 %

Container -0.23 dB -0.10 dB 3.70 % 0.77 % 56.36 % 71.86 %

Foreman -0.29 dB -0.07 dB 4.44 % 1.97 % 65.38 % 72.47 %

News -0.29 dB -0.09 dB 3.90 % 1.18 % 55.39 % 73.13 %

Paris (CIF) -0.23 dB -0.09 dB 3.21 % 1.47 % 57.78 % 73.43 %

Silent -0.18 dB -0.05 dB 3.54 % 2.00 % 65.17 % 71.86 %

Page 40: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006
Page 41: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

Conclusions

• A fast algorithm using matrix formulae in the DCT domain is presented.

• The proposed Fintra algorithm does not need any a priori knowledge or magic number.

• The simulation results verify that the proposed algorithm outperforms two other algorithms in terms of all measurements.

• The rate-distortion curves show that the proposed algorithm achieves the same coding performance, yet reduces the computation requirement by up to 75%.

Page 42: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

Future Work• The proposed Finter algorithm has been

developed to achieve speed up for encoding P- and B- frames.

• The Finter algorithm produces competitive results to other algorithms recorded in the literatures.

• An integration for both proposed Fintra and Finter algorithms has been done to attain a time of saving up to 86% without sacrificing both picture quality and bit rate efficiency.

Page 43: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

References[1] Andy C. Yu, Ngan King Ngi, and Graham Martin, “Efficient Intra- and Inter-mode Selection Algorithms for

H.264/AVC,” in Journal of Visual Communication and Image Representation – Special Issue on Emerging H.264/AVC Video Coding Standard, vol. 17, issue 2, pp. 322-343, Elsevier Press, Apr 2006.

[2] Andy C. Yu, Graham Martin, and Heechan Park, “Improved Schemes for Inter-frame Coding in the H.264/AVC Standard,” in Proc. of 12th IEEE Conference on Image Processing (ICIP) 05, vol. 2., pp. 902-905, Genoa, Italy, Sep. 2005.

[3] Andy C. Yu, Graham Martin, and Heechan Park, “A Frequency Domain Approach to Intra Mode Selection in H.264/AVC,” in Proc. of 13th European Signal Processing Conference (EUSIPCO) 05, 4pp., Antalya, Turkey, Sep. 2005

[4] Andy C. Yu and Graham Martin, “Advanced Block Size Selection Algorithm for INTER frame Coding in H.264/AVC,” in Proc. of 11th IEEE International Conference on Image Processing (ICIP) 04, vol. 1, pp.95-98, Singapore, Oct 2004.

[5] Andy C. Yu, “Efficient Block Size Selection Algorithm for INTER frame Coding in H.264/AVC,” in Proc. of 29th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) 04 , vol. 3, pp. 169-172, Montreal, Canada, May 2004.

[6] F. Pan, X. Lin, et al., “Fast mode decision algorithm for Intraprediction in H.264/AVC video coding,” IEEE transaction on Circuits and Systems for Video Technology, pp. 813-823, Jul. 2005.

[7] C. Kim, H. Shih, and C. Kuo, “Fast H.264 Intra-prediction mode selection using joint spatial and transform domain features”, Journal of Visual Communication and Image Representation, vol. 17, pp. 291-310, 2006.

[8] __, “Information technology – coding of audio visual objects – Part 10: advance video coding,” ISO/IEC 14496-10:2003, Dec. 2003.

[9] JVT reference software, JM6.1e, downloaded from http://bs.hhi.de/~suehring/

Page 44: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006

Q & A

Page 45: A frequency domain approach for Intra-coding pictures Presenter: Andy C. Yu for Microsoft Research Cambridge 16 th May 2006