Overview of Error Overview of Error Resiliency Schemes in Resiliency Schemes in H.264/AVC StandardH.264/AVC Standard

Overview of Error Overview of Error Resiliency Schemes in Resiliency Schemes in H.264/AVC StandardH.264/AVC Standard

Sunil Kumar, Liyang Xu, Mrinal K. Mandal, Sunil Kumar, Liyang Xu, Mrinal K. Mandal, and Sethuraman Panchanathanand Sethuraman Panchanathan

Elsevier Journal of Visual Communication and Image Representation, to appear in2005.

S. Wenger, “H.264/AVC over IP,” IEEE Trans. Cir. Syst. Video Technol., vol. 13, pp. 645–656, July 2003

Outline• Introduction• Error-Resilience Tools in

H.264/AVC• Experimental Results• References

Introduction• H.264/AVC Video Coding Standard

– http://vc.cs.nthu.edu.tw/home/paper/codfiles/cycho/200310132110/Overview_of_the_H.264_AVC_Video_Coding_Standard.ppt

Introduction• Error-resilience tools in older video c

ompression standard– Picture segmentation (slices, GOBs);– Placement of Intra MBs, Intra slices, and i

ntra pictures;– Reference picture selection (with and wit

hout feedback);– Data partitioning

MPEG 1 part 2MPEG 2 part 2H.261H.263

H.263

Error-Resilience Tools in H.264/AVC

• Reference picture selection– Feedback-based

– Without feedback• e.g. Video redundancy coding

• Picture segmentation– Slice

• Flexible macroblock ordering (FMO)– 7 modes– Overhead bits highly depends on the picture forma

t, the content, and the QP. (S. Wenger and M. Horowitz)

• < 5% penalty at QP = 16; on average 20% at QP = 28.

Encoder Decoder

feedback channel

I0p1 p3 p5 p7

p9

p2 p4 p6 p8Channel 1

Channel 2

Checker board modeInterleaving mode

Error-Resilience Tools in H.264/AVC

• SP/SI frame http://vc.cs.nthu.edu.tw/home/paper/codfiles/cycho/200402181718/The_SP-_and_SI-Frames_Design_for_H.264_AVC.ppt

– Instantaneous refreshing– Gradual refreshing

P1, n-2 P1, n-1 S1, n P1, n+1

S 21, n

SI1, n

P1, n-3

Error-Resilience Tools in H.264/AVC

• Intra placement– Used to combat drifting effects.– ConstrainedIntraPrediction flag

• An Intra block can not be predicted from an Inter block.

– IDR slice and IDR picture• An IDR picture contains only IDR slices.• An IDR slice can only be part of an IDR picture.• An IDR picture invalidates all short-term reference me

mory buffers.

Intra

Intra

inter

Intra

I P IDR P

Error-Resilience Tools in H.264/AVC

• Data partitioning– For unequal error protection (UEP).– Three different partition types in H.264.– Type A

• Header information (MB types, QP, MVs)– Type B

• Intra coded block patterns (CBPs) and Intra coefficients– Type C

• Inter CBPs and Inter coefficients– Importance

• A > B > C

Error-Resilience Tools in H.264/AVC

• Redundant slices (RSs)– For a MB, an encoder can place redundant repr

esentation of the same MBs into the same it stream.

– e.g.• One slice is coded using different QP.• If the slice of low QP is available, the decoder discard

s the RS; otherwise, the RS is reconstructed by the decoder

Slice AQP1

Slice AQP2

Decoder

Error-Resilience Tools in H.264/AVC

• Parameter sets– Including picture size, entropy coding method,

MV resolution, and so on.– Sequence parameter set

• Containing all information related to the picture sequence between two IDR pictures.

– Picture parameter set• Containing all information related to all slices in a pic

ture.– e.g. Sending multiple copies of SPSs so to enha

nce the arrival rate.– e.g. SPSs can be sent out-of-band.

Error-Resilience Tools in H.264/AVC

• Intra block refreshing by RD control– – Better than periodically inserting intra M

Bs.

))()((minarg* oRoDoXo

Experimental Results

Stephan Wenger

Experimental Results• Error concealment scheme

– Intra

– Inter• }|)mv(|{

1},,,{

argmin NYYdN

j

OUTj

INj

dirsm

rightleftbottopdir

foremanparis

Stephan Wenger

Experimental Results• Experiment 1: One picture, one

packet, without any error-resilience– Packet loss rates > 3% unprotected

H.264 video becomes unusable.Foreman

Paris

Stephan Wenger

Experimental Results• Experiment 2: One picture, one pa

cket, with intra macroblock refresh– Very good PSNR results– Very high amount of Intra MBs of signi

ficant size• A coarser quantizer is needed for CBR – l

ess spatial detail.

Stephan WengerForeman

PSNR = 35

PLR = 0%

Experimental Results• Experiment 3: 2-4 slices in one

frame– Unfavorable PSNR results for

Foreman.– 2nd best PSNR results for Paris.

ForemanParisPacket header overhead

?

Stephan Wenger

Experimental Results• Experiment 4: Slices interleaving

– Better than Exp. 3 for Foreman.– Worse than Exp. 3 for Paris.

ForemanParis

Stephan Wenger

Experimental Results• Experiment 5: Data partitioning

– Partition A is sent twice (3% PLR) or 3 times (>3% PLR).• Degraded PSNR due to CBR.

ForemanParis

Stephan Wenger

Experimental Results• Experiment 6: FMO

– Randomly inserting Intra MB. (3% of PLR)

– Best PSNR results

ForemanParis

Stephan Wenger

Experimental Results

QCIF, 10fps,100Kbps

M. M. Ghandi and M. Ghanbari

Experimental Results

QCIF, 10fps,100Kbps

M. M. Ghandi and M. Ghanbari

References• S. Kumar, L. Xu, M. K. Mandal, and S. Panchanathan, “Overview of

Error Resiliency Schemes in H.264/AVC Standard,” Elsevier Journal of Visual Communication and Image Representation (Special issue on H.264/AVC Video Coding Standard), to appear in 2005.

• S. Wenger, “H.264/AVC over IP,” IEEE Trans. Cir. Syst. Video Technol., vol. 13, pp. 645–656, July 2003.

• S. Wenger and M. Horowitz, “Scattered slices: A new error resilience tool for H.26L,” in JVT-B027, Feb 2002.

• M. M. Ghandi and M. Ghanbari, “Layered H.264 video transmission with hierarchical QAM,” Elsevier J. of Visual Communication and Image Representation (Special issue of H.264/AVC), to appear in 2005.