Quantitative Comparison of H.264/AVC and VP8 Video

Codecs for Video ConferencingScott KristjansonCMPT-820Multimedia SystemsInstructor: Dr. Mohamed Hefeeda Dec 2nd 2010

Scott Kristjanson – CMPT 820 - SFU

2

Slide 2

Presentation Overview

Project Overview Methods Results Conclusions Future Work References

Scott Kristjanson – CMPT 820 - SFU

3

Slide 3

Project Overview

Compare Video Codecs for use with Video Conferencing:• VP8 – by Google’s WebM project • H.264/AVC – used x256 Open Source Codec

Scott Kristjanson – CMPT 820 - SFU

4

Slide 4

VP8 Overview

Video Codec owned by Google and made open Source• WebM Project: Matroska container with VP8 video and Vorbis audio • VP8 - Earlier versions VP7 (Skype), VP6 (Flash Video), VP3 (Theora)

Key Features:• Adaptive Loop Filtering – Identifies Fast/Slow/Static regions and

adapts encoding to match for improved quality/performance• Golden Frames – for storing reference point to background layer. Can

be used for reference without display them.• Faster encoding time by avoiding use of Arithemetic VLE (CABAC).• Dynamically scales resolution. No need to involve application,

encoder decides. No I-frame required as a transition point.

Structure:• Frame same as H.264, Segment like a Slice• Partitions used to separate critical data from video data• I-Frame, P-Frames, Golden-Frames, no B-Frames due to Patents

For more information, see Google’s “Intro to VP8” slides at:http://www.slideshare.net/DSPIP/google-vp8

Scott Kristjanson – CMPT 820 - SFU

5

Slide 5

Methods

Quantitative Comparison: Invoke encoders/decoders with various presets:• Manually from DOS, then use YUV_Compare to compute Y-PSNR • ViCoS Codec Comparison utility from MSU – gives ranking only• Used sequences foreman and salesman to simulate video conference calls• Encode with bit rates 100, 800, 1500 kbps to produce bit rate curves

Extract Quantitative Data• Y-PSNR – Signal to Noise Ratio for Luminosity• Y-SSIM – Structural Simularity Data for Luminosity• Encoding Time• Frame Rate• Bit Rate

Qualitative Comparison

• View original YUV file and encoded/decoded files using YUVviewer• Test VP8 in Video Conferencing System to verify Quantitative Data

Scott Kristjanson – CMPT 820 - SFU

6

Slide 6

Reference Video Sequences

Salesman • Ideal choice for simulating video conference• Not used since not available in CIF format• Tried to convert from .avi to CIF format but ran out of time

Foreman• Good choice, but with some background movement• Used in detailed quantitative analysis• Used with Vicos for codec over-all evaluation

Mobile• Slow motion with detailed cartoon-like background• Used with Vicos for codec over-all evaluation

Stefan• Lots of motion and lots of background detail• Used with Vicos for codec over-all evaluation

Scott Kristjanson – CMPT 820 - SFU

7

Slide 7

DOS Command Line Testing

x264 encode/decode:

vp8 encode/decode:

Visually compare original video toencoded video

with YUV_Viewer

Scott Kristjanson – CMPT 820 - SFU

8

Slide 8

Computing Y-PSNR with AviSynth

AVS script to read and compare two YUV files:• Encoded/Decoded YUV video from DOS command line testing• Original YUV video

yuv_compare_foreman.avs

foreman_yuv_compare.log

loadplugin("C:\megui\tools\avisynth_plugin\rawsource.dll")decodedClip = RawSource("C:\foreman_352x288_decoded_vp8.yuv",352, 288, "I420")originalClip = RawSource("C:\foreman_352x288.yuv", 352, 288, "I420")Compare(decodedClip, originalClip, "YUV", "foreman_yuv_compare.log")

Mean Max Max Absolute Mean Pos. Neg. Frame Dev. Dev. Dev. Dev. PSNR (dB) ----------------------------------------------------- 0 0.7591 +0.0252 5 -3 47.6277 1 0.7566 -0.0050 5 -4 47.6655 … … … … … … 299 0.8597 +0.0070 4 -4 46.8949Total frames processed: 300 Minimum Average MaximumMean Absolute Deviation: 0.6941 1.2438 3.7930 Mean Deviation: -0.2703 -0.0194 +0.0527 PSNR: 33.2940 44.9776 48.2338 Overall PSNR: 41.3285

PSNR

Scott Kristjanson – CMPT 820 - SFU

9

Slide 9

MSU’s Video Codecs Scoring System (ViCoS)

Allows custom Codecs to be bound in and tested• bind in encoders and decoders (x264 and vp8)• define encoding and decoding parameters and presets to be tested• encodes/decodes test sequences automatically and computes distortion• generates relative ranking and relative scores for PSNR and SSIM only

Versions:• Demo version used – provides ranking scores only • Professional version – creates PNSR/SSIM charts automatically

Scott Kristjanson – CMPT 820 - SFU

10

Slide 10

Video Quality – Peak Signal to Noise Ratio (Y-PSNR)

25

27

29

31

33

35

37

39

41

43

100 800 1500

Bit Rate (kbps)

Y-P

NS

R (d

b)

VP8 Bestx264 PlaceboVP8 Good1VP8 Good2VP8 Goodx264 FastVP8 Good4x264 Defaultvp8 1Pass Fast VBRVP8 RT CBR w DLVP8 RealTime CBRx264 fast zero latencyx264 Baselinex264 ultraFast

Bit-Rate/Quality Trade-Off:• 100kbps – Poor Quality• 800kbps – Acceptible to good• 1500kbps – Very Good to Excellent

PSNR Ranges:38+ Excellent 36-37 Very good34-35 Good26-33 Poor25 or less Garbage

Scott Kristjanson – CMPT 820 - SFU

11

Slide 11

Video Quality – Structural Simularity (SSIM)

0.60

0.70

0.80

0.90

1.00

100 800 1500

Bit Rate kbps

Y-SS

IM

VP8 Bestx264 PlaceboVP8 Good1VP8 Good2VP8 Goodx264 FastVP8 Good4x264 DefaultVP8 Fast VBR 1PassVP8 RT CBR w DLVP8 RealTime CBRx264 fast zero latencyx264 Baselinex264 ultraFast

SSIM score under 0.95 is quite noticible:• 100kbps – Poor Quality for all codecs

(especially x264 Fast Zero-Latency & Ultrafast)• 800kbps – most codecs Acceptible to good

(except VP8 RT CBR and x264 Ultrafast)• 1500kbps – all Very Good

(except x264 Ultrafast)

Scott Kristjanson – CMPT 820 - SFU

12

Slide 12

Encoder Frame Rates

x264 Encoder significantly faster than VP8 for similar Quality

0

15

30

45

60

75

90

105

120

135

150

165

180

195

210

225

240

255

VP8 100 x264 VP8 800 x264 VP8 1500 x264

Bit Rate (kbps)

Fram

es p

er S

econ

d

VP8 BestVP8 GoodVP8 Good1VP8 Good2VP8 Good4VP8 Fast VBR 1PassVP8 RT CBR w DLVP8 RealTime CBR---------------------x264 Placebox264 Fastx264 Defaultx264 fast zero latencyx264 Baselinex264 ultraFast

Scott Kristjanson – CMPT 820 - SFU

13

Slide 13

Good Codec Presets for Video Conferencing

Need min of 15 fps for Video Conferencing, prefer 30 fpsWant decent quality – Y-PSNR of 34 or betterLow Latency: under 150ms required4 Presets met these requirements:

Codec Preset

Scott Kristjanson – CMPT 820 - SFU

14

Slide 14

Conclusion

VP8 and x264 provide similar video quality• For higher bit rates, x264 PSNR better by 2%• For lower rates, VP8 wins by almost 10%

x264 provides better performance in some cases• x264 is between 36% and up to 2x faster than VP8• A big improvement! VP8 used to be 5-20 times slower!

Scott Kristjanson – CMPT 820 - SFU

15

Slide 15

References:

[1] Image Quality Assessment: From Error Visibility to Structural SimilarityZhou Wang, Alan Conrad Bovik, Hamid, Rahim Sheikh, and Eero P. SimoncelliIEEE Transactions on Image Processing, Vol 13, No. 4, April 2004http://ieeexplore.ieee.org.proxy.lib.sfu.ca/stamp/stamp.jsp?tp=&arnumber=1284395

[2] 6th Annual MPEG-4 AVC H.264 video codecs comparisonD. Vatolin, D. Kulikov, and A. Parshin, April 2010http://compression.ru/video/codec_comparison/h264_2010/

Scott Kristjanson – CMPT 820 - SFU

16

Slide 16

Backup Slides

Scott Kristjanson – CMPT 820 - SFU

17

Slide 17

H.264/AVC Presets Used

Codecs tested with various presets and tunings• Looking for good balance of quality and speed for video conferencing• Each tested with 3 target Encoding bit rates: 100, 800, 1500 kbps

x264 Presets:• Placebo

• 2-Pass Slower Encoding Speed, higher quality• Fast (tested with 2 tunings)

• 2-Pass Faster Encoding Speed, good quality• 2-Pass tuned with zero latency specified

• Default • 1-pass with default settings

• Baseline• 1-pass fast encoding but poor quality at lower bit rates

• Ultrafast• When speed is all that matters!

Scott Kristjanson – CMPT 820 - SFU

18

Slide 18

VP8 Presets Used

Best• 2-Pass encoding• Maximimizes Quality but slower encode times

Good 2-Pass• 2-Pass encoding, less quality but faster encode times • measured with 1, 2, and 4 cpu cores specified

Good 1-Pass• 1-Pass encoding, less quality but faster encode times

Fast VBR• 1-Pass variable bit-rate fast encoding

RealTime CBR • Encoder Frame buffer size 6• with and without deadline specified

Scott Kristjanson – CMPT 820 - SFU

19

Slide 19

VP8 Support for MultiCore Processors

VP8 Encoding Rate vs # of CPUs Preset Used: "Good"

0

10

20

30

40

50

60

70

80

90

100 800 1500

Bit Rate (kbps)

Fram

es P

er S

econ

d

1 CPU2 CPUs4 CPUs

Scott Kristjanson – CMPT 820 - SFU

20

Slide 20

Encoding Bit Rate vs Target Bit Rate

Both Codecs appear to over-run target slightlyVP8 tracks to target bit-rate slightly better

Actual versus Target Bit Rate

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

1400

1500

1600

1700

1800

1900

1 2 3 4 5 6 7 8

Codec

Targ

et B

it R

ate

VP8 1500 kbpsVP8 800 kbpsVP8 100 kbpsx264 1500 kbpsx264 800 kpbsx264 100 kbps

Scott Kristjanson – CMPT 820 - SFU

21

Slide 21

Video Quality – Structural Simularity (SSIM)

0.6

0.7

0.8

0.9

1.0

Y-S

SIM

100 kbps800 kbps1500 kbps

SSIM score under 0.95 is quite noticible:• 100kbps – Poor Quality for all codecs (especially x264 Fast 0-Latency and Ultrafast)• 800kbps – most codecs Acceptible to good (except VP8 RT CBR and x264 Ultrafast)• 1500kbps – all Very Good