2011 01 ICCE Video Developments Garysull d2

8/3/2019 2011 01 ICCE Video Developments Garysull d2

1/40

Developments in Digital Video &

Related Standardization Efforts

(especially the new HEVC initiative)

Gary J. Sullivan

IEEE Intl. Conf. on Consumer Electronics10 January 2011


2/40

1990 1996 20021992 1994 1998 2000

H.263

(1995-2000+)

MPEG-4 Visual

(1998-2001+)MPEG-1

(1993)

ISO/IEC

ITU-T

H.120(1984-1988)

H.261

(1990+)

H.262 / MPEG-2(1994/95-1998+)

H.264 / MPEG-4

AVC

(2003-2009)

Chronology of International

Video Coding Standards

2004


3/40

Video Coding Standards Organizations

ISO/IEC MPEG = Moving Picture Experts Group(ISO/IEC JTC 1/SC 29/WG 11 = International Standardization Organization andInternational Electrotechnical Commission, Joint Technical Committee Number 1,Subcommittee 29, Working Group 11)

ITU-T VCEG = Video Coding Experts Group(ITU-T SG16/Q6 = International Telecommunications Union Telecommunications

Standardization Sector (ITU-T,a United Nations Organization, formerly CCITT),Study Group 16, Working Party 3, Question 6)

JVT = Joint Video Team collaborative team of MPEG & VCEG

SMPTE (Society for Motion Picture and Television Engineers) has also

issued some video coding standards. New: JCT-VC = Joint Collaborative Team on Video Coding team of MPEG

& VCEG, continuing the collaborative relationship for a new project(established January 2010)


4/40

The Scope of Picture and Video Coding

Standardization

Only the Syntaxand Decoderare standardized:

Permits optimization beyond the obvious

Permits complexity reduction for implementability

Provides no guarantees of Quality

Pre-Processing EncodingSource

Destination

Post-Processing

& Error Recovery

Decoding

Scope of Standard


5/40

H.264 / 14496-10 AVC Structure

Entropy

Coding

Scaling & Inv.

Transform

Motion-Compensation

Control

Data

Quant.

Transf. coeffs

Motion

Data

Intra/Inter

Coder

Control

Decoder

Motion

Estimation

Transform/

Scal./Quant.-

Input

VideoSignal

Split into

Macroblocks

16x16 pixels

Intra-frame

Prediction

Deblocking

Filter

Output

VideoSignal


6/40

Example Compression Comparison

0 100 200 300

28

30

32

34

36

38

40

Rate [kbit/s]

PSNR

[dB] Half-pelmotion compensation

(MPEG-1 1993

MPEG-2 1994)

Integer-pel

motion

compensation

(H.261, 1991)

Variable block size

(16x16 8x8)

(H.263, 1996) +

quarter-pel

motion compensation

(MPEG-4, 1998)

Variable block size

(16x16 4x4) +

quarter-pel +

multi-frame

motion compensation

(H.264/AVC, 2003)

Foreman

10 Hz, QCIF100 frames

Good

Picture

Quality

BadPicture

Quality


7/40

ITU-T H.264 / ISO/IEC 14496-10 MPEG-4

AVC Basic Milestones

First version of standard May 2003

Fidelity range extensions (incl. High Profile) Mid 2004

Extended-gamut color spaces: Mid 2006

Professional profiles: Mid 2006

Scalable Video Coding (SVC) Extension: Fall 2007

Multi-view Video Coding (MVC) Extension: Fall 2008


8/40

Scalable Video Coding (SVC)

scene

SVCencoder

SVC

decoder

SVC

decoder

SVC

decoder

H.264/AVC

decoder

128

kbit/s

256

kbit/s

512

kbit/s

1024

kbit/s

CIF@

30 Hz

CIF@

15 Hz

QCIF@

7,5 Hz


9/40

1st Concept: Temporal Hierarchy

I P P P P P P P P

B0

B0

B0

B0

Temporal

Scalability

B0

B0

B1

B1

B1

B1

B0

B1

B1

B2

B2

B2

B2

N=1

I P P P P

N=2

I P P

N=4

I P

N=8


10/40

2nd New SVC Concept: Use other Data

Older scalable designs (MPEG-2, H.263+, MPEG-4 Part 2)used only decoded pictures to predict enhancement layers

Here, other information is also used Motion vectors

Partitioning

Residual signal

Intra- and Inter-picture coded regions treated distinctly

Result: Capability enhancement

Less need for decoded picture values

Opportunity for complexity reduction


11/40

3rd New Concept: Single-loop Decode

Past spatial scalable video standards: Inter-layer intra-picture prediction requires that base layer is

completely reconstructed

Decode using multiple motion compensation loops

(and deblocking filter applications) Decode complexity greater than simulcast: multi-layer full decoding

plus inter-layer prediction

In SVC: Inter-layer intra prediction is restricted tomacroblocks for which the co-located base layer signal is

intra-coded No need to decode multiple motion compensation loops

Complexity comparable to single-layer coding

Common building blocks with non-scalable design


12/40

SVC Industry Adoption

Real-time communication Vidyo implementation licensed by Google, Cisco, others

Unified Communications Interop Forum Founders: HP, Juniper Networks, Logitec / LifeSize, Microsoft, Polycom

Contributors: Broadcom, ClearOne, Ericsson, Jabra, Network Equip.Tech., Plantronics, Radvision, Siemens, Teliris, Vidyo

Adopters: Acme Packet, Alcatel-Lucent, AMD, Aspect, AudioCodes,Avistar, Broadsoft, Brocade, Crestron Electronics, , Dialogic, EdgewaterNetworks, Glowpoint, Immedia Semiconductor, Intel, News Corp,Quanta Computer, Sonus Networks, Sunplusit, TI, Voss, VXI

Broadcast Adopted by DVB

Expected for 1080p60 enhancement of 720p60

Mass-market chips expected


13/40

Stereo 3D / Multiview Video Coding (MVC)

Stereo 3D is a major industry movement

Avataris the top grossing film ever(not inflation adjusted)

Most of its theater revenue was from 3D screenings

3D is a major factor in the theater revenue stream

3D commands premium pricing Stereo 3D is a hot feature for display vendors

Supported in HDMI 1.4a

Nvidia and others are bringing it to the PC

Direct broadcast 3D service to the home has begun Blu-ray uses the MVC standard approach

Frame-compatible encoding with ordinary AVC decoders isanother approach


14/40

MVC Extension of AVC

Inter-view prediction

Enabled through flexible reference picture management

Allow decoded pictures from other views to be inserted and removed fromreference picture buffer

Core decoding modules do not need to be aware of whether referencepicture is a time reference or multiview reference

Syntax

Does not require anychanges to lower-level syntax (slice and lower), sovery compatible with single-layer AVC hardware

Base layer required and easily extracted from video bitstream (identified byNAL unit type)

Small changes to high-level syntax

E.g., specify view dependency, random access points

Used to provide full-HD stereo in 3D Blu-ray players


15/40

Frame Compatible Stereo 3D

Squeeze two views (e.g., side-by-side) into the coded frames

Half resolution each (notfull resolution) Or half frame rate

Decoder needs no customization for 3D (just detect and display)

AVC Frame Packing Arrangement SEI message Indicates type of packing of two views in the frame Side-by-side, top-bottom, checkerboard / quincunx, column-interleaved,

row-interleaved, and frame sequential interleaving Most adoption momentum for Side-by-Side and Top-Bottom approaches

Used in near-term broadcast Included in CableLabs and Dolby specs Recent interest in scalable enhancement to full resolution


16/40

Dynamic Adaptive Streaming on HHTP (DASH)

Modern Media Transport (MMT): MPEGs exploration for a new standard

for delivery of multimedia over IP networks.

Two workshops in July 2009 and January 2010.

Industry and academia participation.

Identified two main threads: streaming over HTTP (short term) vs.

long-cross layer optimization (long term).

Industrys urgency on HTTP streaming standard

Dynamic Adaptive Streaming over HTTP (DASH): CFP issued April 2010,

responses due July 2010.

MMT: CFP issued in July 2010, responses due Jan 2011.

15 Responses to DASHs CFP (Over 200 line items in various areas)

Issued Working Draft (WD) and established 9 evaluation experiments (EE)

in July.

50+ contributions including in September and October meetings.

DASH Committee Draft (CD), File format 3rd amendment (PDAM) and

established 7 EEs in October.16


17/40

DASH Scope and Main Features

17

Legend:

MF Manifest

DF Delivery Format

FF File Format (extensions)

TS Transport Stream

Server

MF

DF

FF

TS

MF

DF

FF

TS

Client Closely related to 3GPP 26.234.

Supports adaptive on demand

and live streaming of MPEG-4

file format and MPEG-2TS.

Efficient and ease of use of

existing CDNs, proxies, caches,

NATs and firewalls.

Signaling, delivery, utilization of

multiple DRM schemes.

Manifest fragmentation and

assembly for external inclusion

of periods, Representation

Groups and URL lists.


18/40

The new JCT-VC Partnership

Initial groundwork in VCEG and MPEG

Agreement in January 2010 to form new team VCEG-AM90 / N11112

Joint Call for Proposals on Video Coding Technology issued January 2010VCEG-AM91 / WG 11 N11113

Joint Collaborative Team (JCT) on Video Coding (JCT-VC)

Chairs: Gary Sullivan (Microsoft) & Jens-Rainer Ohm (RWTH Aachen Univ.)

Meetings so far First meeting: Dresden Germany 15-23 April 2010

Second meeting: Geneva, Switzerland 21-28 July 2010

Third meeting: Guangzhou, China 7-15 Octobter 2010

Project name High Efficiency Video Coding (HEVC) agreed April 2010

Document archives are publicly accessible http://phenix.it-sudparis.eu/jct

http://ftp3.itu.ch/av-arch/jctvc-site http://www.itu.int/ITU-T/studygroups/com16/jct-vc/index.html

Meeting reports JCTVC-A200 and JCTVC-B200

Email reflector http://mailman.rwth-aachen.de/mailman/listinfo/jct-vc
http://phenix.it-sudparis.eu/jcthttp://ftp3.itu.ch/av-arch/jctvc-sitehttp://www.itu.int/ITU-T/studygroups/com16/jct-vc/index.htmlhttp://mailman.rwth-aachen.de/mailman/listinfo/jct-vchttp://mailman.rwth-aachen.de/mailman/listinfo/jct-vchttp://mailman.rwth-aachen.de/mailman/listinfo/jct-vchttp://mailman.rwth-aachen.de/mailman/listinfo/jct-vchttp://mailman.rwth-aachen.de/mailman/listinfo/jct-vchttp://mailman.rwth-aachen.de/mailman/listinfo/jct-vchttp://www.itu.int/ITU-T/studygroups/com16/jct-vc/index.htmlhttp://www.itu.int/ITU-T/studygroups/com16/jct-vc/index.htmlhttp://www.itu.int/ITU-T/studygroups/com16/jct-vc/index.htmlhttp://www.itu.int/ITU-T/studygroups/com16/jct-vc/index.htmlhttp://www.itu.int/ITU-T/studygroups/com16/jct-vc/index.htmlhttp://ftp3.itu.ch/av-arch/jctvc-sitehttp://ftp3.itu.ch/av-arch/jctvc-sitehttp://ftp3.itu.ch/av-arch/jctvc-sitehttp://ftp3.itu.ch/av-arch/jctvc-sitehttp://ftp3.itu.ch/av-arch/jctvc-sitehttp://phenix.it-sudparis.eu/jcthttp://phenix.it-sudparis.eu/jcthttp://phenix.it-sudparis.eu/jct


19/40

Call for Proposals Testing

27 complete proposals submitted (some multi-organizational)

Each proposal was a major package lots of encoded video,

extensive documentation, extensive performance metric

submissions, sometimes software, etc.

Extensive subjective testing (3 test labs, 4 200 video clips

evaluated, 850 human subjects, 300 000 scores)

Quality of proposal video was compared to AVC (ITU-T Rec.

H.264 | ISO/IEC 14496-10) anchor encodings

Test report issued JCTVC-A204

In a number of cases, comparable quality at half bit rate


20/40

Test Classes and Bit Rates

3-5 video clips subjectively tested in Classes B-E

Testing for both random access (1 sec) andlow delay (no picture reordering) conditions

Complexity also considered in anchor encodings

Class Bit Rate 1 Bit Rate 2 Bit Rate 3 Bit Rate 4 Bit Rate 5A: 2560x1600p30 2.5 Mbit/s 3.5 Mbit/s 5 Mbit/s 8 Mbit/s 14 Mbit/s

B1: 1080p24 1 Mbit/s 1.6 Mbit/s 2.5 Mbit/s 4 Mbit/s 6 Mbit/s

B2: 1080p50-60 2 Mbit/s 3 Mbit/s 4.5 Mbit/s 7 Mbit/s 10 Mbit/s

C: WVGAp30-60 384 kbit/s 512 kbit/s 768 kbit/s 1.2 Mbit/s 2 Mbit/s

D: WQVGAp30-60 256 kbit/s 384 kbit/s 512 kbit/s 850 kbit/s 1.5 Mbit/s

E: 720p60 256 kbit/s 384 kbit/s 512 kbit/s 850 kbit/s 1.5 Mbit/s


21/40

Example Results Graph

Anchor at 2.5 Mbps

Anchor at 1.6 Mbps

Anchor at 1 Mbps

Best Performing

Proposal at 1 Mbps


22/40

Overall Average Mean Opinion Score


23/40

Basic Technology Architecture

All proposals basically conceptually similar to AVC (and priorstandards) Block-based Variable block sizes Block motion compensation

Fractional-pel motion vectors Spatial intra prediction Spatial transform of residual difference Integer-based transform designs Arithmetic or VLC-based entropy coding In-loop filtering to form final decoded picture

Lots of variations at the individual tool level Proposal survey output documents:

Decoder speed JCTVC-A201 Architectural outline JCTVC-A202 Table of design elements JCTVC-A203


24/40

Test Model under Consideration

(TMuC) Dresden output JCTVC-A205

Selected design elements from among best-performing proposals Should provide compression capability close to the best

Also a complexity point close to the lowest with substantial improvement of

coding efficiency

Some tools can be seen as placeholders at the respective position in thearchitecture

Further evaluated in experiments

Initiative to build a common software basis according to the test model Publicly-accessible CVS servers set up

Top-priority integrations to be completed by 9 Aug for next round experiments

Geneva refinements JCTVC-B205 High level syntax carry-over

Unified intra prediction

Rounding control and transform precision expansion

Reference configurations established for experiments


25/40

Contributions to TMuC Features

First TMuC April 2010, Dresden: JCTVC-A205 JCTVC-A114 (from France Telecom, NTT,

NTT DOCOMO, Panasonic and Technicolor)

JCTVC-A116 (from HHI)

JCTVC-A119 ("TENTM" = Tandberg, Ericsson, Nokia)

JCTVC-A120 (from RIM) JCTVC-A121 (from Qualcomm)

JCTVC-A124 (from Samsung, with BBC)

JCTVC-A125 (from BBC, with Samsung)

Second TMuC July 2010, Geneva: JCTVC-B205

JCTVC-B074 (from Qualcomm rounding control and transformprecision expansion)

JCTVC-B100 (from DoCoMo unified intra prediction)

JCTVC-B121 (meeting BoG on high-level syntax)


26/40

HEVC Test Model 1 (HM 1): Oct 2010

Follows the one tool one functionalityapproach

Planned to contain clusters of tools (could becomea seed for profiles), with as much commonality as

possible High efficiency (HE)

Low complexity (LC)

Software available

Closely resembles the performance of best-performing proposals

Basis to perform investigation on further tools thatare expected to further improve performance


27/40

First Working Draft and HEVC Model

(HM 1) Technical Overview

High Efficiency Low Complexity

Coding unit tree structure (8x8 up to 64x64 luma samples)

Prediction units

Transform unit tree structure (maximum of 3 levels) Transform unit tree structure (maximum of 2 levels)

Transform block size of 4x4 to 32x32 samples (always square)Angular intra prediction (maximum of 34 directions)

DCT-based interpolation filter for luma samples(1/4-sample, 12-tap)

Directional interpolation filter for luma samples(1/4-sample, 6-tap)

Bi-linear interpolation filter for chroma samples (1/8-sample)

Advanced motion vector prediction

Context adaptive binary arithmetic entropy coding Low complexity entropy coding phase 2

Internal bit-depth increase (4 bits) X

X Transform precision extension (4 bits)

Deblocking filter

Adaptive loop filter X


28/40

28

HEVC Test Model Overview High LevelSummary by Category

Coding Structure Coding unit tree structure (CU)

Prediction unit (PU)

Transform unit tree structure / Residual quadtree (RQT)

Intra Prediction Angular intra prediction

Inter Prediction Luma interpolation filters

1/4-sample, 12-tap DCT-based interpolation filter (DCT-IF) for high efficiency configuration (HE)

1/4-sample, 6-tap directional interpolation filter (DIF) for low complexity configuration (LC) Chroma interpolation filter

1/8-sample, bi-linear interpolation filter for both HE and LC


CU merging + CU skip / direct

Transforms Transform block size of 4x4 to 32x32 transforms (always square)

Entropy Coding Context adaptive binary arithmetic coding (CABAC) for high efficiency configuration Low complexity entropy coding (LCEC) phase 2 for low complexity configuration

Loop Filter Deblocking filter

Adaptive loop filter (ALF) for high efficiency configuration

Others Higher level syntax


29/40

29

HEVC Test Model Coding Structure

Tools adopted in HM Coding unit tree structure (CU)

CU splitting into PUs

Transform unit tree structure / RQT (JCTVC-C311, JCTVC-C319) 3-level quadtree for high efficiency configuration

LCEC phase 2 + 2-level quadtree is equivalent to LCEC phase 2 with RQT off for low

complexity configuration Includes Qualcomm coded block pattern (CBP) flag

Encoder setting for fast intra encode as per JCTVC-C311 HHI_RQT_INTRA_SPEEDUP = 1

HHI_RQT_INTRA_SPEEDUP_MOD=0

(Slower search for Intra modes to remain in software)

Same maximum quadtree depth for luma and chroma

TMuC features for further investigation (Not in the HM) Asymmetric motion partitions (AMP)

Geometric partitioning

Note: Items in red are features that are currently not in the TMuC 0.8 software but shallbe integrated in the TMuC 0.9 / HM1.0 software.


30/40

30

HEVC Test Model Intra Prediction

Tools adopted in HM Simplified unified intra prediction (JCTVC-C042)

Encoder modification for intra prediction search (JCTVC-C207)

TMuC features for further investigation (Not in the HM) Adaptive intra smoothing (AIS)

Note: For HM, AIS is turned off and DEFAULT_IS is set to 0

Combined intra prediction (CIP)

Planar prediction

Edge base prediction

Note: Items in red are features that were not in the TMuC 0.8 software but have beenintegrated in the TMuC 0.9 / HM1.0 software.


31/40

31

HEVC Test Model Inter Prediction Tools adopted in HM

Luma interpolation filters

1/4-sample, 12-tap DCT-based interpolation filter (DCT-IF) for high efficiency configuration (HE)

1/4-sample, 6-tap directional interpolation filter (DIF) for low complexity configuration (LC)

Chroma interpolation filter

1/8-sample, bi-linear interpolation filter for both HE and LC (based on follow-up reflector email

agreement 27 Oct.)

Bi-direction rounding control Rounding offset for bi-predictive rounding is signalled. (0 or 1)

Enable this when internal bit depth increase (IBDI) is off and disable when IBDI is on.

Encoder only modifications for software speedup. (JCTVC-C253)

CU merging + CU skip / direct


Bi-directional prediction for temporal level 0 (JCTVC-C278, JCTVC-C285)

TMuC features for further investigation (Not in the HM)

Interleaved motion vector prediction (IMVP)

Adaptive motion vector resolution (AMVRES)

Motion vector prediction scaling

PU merging + modified CU skip / direct

Partition based illumination compensation (PBIC).


32/40

32

HEVC Test Model Transforms

Tools adopted in HM

Transform block size of 4x4 to 32x32 samples (always square)


Mode dependent directional transform (MDDT)

Rotational transform (ROT)

Transform block size of 64x64 samples


33/40

33

HEVC Test Model Entropy Coding

Tools adopted in HM Context adaptive binary arithmetic coding (CABAC) for high efficiency

configuration

Low complexity entropy coding (LCEC) phase 2 for low complexityconfiguration

Coefficient sign PCP (JCTVC-B088 Section 3.2) Coefficeint level BinIdx 0 PCP (JCTVC-B088 Section 3.3)

Coded block flag signaling in VLC (JCTVC-C262)

Coded block flag redundancy removal (JCTVC-C277)

HHI transform coefficient coding

TMuC features for further investigation (Not in the HM) Probability interval partitioning entropy (PIPE) coding

Variable length to variable length (V2V) codes


34/40

34

HEVC Test Model Loop Filter

Tools adopted in HM

Deblocking filter

Adaptive loop filter (ALF) for high efficiency configuration

Signaling ALF flag in slice header


3-input ALF


35/40

35

HEVC Test Model Others

Tools adopted in HM

Higher level syntax (as decided in Geneva, July 2010)

Internal bit depth increase (IBDI) with 4 bits added

precision for 8-bit per sample decoding Transform precision extension (TPE) with 4 bits added

precision for 8-bit per sample decoding

Rate distortion optimized quantization (RDOQ) (encoder

only)


36/40

Current Core Experiments (CEs) CE1 Decoder-side Motion Vector Derivation, Coordinator:

Yi-Jen Chiu (Intel)

CE2 Flexible Motion Partitioning, Coordinator: EdouardFrancois (Technicolor)

CE3 Interpolation for MC (Luma), Coordinator: Takeshi

Chujoh (Toshiba) CE4 Interpolation for MC (Chroma), Coordinator: Elena

Alshina (Samsung)

CE5 LCEC improvements, Coordinator: X. Wang(Qualcomm)

CE6 Intra prediction improvements, Coordinator: AliTabatabai (Sony)

CE7 Alternative transforms, Coordinator: Robert Cohen(Mitsubishi)


37/40

Current Core Experiments (CEs) CE8 In-Loop filtering, Coordinator: T. Yamakage (Toshiba)

CE9 MV coding, Coordinator: Joel Jung (Orange)

CE10 Number of Intra Prediction Directions, Coordinator:

Kazuo Sugimoto (Mitsubishi)

CE11 Coefficient Scanning and Coding, Coordinator:

Vivienne Sze (TI)

CE12 Adaptive Motion Vector Resolution, Coordinator:

Wei-Jung Chien (Qualcomm)

CE13 Intra Smoothing, Coordinator: Muhammed Coban

(Qualcomm)


38/40

Timeline Plan Current plan:

2011 Meetings January 20-28, 2011 DaeguMarch 15-23, 2011 GenevaJuly 14-22, 2011 TorinoNov 23-30, 2011 Geneva

CD February 2012

Meeting April 2012

DIS July 2012 Meeting October 2012

FDIS & Consent January 2013 [Final spec]


39/40

JCT-VC Ad Hoc Groups Established JCT-VC project management

HEVC Draft and Test Model editing

Software development and HM software technical evaluation

Slice support development and characterization

Spatial transforms

In-loop and post-processing filtering Coding block structures

Reference pictures memory compression

Entropy coding

Entropy slices Video test material selection

Complexity assessment

Motion compensation interpolation


40/40

HEVC Expectations & Final Words Very active project (200+ documents & people per meeting)

Very diverse company & university participation

Significant technical advance over prior standards

Computational/implementation complexity is a big challenge

Parallelism is an increased theme

Deliverables Video coding specification

Reference software

Conformance testing specification

Profiles for various applications (mobile, high-qualityentertainment, etc.)

Likely multiple versions and extensions (SVC, MVC, etc)

Contact: JVT, JCT-VC, VCEG, MPEG video chairs: Gary J. Sullivan ([email protected])

i h ( h @i h h d )

Documents

2011 01 ICCE Video Developments Garysull d2