Advanced Intra Coding · Intra-PicturePrediction Intra-PicturePrediction Transformcoding: Typicaldesign 2DDiscreteCosineTransformoftypeII(orintegerapproximation) Scalarquantization

Advanced Intra Coding

Intra-Picture Prediction

Intra-Picture Prediction

Transform coding: Typical design2D Discrete Cosine Transform of type II (or integer approximation)Scalar quantization: URQs with same quantization step size ∆

Entropy coding (employing remaining statistical dependencies)Can only utilize dependencies within transform blocks

Intra-picture predictionCan additionally utilize dependencies between transform blocksVery simple variant (JPEG and H.262 | MPEG-2 Video):Predict DC coefficient using DC coefficient of previous blockMore advanced approaches can significantly increase coding efficiency

Two approaches of intra-picture predictionPrediction in transform domainPrediction in spatial domain (before transform coding)

T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 2 / 22

Intra-Picture Prediction / Prediction of Transform Coefficients

Prediction of DC Coefficient

Simplest Variant of Intra-Picture Prediction (JPEG & MPEG-2 Video)Predict DC level with DC level of previous block (left to current block)Transmit only difference to predictor

∆DCk = DCk −DCk−1

Reduces variance of pmf of transmitted DC levelImproves coding efficiency

Example: Histogram with and without DC prediction (8×8 blocks)

0 10 20 30 40 50 60

without DC prediction

-30 -20 -10 0 10 20 30

with DC prediction



Example: Effect of DC Prediction (8×8 Blocks)

22

24

26

28

30

32

34

36

38

40

42

44

46

48

50

52

0 0.5 1 1.5 2 2.5 3 3.5 4

Lum

a P

SN

R [d

B]

bits per luma sample

testImage02

no dcPredmid range dcPred

adaptive dcPred



Advanced Intra Prediction in Transform Domain

Advanced Intra Coding mode of H.263: Three coding modesDC prediction and zig-zag scanHorizontal prediction and alternate-vertical scan

Suitable for blocks with mainly horizontal structuresVertical prediction and alternate-horizontal scan

Suitable for blocks with mainly vertical structures

Similar concept is also used in MPEG-4 VisualT. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 5 / 22


How to Select Suitable Prediction Mode?

Mode DecisionSyntax supports multiple intra prediction modes (mode is transmitted)Encoder has to select one of the supported modesGoal: Maximize coding efficiency

Lagrangian Mode Decision

Evaluate all supported coding modesPerform prediction, quantization, entropy coding, reconstructionCalculate distortion D =

∑k(s ′k − sk)2

Determine number of bits R for transmitting mode and transformcoefficient levels

Select mode that minimizes Lagrangian cost function

J = D + λ · R


Intra-Picture Prediction / Spatial Intra Prediction

Intra Prediction: Transform Domain — Spatial Domain

verticalprediction

intransformdomain

equivalentvertical

predictionin spatialdomain

simplifiedand

improvedvertical

predictionin spatialdomain

Example: Vertical predictionTransform domain: Predict first row of transform coefficientsEquivalent prediction in spatial domain

sver[x , y ] =1N

N−1∑k=0

s ′[x ,−1− k]

Simplified prediction in spatial domain

sver[x , y ] = s ′[−1, y ]



Spatial Intra Prediction

Spatial intra predictionSimilar complexity than similar operation in transform domainUsage of directly adjacent samples =⇒ Improved coding efficencyMain advantages:

Can also be applied if neighboring blocks are coded in an inter modeStraightforward extension to multiple prediction directions(can include interpolation of border samples)

Intra prediction in video coding standardsH.262 | MPEG-2: Predict DC coefficient from previous blockH.263 & MPEG-4: DC, horizontal, vertical (in transform domain)H.264 | AVC: 9 spatial intra prediction modes (for 4×4/8×8 blocks)H.265 | HEVC: 35 spatial intra prediction modes (for all block sizes)

Number of supported intra prediction modes is increased from one generationof video coding standards to the next



Spatial Intra Prediction in H.264 | MPEG-4 AVC



Spatial Intra Prediction in H.265 | MPEG-H HEVC

33 directional prediction modes (linear interpolation / ref. sample smoothing)

DC prediction mode (similar to H.264 | MPEG-4 AVC)

Planar prediction mode

In total: 35 spatial intra prediction modes



HEVC Planar Mode: General Idea

Not all image blocks fit an edge model (well representable by directionalimage model)DC prediction only very coarse approximation (order-0 model)Alternative: Planar prediction mode

s[x , y ] =N − 1− x

2N· s ′[−1, y ] +

1 + x

2N· s[N − 1, y ]+

N − 1− y

2N· s ′[x ,−1] +

1 + y

2N· s[x ,N − 1]

s[N − 1, y ] =N − 1− y

N· s ′[N − 1,−1] +

1 + y

2N· s[N − 1,N − 1]

s[x ,N − 1] =N − 1− x

N· s ′[−1,N − 1] +

1 + x

2N· s[N − 1,N − 1]

s[N − 1,N − 1] =12· s ′[N − 1,−1] +

12· s ′[−1,N − 1]



HEVC Planar Mode: Actual Implementation

sH[x , y ] = (N − 1− x) · s ′[−1, y ] + (1 + x) · s ′[N − 1,−1]

sV[x , y ] = (N − 1− y) · s ′[x ,−1] + (1 + y) · s ′[−1,N − 1]

s[x , y ] =12N

(sH[x , y ] + sV[x , y ]

)T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 12 / 22


Spatial Intra Prediction — Coding Efficiency

0

10

20

30

40

50

30 32 34 36 38 40

bit-

rate

sav

ing

vs D

C p

red.

[%]

PSNR (Y) [dB]

Cactus (1920×1080, 50 Hz), 8×8 blocks

DC, horizontal, and vertical prediction

9 prediction modes

35 prediction modes

0

10

20

30

40

50

34 36 38 40 42 44

bit-

rate

sav

ing

vs D

C p

red.

[%]

PSNR (Y) [dB]

Kimono (1920×1080, 24 Hz), 8×8 blocks

DC, horizontal, and vertical prediction

9 prediction modes

35 prediction modes

Experimental investigation with H.265 | MPEG-H HEVCRestricted to 8×8 blocks (effect of block size is discussed later)Limited number of used prediction modes (reference: DC prediction only)

Coding efficiency increases with number of supported intra prediction modes


Variable Block Sizes for Prediction and Transform Coding

Block Sizes for Prediction and Transform Coding

Impact of block size selection for transform codingCoding efficiency of transform coding typically increases with block sizeCoding efficiency improvement becomes small beyond a certain block sizeComplexity increases with block size

Impact of block size selection for spatial predictionCorrelation decreases with increasing sample distancesIntra prediction is more effective for smaller block sizesSide information rate (for intra modes) increases with decreasing block size

Combination of intra prediction and transform codingOptimal block size depends on actual signal propertiesNatural images: Highly non-stationary statistical propertiesNo single optimal block sizeAdaptive block size selection can improve coding efficiency



Block Sizes in Video Coding Standards

H.262 | MPEG-2 Video, H.263, MPEG-4 VisualFixed block sizes for prediction and transform coding16×16 macroblocks (for signaling intra prediction mode)8×8 transform blocks

H.264 | MPEG-4 AVC (High profile)16×16 macroblocks3 intra coding modes: Intra4x4, Intra8x8, Intra16x16Block sizes for prediction and transform coding: 4×4, 8×8, 16×16Intra prediction mode selected on basis of transform blocksIntra16x16: Only 4 prediction modes & low-complexity 16×16 transform

Intra4x4 Intra8x8 Intra16x16



Picture Partitioning in H.265 | MPEG-H HEVC

Picture partitioning into coding tree blocksCoding tree blocks (CTBs): Fixed size of16×16, 32×32 or 64×64 luma samplesSize of CTBs chosen by encoderLuma and chroma CTBs together withsyntax are called coding tree unit (CTU)

Partitioning of coding tree blocksQuad-tree partitioning into coding blocks (CBs)Luma and chroma CBs together with syntax arecalled coding unit (CU)Minimum CU size: Selected by encoder, butequal to or larger than 8×8 luma samplesCoding mode (intra or inter) is chosen for CUCoding order: Z-scan



Example: Picture Partitioning into Coding Units

Example for picture partitioning into coding unitsPicture with 2560×1600 luma samples of HEVC test sequence “Traffic”Quadtree-based partitioning into coding unit represents a simple scheme forlocally adapting the block sizes to the image structure



Intra Coding of Coding Units in H.265 | MPEG-H HEVC

Partitioning of a CB into transform blocks (TBs)Nested quad-tree partitioningTB corresponds to a single block transformSupported sizes: 4×4, 8×8, 16×16, 32×32Luma and chroma TBs together with syntaxform a transform unit (TU)

Intra prediction and mode signalingOne or four luma intra predictionmodes per coding unitOne chroma prediction mode per CUActual intra prediction is performedtransform block by transform blockImproved prediction accuracy



Block Sizes for Intra-Picture Coding — Coding Efficiency

30

32

34

36

38

40

0 20 40 60 80 100

PS

NR

(Y

) [d

B]

bit rate [Mbit/s]

Cactus (1920×1080, 50 Hz), DC prediction

4×4 blocks

8×8 blocks

16×16 blocks

32×32blocks

all block sizes

34

36

38

40

42

44

0 10 20 30 40 50

PS

NR

(Y

) [d

B]

bit rate [Mbit/s]

Kimono (1920×1080, 24 Hz), DC prediction

4×4 blocks

8×8 blocks

16×16 blocks

32×32 blocks

all block sizes

First coding experiment with H.265 | MPEG-H HEVCReduce impact of intra prediction: Only DC prediction is enabledCheck different fixed block sizes & variable block sizesFixed block sizes: Coding efficiency increases with block sizeVariable block sizes provide coding gains




30

32

34

36

38

40

0 20 40 60 80 100

PS

NR

(Y

) [d

B]

bit rate [Mbit/s]

Cactus (1920×1080, 50 Hz), all pred. modes

4×4 blocks

8×8 blocks

16×16 blocks

32×32 blocks

all block sizes

34

36

38

40

42

44

0 10 20 30 40 50

PS

NR

(Y

) [d

B]

bit rate [Mbit/s]

Kimono (1920×1080, 24 Hz), all pred. modes

4×4 blocks

8×8 blocks

16×16 blocks

32×32 blocks

all block sizes

Second coding experiment with H.265 | MPEG-H HEVCAll intra prediction modes are enabledPrediction increases effectiveness of smaller block sizesFixed block sizes: Medium block sizes provide best coding efficiencyVariable block sizes provide coding gains




0

10

20

30

40

50

30 32 34 36 38 40

bit-

rate

sav

ing

vs 8×

8 bl

ocks

[%]

PSNR (Y) [dB]

Cactus (1920×1080, 50 Hz), all pred. modes

4×4 and 8×8 blocks

4×4, 8×8 blocks,and 16×16 blocks

all block sizes (4×4 to 32×32)

0

10

20

30

40

50

34 36 38 40 42 44

bit-

rate

sav

ing

vs 8×

8 bl

ocks

[%]

PSNR (Y) [dB]

Kimono (1920×1080, 24 Hz), all pred. modes

4×4 and 8×8 blocks

4×4, 8×8 blocks,and 16×16 blocks

all block sizes (4×4 to 32×32)

Third coding experiment with H.265 | MPEG-H HEVCAll intra prediction modes are enabledStart with 8×8 blocks and successively enable additional block sizesAdditional block sizes provide coding efficiency improvementsBeside intra-picture prediction, the support of additional block sizes is a mainfactor for the improvement in intra-picture coding


Summary

Part Summary

Transform coding of sample blocksSeparable orthogonal transform: DCT or integer approximationScalar quantization: URQs with same quantization step size ∆

Entropy coding: Utilize remaining dependencies between quantization indexes

Intra-picture predictionUtilize dependencies between transform blocksTwo methods: Prediction in transform domain or spatial domainSpatial prediction: Straightforward realization of multiple prediction modesCoding efficiency typically increases with number of supported intra modes

Block sizes for intra prediction and transform codingDetermine efficiency of prediction and transform codingNon-stationary character of natural images =⇒ Variable block sizesSimple and flexible partitioning: Quadtree-based approachesVariable block sizes significantly increase coding efficiency


Documents

Advanced Intra Coding · Intra-PicturePrediction Intra-PicturePrediction Transformcoding: Typicaldesign 2DDiscreteCosineTransformoftypeII(orintegerapproximation) Scalarquantization