Communication & Multimedia C. -Y. Tsai 2005/12/15 1 Vidwav Wavelet Video Coding Specifications Student: Chia-Yang Tsai Advisor: Prof. Hsueh-Ming Hang Institute

C. -Y. Tsai 2005/12/151Communication & Multimedia

Vidwav Wavelet Video Coding Vidwav Wavelet Video Coding SpecificationsSpecifications

Student: Chia-Yang TsaiStudent: Chia-Yang TsaiAdvisor: Prof. Hsueh-Ming HangAdvisor: Prof. Hsueh-Ming Hang

Institute of Electronics, NCTUInstitute of Electronics, NCTU


OutlineOutline FrameworkFramework Main modulesMain modules

MotionMotion Temporal transformTemporal transform Spatial transformSpatial transform Entropy codingEntropy coding Bitstream formationBitstream formation

Additional modulesAdditional modules Base layerBase layer In-band codingIn-band coding Wavelet ringing reductionWavelet ringing reduction

ConclusionsConclusions


ReferencesReferences

Microsoft Research Asia + ENST + INRIA , “Vidwav Wavelet Video Coding Specifications ”, ISO/IEC JTC1/SC29/WG11, M12339, Poznan, July. 2005.


FrameworkFramework


T+2DT+2D

VideoFrames

TemporalWavelet

Decomposition

MotionEstimation

Post- 2D SpatialWavelet

Decomposition

MV & ModeCoding

EntropyCoding

...

...

...

Pre- 2D SpatialWavelet

Decomposition

(Pre-spatial decomposition is void)


2D+t2D+t

VideoFrames

TemporalWavelet

Decomposition

MotionEstimation

Post- 2D SpatialWavelet

Decomposition

MV & ModeCoding

EntropyCoding

...

...

...

Pre- 2D SpatialWavelet

Decomposition


MotionMotion

ME/MCME/MCModes/MV codingModes/MV coding


ME/MCME/MC

Adopted from H.264/AVC.Adopted from H.264/AVC. Performed by minimizing the Lagrangian cost functionPerformed by minimizing the Lagrangian cost function

J= D + λRJ= D + λR

16x16 (a) 16x8 (b) 8x16 (c) 8x8 (d)

8x8 (d0) 8x4 (d1) 4x8 (d2) 4x4 (d3)


MV CodingMV Coding

Median predictionMedian prediction For squre MC block sizeFor squre MC block size

16x16, 8x8, 4x416x16, 8x8, 4x4 (ABC) or (ABD)(ABC) or (ABD)

E

A

B CD

Directional segmentation predictionDirectional segmentation prediction If predictor is not available, use median predictionIf predictor is not available, use median prediction

A

B

A

c

C

A

BDC

A

BD

16x8 8x16 8x4 4x8


Temporal TransformTemporal Transform

Motion aligned temporal filteringMotion aligned temporal filteringPrediction stepPrediction step

Update stepUpdate step


Motion Aligned Temporal FilteringMotion Aligned Temporal Filtering

Lifting structureLifting structure MAP (motion aligned prediction)MAP (motion aligned prediction) MAU (motion aligned update)MAU (motion aligned update)

Split M AP MAUiF

i 2i+1B F

i 2iA F

++

++ +

-

iL

iH iH

iL

iU

iPodd

even

MergeM APMAUiF

i 2i+1B F

i 2iA FiL

iHiH

iL

iU

iP odd

even

Split MAP MAU MAPMAU Merge


Motion Aligned Temporal FilteringMotion Aligned Temporal Filtering(Cont.)(Cont.)

Temporal lifting steps of 5/3 biorthogonal Temporal lifting steps of 5/3 biorthogonal waveletwavelet


PredictionPrediction

Bidirectional connectionBidirectional connection

[ , ] { [ , ] [ , ]}/ 2Left RightP n m P n m P n m

Uni-directional connectionUni-directional connection

[ , ] [ , ]LeftP n m P n m[ , ] [ , ]RightP n m P n m

When left-unidirectional connected

When right-unidirectional connected


Prediction (Cont.)Prediction (Cont.)

OBMCOBMC

, ,[ , ] [ , ][ , ] [ [ , ]] [ , ]Left pm Left Left Leftn m n mP n m F n m R n n m m

, ,[ , ] [ , ][ , ] [ [ , ]] [ , ]Right pm Right Right Rightn m n mP n m F n m R n n m m


UpdateUpdate

Generated from H-frames through MCGenerated from H-frames through MC

],[

],[

],[

],[

],[],[

4

3

2

1

mmnnH

mmnnH

mmnnH

mmnnH

mnLmnU

ClippingClipping

TmnHC

TmnHCT

TmnHC

T

mnHC

T

mnU

],[

],[

],[

],[],[

)()(

)(1)(

)()(1

)(1)(1

4

3

2

1

mmmmnnnn

mmmmnnnn

mmmmnnnn

mmmmnnnn


Spatial TransformSpatial Transform


StructureStructure

Decomposition structureDecomposition structurePre Spati al

Decomposi t i onPost Spati alDecomposi t i on

TemporalDecomposi t i on

DescriptionDescription

S<>S<>E<>E<>E<>E<>E<>E<>E<>

S<> S<> S<> E<> E<> E<> E<>E<> E<> E<> S<> E<> E<> E<> E<> S<> E<> E<> E<> E<> S<> E<> E<> E<> E<>


Entropy CodingEntropy Coding

3D-EBCOT3D-EBCOTBitstreamBitstream


EBCOTEBCOT

Embedded Block CodingEmbedded Block Coding

Code block

Fractional bitplanes

Bitplanes

coding pass


Coding Pass OperationCoding Pass Operation

Previous bit-planePrevious bit-plane

Current bit-planeCurrent bit-plane

Insignificant sampleInsignificant sample

Significant sampleSignificant sampleIn previous bit-planeIn previous bit-plane

Significant sampleSignificant sampleIn current bit-planeIn current bit-plane

Significant Propagation PassSignificant Propagation Pass

Magnitude Refinement PassMagnitude Refinement Pass

Normalization PassNormalization Pass

ZCZC

ZCZC

ZCZCSCSCMRMR

ZCZC

SCSC


BitstreamBitstream

NN of total bitplanes in the block of total bitplanes in the block The coded bitstream of the block, The coded bitstream of the block,

which consists of which consists of 3N-2 3N-2 segments, segments, each segment corresponding to the each segment corresponding to the output bitstream of one coding pass.output bitstream of one coding pass.

The length of each bitstream segment The length of each bitstream segment and/or the position of end-point of and/or the position of end-point of each coding pass.each coding pass.

The R-D slope information at the end The R-D slope information at the end of each coding pass. of each coding pass.


Bitstream FormationBitstream Formation

SyntaxSyntax Bitstream selection Bitstream selection


SyntaxSyntax

GlobalGlobal GOPGOP

Group of picture for MCTFGroup of picture for MCTF LayerLayer

For quick bitstream truncationFor quick bitstream truncation PacketPacket

One component (Y, U, or V) of one temporal sunbbadOne component (Y, U, or V) of one temporal sunbbad SubbandSubband

3D subband3D subband BlockBlock

Block of 3D EBCOTBlock of 3D EBCOT PassPass

Coding passCoding pass


Bitstream SelectionBitstream Selection

Optimized TruncationOptimized TruncationRD slope

Block n Block n+1

iteration m

iteration m-1

Rn Rn+1

ΣRi <= bitrate constrain

Coding pass index


Base LayerBase Layer


Embedded Base Layer CodecEmbedded Base Layer Codec


Motion Information PredictionMotion Information Prediction

BASESKIP modeBASESKIP mode Use base-layer motion vectors as motion predictorsUse base-layer motion vectors as motion predictors

8x16 16x16

8x8 16x8

8x8

8x4

4x8

4x4

Macroblock partitions mode map rule to generate the predictorsfor higher spatial resolution


Hierarchical B-PicturesHierarchical B-Pictures

Provide extra- temporal resolutionsProvide extra- temporal resolutions


In-Band CodingIn-Band Coding

Leaky motion compensationLeaky motion compensationMode-based temporal filteringMode-based temporal filtering


In-Band MCTFIn-Band MCTF

LL

Sinc4x4interpolation ¼-pixel

interpolationreference based

on LL

LH

HL

HH

IDWT+ODWT ODWT

LL

Sinc2x2interpolation

¼-pixelinterpolation

reference basedon ODWT LL

LL

The forming of different quality reference The forming of different quality reference of LL of LL Low quality reference as IP_DIRLow quality reference as IP_DIR

High quality reference as IP_LBSHigh quality reference as IP_LBS



Leaky motion compensationLeaky motion compensation leaky factor leaky factor

Attenuate the prediction based on the Attenuate the prediction based on the unknown information at the decoder unknown information at the decoder

make a good trade-off between drifting make a good trade-off between drifting errors and coding efficiency errors and coding efficiency

1,...,0

)),(_)(_)1((2

1

)),(_)(_)1((2

1),(

)(

22122222

21222222

222121

Ni

MVLLBSIPLDIRIPMC

MVLLBSIPLDIRIPMCLLP

LLPLH

nni

ni

n

nni

ni

ni

ni

n

in

in

in

in



Mode-based Mode-based temporal filteringtemporal filtering Mode I: Low quality Mode I: Low quality

referencereference Mode 2: High Mode 2: High

quality referencequality reference Mode is selected by Mode is selected by

RD costRD cost

LL

LL HL

LH HH

LL

LL

LL HL

LH HH


Wavelet Ringing ReductionWavelet Ringing Reduction


DescriptionDescription

PurposePurpose Reduce artifact after EDWTReduce artifact after EDWT

Similar like the de-blocking filter in DCT based Similar like the de-blocking filter in DCT based codingcoding

2 2

1 2,

2 2

1 2,

, , ,

ˆ ,, ,

k l

k l

W n k m l W x n m x k l x k l

x n mW n k m l W x n m x k l

2

1 2exp

S

dW d

2

2 2exp

V

dW d

After experiments,After experiments,2

1S c 2 222

1

11.5133 2 n

Ns b

Vn

cN


ConclusionsConclusions


Wavelet Based SVCWavelet Based SVC

AdvantagesAdvantages Nature for multi-resolution scalability Nature for multi-resolution scalability Open-loop prediction structureOpen-loop prediction structure

Provides elegant SNR scalability without Provides elegant SNR scalability without impairing full exploitation of spatial-impairing full exploitation of spatial-temporal correlation temporal correlation

Simplifies the R-D model of the bitstreams. Simplifies the R-D model of the bitstreams. Facilitates the bitstream truncation Facilitates the bitstream truncation

each subband is independent with other each subband is independent with other subbands subbands


Wavelet Based SVCWavelet Based SVC

DisadvantagesDisadvantages Decomposition modes (coding modes) Decomposition modes (coding modes)

selection selection Texture & side information trade offTexture & side information trade off Intra-prediction Intra-prediction Badly-matched blocks Badly-matched blocks Downsampling filter problemsDownsampling filter problems


Thanks for your attention!Thanks for your attention!Any questions?Any questions?

Documents

Communication & Multimedia C. -Y. Tsai 2005/12/15 1 Vidwav Wavelet Video Coding Specifications Student: Chia-Yang Tsai Advisor: Prof. Hsueh-Ming Hang Institute