Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain

Distributed Video Coding

João Ascensojoao.ascenso@lx.it.pt

Instituto Superior de Engenharia de Lisboa Instituto de Telecomunicações

Portugal

EC–SIP–M 200501/07/05

Summary

1. Context and Applications

2. Objectives

3. IST-PDWZ Video Codec

4. Frame Interpolation

5. Experimental Results

6. Conclusions and Future Work

EC–SIP–M 200501/07/05

Traditional Video Coding

– Predictive coding framework:– Encoder is 5 to 10 times more complex than decoder mainly due to

the motion estimation/compensation tools.– Well-suited for “one-to-many” topologies:

– Broadcasting or video-on-demand applications.

EC–SIP–M 200501/07/05

Emerging Applications, New Requirements

– Applications:– Mobile video.– Multimedia sensor networks.– Wireless video surveillance.– Multi-view acquisition.– …

– Encoding requirements:– Low complexity.– Low power-consumption.– Low cost.– ...

EC–SIP–M 200501/07/05

Need for a New Coding Paradigm

– Coding configuration with low-complexity and low-power encoder at the expense of a high-complexity decoder– Without compromising coding efficiency.

New video coding paradigm:

Distributed Video Coding (DVC) Distributed Video Coding (DVC) Exploits video statistics, partially or totally, at the decoder.

EC–SIP–M 200501/07/05

Summary

1. Context and Applications

2. Objectives

3. IST-PDWZ Video Codec

4. Frame Interpolation

5. Experimental Results

6. Conclusions and Future Work

EC–SIP–M 200501/07/05

Main Goal

The challenge is:

How to generate the best side information (a frame) as close as How to generate the best side information (a frame) as close as

possible to the current frame to be decoded ? possible to the current frame to be decoded ?

Distributed videodecoder

Lightweight encoder

Side Information

The rate-distortion (RD) performance of a distributed video coding scheme is highly dependent on the quality of the side information.

EC–SIP–M 200501/07/05

Summary

1. Context and Applications

2. Objectives

3. IST-PDWZ Video Codec

4. Frame Interpolation

5. Experimental Results

6. Conclusions and Future Work

EC–SIP–M 200501/07/05

IST Pixel-Domain Wyner-Ziv Video Codec

– Based on the Wyner-Ziv coding scenario.– Intra-frame encoder and inter-frame decoder.– X source: even frames of the video sequence.– Y source corresponds to the Side Information:

– Generated using the odd frames at the decoder – Y is an estimate of the current frame X and is available at the decoder.

– Channel coding of X allows to improve the quality of Y.

Inter-framedecoder

Intra-frame encoder

Virtual channeldependence

Side Information

EC–SIP–M 200501/07/05

IST Pixel-Domain Wyner-Ziv Video Codec

– Turbo code based Slepian-Wolf codec.– Each bitplane is independently turbo coded.– Two frame types:

– Key-frames (odd frames) and Wyner-Ziv frames (even frames).– Side information (Y2i) is obtained by frame interpolation.

Wyner-Ziv Encoder Wyner-Ziv Decoder

Slepian-Wolf Encoder Slepian-Wolf Decoder

2N- Level Uniform

Quantizer

X2i ..

..

Bitplane 1X'2i

Y2i

X2i - 1

X2i + 1

Wyner-Ziv Bitstream

Turbo Encoder

Turbo Decoder Reconstruction

FrameInterpolation

Buffer

Feedback Channel

Bitplane N

EC–SIP–M 200501/07/05

Slepian-Wolf Codec

– Turbo encoder:– Two identical recursive systematic

convolutional (RSC) encoders.– Pseudo-random interleaver.

– Each RSC encoder outputs the systematic and the parity streams:– Systematic stream is discarded.– Parity stream is stored in the buffer

and punctured.– Decoder request parity bits until

successful decoding.

– Iterative turbo decoder:– Two SISO decoders. – Maximum A Posteriori (MAP) algorithm.– Laplacian distribution to model the correlation between X and Y.

EC–SIP–M 200501/07/05

Summary

1. Context and Applications

2. Objectives

3. IST-PDWZ Video Codec

4. Frame Interpolation

5. Experimental Results

6. Conclusions and Future Work

EC–SIP–M 200501/07/05

Frame Interpolation: Architecture

– Block-based motion compensated interpolation (5 steps):– 1) Low pass filter to improve the motion vectors reliability.– 2) Forward motion estimation.– 3) Bi-directional motion refines initial motion vector estimate.– 4) Spatial smoothing of motion vectors.– 5) Bi-directional MC to fill the interpolated frame.

EC–SIP–M 200501/07/05

Initial MV & Bi-directional ME

– Motion vectors obtained by FME are candidates for each non-overlapped block in the interpolation frame:– Selected the MV that intercepts the interpolated frame closest to the

center of block.– Find linear trajectory (symmetric MVs) between key frames passing at the

center of the block in the interpolated frame:– Small displacement around the initial block position.

EC–SIP–M 200501/07/05

Spatial Motion Smoothing

– Spatial smoothing algorithms target: – Reduction on the number of false motion vectors when compared to

the true motion field.– Better spatial homogeneity of the resulting motion field.

– Weighted vector median filters are proposed:– Filter smoothing strength depends on the prediction MSE.– Low value weights when MSE for the candidate vector is high.– High value weights when MSE for the candidate vector is low.

1 1

N N

j wvmf j j i jL Lj j

w x x w x x= =

− ≤ −∑ ∑ ( )( )

,,

cj

j

MSE x Bw

MSE x B=

EC–SIP–M 200501/07/05

Spatial Motion Smoothing

– Without motion vector smoothing filter

EC–SIP–M 200501/07/05

Spatial Motion Smoothing

– With motion vector smoothing filter

EC–SIP–M 200501/07/05

Summary

1. Context and Applications

2. Objectives

3. IST-PDWZ Video Codec

4. Frame Interpolation

5. Experimental Results

6. Conclusions and Future Work

EC–SIP–M 200501/07/05

Coastguard Sequence

Coastguard sequence

26.5

28.5

30.5

32.5

34.5

36.5

38.5

40.5

42.5

0 50 100 150 200 250 300 350 400 450 500 550 600Rate of even frames (kbps)

PSN

R o

f eve

n fr

ames

(dB

)

H.263+ Intra

H.263+ I-B-I-B

PDWZ Average

PDWZ FME

PDWZ BiME

PDWZ SS

11.5 dB

EC–SIP–M 200501/07/05

Foreman Sequence

Foreman sequence

27.0

29.0

31.0

33.0

35.0

37.0

39.0

41.0

43.0

0 50 100 150 200 250 300 350 400 450 500 550 600 650Rate of even frames (kbps)

PSN

R o

f eve

n fr

ames

(dB

)

H.263+ IntraH.263+ I-B-I-BPDWZ AveragePDWZ FMEPDWZ BiMEPDWZ SS

10 dB

EC–SIP–M 200501/07/05

Rate-Distortion Comparison

Foreman QCIF Sequence @ 30fps

35.0

36.0

37.0

38.0

39.0

40.0

41.0

42.0

43.0

0 50 100 150 200 250 300 350 400

Rate of even frames [kbps]

PS

NR

of e

ven

fram

es [d

B]

H.263+ I-B-I-BSpatial Motion SmoothingStanford

2.6 dB

2 dB

EC–SIP–M 200501/07/05

Summary

1. Context and Applications

2. Objectives

3. IST-PDWZ Video Codec

4. Frame Interpolation

5. Experimental Results

6. Conclusions and Future Work

EC–SIP–M 200501/07/05

Conclusions & Future Work

– New motion compensated frame interpolation tools are proposed and compared in a DVC framework.

– Major gains in RD performance:– Essentially due to spatial motion smoothing.

– Future Work:– Adaptation to longer GOPs.– Improve performance when large camera movements occur.– Iterative motion refinement approach using decoded image and

keyframes.

Thanks for your attention !