Improving Scene Cut Quality for Real-Time Video Decoding

Giovanni Motta, Brandeis UniversityJames A. Storer, Brandeis UniversityBruno Carpentieri, Universita’ di Salerno

Outline

IntroductionH.263+ and TMN-8 Rate ControlProblem DescriptionOptimal Algorithm based on Dynamic

ProgrammingExperimental ResultsConclusions and Future Research

Introduction

High variability in video sequences may cause the encoder to skip frames

Frame skipping occurs after a “scene cut” (i.e. when MC-prediction model fails)

If the encoder has some look-ahead capability it is possible to improve quality in proximity of scene cuts

H.263+ Video Encoding

State of the art Video Coding

MC-prediction and DCT coding

I and P macroblocks

Rate control

CC

T Q

Q

T

P

p

t

qz

q

v

Videoin

Tovideomultiplexcoder

TQP

CCptqzqv

–1

–1

TransformQuantizerPicture Memory with motion compensated variable delay

Coding controlFlag for INTRA/INTERFlag for transmitted or notQuantizer indicationQuantizing index for transform coefficientsMotion vector

TMN-8 Rate Control

I/P Frame and MB decisionsTarget bit rate for each frameRD optimized bit allocation for MBsBuffer control

Problem Description

Bits per frame (std100.qcif)

0 100 200 300 400 500 600 700 800 9000

2,000

4,000

6,000

8,000

10,000

Problem Description

PSNR and Bits per frame across a scene cut

0 5 10 15 20 25 30 35 40 45 500

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

10,000

PSNR per frame (times 100)bits per frame

Problem Description

Frame n has several “I” macroblocksEncoder is forced to skip n+1, n+2, n+3Frame n-1 frozen on receiver’s displayFrame n+4 has a large prediction errorEncoder forced to skip frame n+5

Basic Idea

Avoid extra skipping and improve quality by selecting which frame should be encoded after a scene cut

Assumption: Encoder has look-ahead capability

Simplified approach

TMN-8 behavior

Last frame of the skipped sequence encoded

Simplified approach

PSNR and Bits per frame across a scene cut

0 5 10 15 20 25 30 35 40 45 500

1,000

2000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

10,000

PSNR Y per frame (x 100)bits per frame

0 5 10 15 20 25 30 35 40 45 500

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

10,000

PSNR per frame (times 100)bits per frame

Optimal Algorithm

Minimizes the number of skipped framesGeneralization of the text-paragraphing

algorithmAssumptions:

When the quality of F[i-j] is fixed to Q, the cost P[i, j] of predicting F[i] from F[i-j], is independent of how F[i-j] is encoded

P[i, j] P[i, j+1] P[i, 0], 1 j d

Optimal Algorithm

Compute P[i, 0] for each frameCompute P[i, j] for 1 j dBuild (right to left) two matrices

R[i, j]: maximum residual capacity when F[i], …, F[n] are encoded so that the first frame that is not skipped is predicted by F[i-j]

S[i,j]: number of skipped frames corresponding to residual capacity R[i, j]

Time is O(d2n) = O(n) (constant d 7)

Test Sequences

Std and Std100: concatenation of standard test sequences

Commercials: Sampled TV commercials

Experimental Results

Gain in Bit/PSNR in proximity of scene cuts (simplified method)

Experimental Results

Gain on whole sequence (simplified method)

Conclusions

Simple yet effective method to improve quality in proximity of scene cuts

Experiments with simplified method show improvements of 14-30% (in Bit/PSNR)

Suitable for encoders of the MPEG family, provided that encoder has look-ahead capability

Decoding is unaffected

Future Research

Assess quality improvement when using optimal algorithm

Experiment with progressive transmission to eliminate frozen frame displayed by the decoder