Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens [email protected] Error Concealment for

Technical University of BerlinCommunication Systems Group

Director: Prof. Thomas Sikora

Carsten [email protected]

Error Concealment for Stereoscopic Sequences

ITG Fachausschusstagung 3.2, Juni 2006

Error Concealment for Stereoscopic Sequences 2Communication Systems GroupC. Clemens

Technische Universität Berlin

• Monoscopic Error Concealment strategies are not well suited for stereoscopic scenario

• Assumtions

– independently coded views of a stereo image pair

• remaining redundancies between the channels, which can be utilized for error concealment

– block based coding (16x16 blocks)

Introduction



Algorithm Overview

Identification of corresponding region– feature extraction– feature matching along epipolar line– selection of matches (M-estimator/RANSAC)

Projective Transformation– initial parameter set from matches– optimization by Newton Method

Smoothing– only in case of discontinuities of depth



Matching and Transformation

projective transformation:



Selection of feature matches

• M-estimator– uses all matches with different weights– In some cases the transformation fails, because pixels from

outside the image were warped into the erroneous burst.

• RANSAC (random sample consensus)– uses a number of subsamples (four feature matches) – minimize the sum of squared residues of the boundary region:

RANSAC yields better results than M-estimator



Adapted Newton Method

• Find the optimal transformation parameter by minimizing a cost function C(k):

• b is the Border Region of the erroneous block burst

T (k ;(xl, yl))

left view right view

b




• Iteration step:

• Problem I: Local minimum solution

• Initial Parameter set is of prime importance

Cost function C(k) over horizontal and vertical translation parameter




• Problem II: Convergence of Newton method

• Successivly decreasing of border pixel size L after every minimum search

Speed of convergence of the Newthon algorithm for different border sizes L



3D Block Smoothing

• In case of great discontinuities in depth (variation of disparity) we perform a linear smoothing algorithm towards the surrounding pixel region (3D-BS)

• Minimization of the intersample variance between neighboring samples and to the block borders



HQ EC: Results / Example



Results / Subjective Evaluation

• Subjective Simulation Results:

– Double Stimulus Continuous Quality Scale Method (DSCQS) as phsychovisual test with 15 subjects

– Shutterglasses (StereoGraphics)

DSCQS method

-0,5

0,5

1,5

2,5

3,5

4,5

5,5

6,5

hall castle tower over all

monoscopic NEWTburst NEWT 3D-BS

DMOS



Fast EC: Algorithm Overview

• Block Search– Directional Diamond Search– SAD

• Side Match Distortion ||1 out

jinjsm bb

Nd



Fast EC: Matching Example „Hall“

• SMD determined for each possible position• Position with minimum SMD selected• Block used for reconstruction



Fast EC: Subjective Evaluation

BALLOON DMOS S.D. conf. interval

Interpolation 5,69 1,19 ±0,65

SBS 4,39 1,12 ±0,62

TBS 2,52 2,64 ±1,45

TSBS 2,49 1,37 ±0,75



Simple Matching - Example

• ‘Balloons’ 720x480, corrupted frame and features




• Reference frame (temporal) with features




• Reference blocks




• Reconstructed frame



Publications

• K. Günther, C. Clemens, and T. Sikora

A Fast Displacement-Estimation Based Approach For Stereoscopic Error Concealment

PCS 2004, San Francisco

• C. Clemens, M. Kunter, S. Knorr, and T. Sikora:

A hybrid approach for error concealment in stereoscopic images

WIAMIS '04, Lissabon

• M. Kunter, S. Knorr, C. Clemens, and T. Sikora:

A gradient based approach for stereoscopic error concealment

ICIP '04, Singapore

• S. Knorr, C. Clemens, M. Kunter, and T. SikoraRobust Concealment for Erroneous Block Bursts in Stereoscopic Images3D Data Processing, Visualization, and Transmission (3DPVT'04), Thessaloniki, Greece

• K. Günther, C. Clemens, and T. Sikora

A Fast Displacement-Estimation Based Approach For Stereoscopic Error Concealment

PCS 2004, San Francisco

• C. Clemens, M. Kunter, S. Knorr, and T. Sikora:

A hybrid approach for error concealment in stereoscopic images

WIAMIS '04, Lissabon

• M. Kunter, S. Knorr, C. Clemens, and T. Sikora:

A gradient based approach for stereoscopic error concealment

ICIP '04, Singapore

• S. Knorr, C. Clemens, M. Kunter, and T. SikoraRobust Concealment for Erroneous Block Bursts in Stereoscopic Images3D Data Processing, Visualization, and Transmission (3DPVT'04), Thessaloniki, Greece

Documents

Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens [email protected] Error Concealment for