A Fast and Efficient Multi-View Depth Image Coding Method Based on Temporal and Inter-View Correlations of Texture

Images

Jin Yong LeeHo Chen WeyDu Sik Park

IEEE Transactions on CSVT 2011

OutlineIntroductionProposed Method

Temporal CorrelationTexture and Depth View SynthesisInter-View Correlation

EvaluationsCoding PerformanceEncoding Complexity AnalysisSubjective Quality Assessment

IntroductionEncode video information of each view individually with

the H.264/AVCA multi-view video coding structure with hierarchical B

pictures

Multi-view video plus depth

OutlineIntroductionProposed Method

Temporal CorrelationTexture and Depth View SynthesisInter-View Correlation

EvaluationsCoding PerformanceEncoding Complexity AnalysisSubjective Quality Assessment

Temporal CorrelationStationary regions have similar pixel values for

successive framesUse sum of the squared difference(SSD) to

measure the temporal correlationIf SSD

=> Strongly Correlated Texture images

Depth images

I P

Temporal Correlation

Texture images

Depth images

P IB

Temporal Correlation

OutlineIntroductionProposed Method

Temporal CorrelationTexture and Depth View SynthesisInter-View Correlation

EvaluationsCoding PerformanceEncoding Complexity AnalysisSubjective Quality Assessment

Texture and Depth View SynthesisSynthesis the next depth view by 3D image warping

Translate gray values into real depth

Project the original points into 3D world

Reproject these 3D points into the image plane of

neighboring view

Hole-filling

Texture and Depth View SynthesisThe pixel position(x,y) at the reference frame can

be projected into a 3D point (u,v,w)

The corresponding pixel location of the virtual image

[ 𝑢𝑣𝑤]=𝑅𝑟×𝐴𝑟− 1×[𝑥𝑦1 ]×𝑍𝑟 (𝑥 , 𝑦 )+𝑇 𝑟

()

[𝑥′

𝑦 ′

𝑧 ′ ]=𝐴𝑡×𝑅𝑡− 1×{[ 𝑢𝑣𝑤]−𝑇 𝑡}

Translate gray values into real depth

Project the original points into 3D world

Reproject these 3D points into the image plane of

neighboring view

Hole-filling

Texture and Depth View SynthesisSome pixels in the synthesized image are missing

or undefinedOcclusionPixel position quantization

Only consider the neighboring pixelsaround the holeIf a view is warped to the right view

position, the hole is filled with its left pixel

Translate gray values into real depth

Project the original points into 3D world

Reproject these 3D points into the image plane of

neighboring view

Hole-filling

Texture and Depth View Synthesis

OutlineIntroductionProposed Method

Temporal CorrelationTexture and Depth View SynthesisInter-View Correlation

EvaluationsCoding PerformanceEncoding Complexity AnalysisSubjective Quality Assessment

Inter-View Correlation

1. A depth image of I-view(), a texture image of I-view() and P-view() are encoded

2. Synthesized a virtual texture image from the reconstructed and

3. If SSD => skip the block

Texture images

Depth images

I-view P-view

Synthesized Image

texturedepth

Inter-View Correlation

Evaluations

Coding performanceEncoding complexity analysisSubjective Quality Assessment

Coding performanceSimulation conditions

Test sequences

Coding performance

Balloons

Kendo

Book Arrival

Newspaper

Champagnetower

Pantomime

BDBR : Average bit-rate difference in % over the whole range of PSNRBDPR : Average PSNR difference in dB over the whole range of bit-rates

Coding PerformanceBalloons

Kendo Book Arrival

Newspaper Champane tower Pantomime

Encoding Complexity AnalysisETR and RM indicate the total encoding time

reduction and the reduced number of the modes performing the RD optimization in percentage

SB and VS represent the proportion of skipped macro blocks in depth image and the time required for the view synthesis process

Encoding Complexity Analysis

Subjective Quality Assessment15 professional subjectsTwo stereoscopic view pairs were rendered using

the depth images decoded by the original method and the proposed method respectively

Subjective Quality AssessmentY-axis indicates a differential score that subtract

a score of the original method from the proposed method