3D Video: From Stereo to Multi-View

Fraunhofer Institute for TelecommunicationsHeinrich-Hertz-Institut Berlin Einsteinufer 3710587 BerlinGermany

3D Video: From Stereo to Multi-View

+49 30 310 02 – [email protected] http://www.hhi.fraunhofer.de

Karsten Müller

TEWI - Kolloquium,Universität Klagenfurt

Kars

ten

Mül

ler

Overview

• Introduction, scenarios, requirements

• 3D Video: Stereo Video + MVC

• 3D Video: 3D Video Formats & Coding

• International 3D media research and development

• Summary

18.03.2010 Universität Klagenfurt

Slide 2

Kars

ten

Mül

ler

3D Video Introduction

• Provision of left and right eye view• Added depth sensation • Resolved 2D viewing

ambiguity• Additional features

(e.g. free viewpoint, depth-controlled object insertion)

3D Video is more than adding 1 dimension to 2D!

Slide 3


3D Video Scenarios3D cinema• Increasing number of 3D productions• First studios start to release all productions also in 3D• Technology: stereoscopic 3D, glasses based

3D home entertainment• Different types of displays available: stereoscopic, auto-stereoscopic

with 2 … N views• Various technologies, input formats and display sizes• Glasses based systems may not be acceptable

3D mobile• auto-stereoscopic 2 view display with fixed viewing position• Good 3D viewing in spite of small display sizes

Kars

ten

Mül

ler

Slide 4


18.03.2010© Fraunhofer HHI

Folie 5

Kars

ten

Mül

ler

Stereo Video Solution

• Original Cameras = Views for Display

• Production format specification (e.g. SMPTE)

• Rectification, color correction, format conversion

• Conventional Stereo Format, e.g. Side-by-side

• Multi-view video coding (MPEG-4 MVC/H.264)

• High-resolution glasses-based stereo display

Stereo-Capture

Post-processing

MVC-Coding

MVC-Decoding

ProductionFormat

TransportFormat

Transport Format

Stereo-Display

Display Format

3D format: Conventional Stereo video (CSV)Ka

rste

n M

ülle

r

Slide 6


Original Image Source: Interactive Visual Media Groupof Microsoft Research

Kars

ten

Mül

ler

Current Coding Standards for Stereo Video

• Advanced Video Coding (MPEG-4 AVC / H.264)– Simulcast– Stereo SEI (Supplementary Enhancement Information)– Auxiliary Picture Syntax

• MPEG-C part 3 (container format for V+D, e.g. with AVC coding)

• Multi-view Video Coding (MPEG-4 MVC / H.264)

Slide 7


Kars

ten

Mül

ler

Multi-view Video Coding

• Joint Coding of camera views with similar content

Slide 8


Kars

ten

Mül

ler

MVC Coding Method

• Coding approach for exploiting temporal and inter-view dependencies using MPEG4-AVC/H.264

• Usage of hierarchical B pictures in temporal direction

• Usage of P or hierarchical B pictures in inter-view direction

• Frame reordering to optimize memory usage

No H.264 changes required for multi-view coding

• Standardized in July 2008• Adopted for 3D Blu Ray in Dec. 2009

Slide 9


Kars

ten

Mül

ler

MVC Coding Structure- Picture Reordering

Slide 10


Kars

ten

Mül

ler

MVC Coding Structure

Slide 11


Kars

ten

Mül

ler

MVC Camera Setting Adjustment - Star

Slide 12


Kars

ten

Mül

ler

MVC Camera Setting Adjustment – 2D Array

Slide 13


Kars

ten

Mül

ler

MVC Coding Results

• Coding structure allows AVC coder, to select the best inter-view-temporal neighbors within the MVC sequence

• Coding gain obtained by usage of hierarchical B-frames and by exploitation of inter-view dependencies

Slide 14


Kars

ten

Mül

ler

MVC Coding Results – delta_PSNR

Slide 15


Broadcasters currently concentrate on stereo and will not invest in new infrastructuresThey will adopt a system, which they can feed through existing pipes: e.g. anaglyph or multiplexed stereo:

checkerboardside-by-sideabove-below etc.

Will Stereo TV be the long term solution?

anaglyph polarized shutter

Kars

ten

Mül

ler

Slide 16


Short Term 3DTV will be Stereo TV

The Answer is Given by Our ChancellorKa

rste

n M

ülle

r

Slide 17


3DTV viewing conditions are more complex than in 3D cinema:• Glasses are generally not accepted in the living room • There are many different 3D display technologies • Varying viewing conditions (distance, display size)• User preferences have to be considered• Not all users can see stereo (≈10%)• Not all users want to see stereo• Users might not want to see all programmes in stereo

Kars

ten

Mül

ler

Slide 18


Special Requirements for 3DTV

Kars

ten

Mül

ler

Multi-view Displays

• The most common form of multi-view displays utilise either lenticularscreens or parallax barriers.

• Lenticular screens consist of a series of vertically aligned or slanted cylindrical lenses and parallax barriers vertically aligned apertures.

• Light is guided in the appropriate directions by either focussing it or by blocking unwanted rays.

VIEW 1 VIEW 2 VIEW 3

LENTICULAR

VIEW 1 VIEW 2 VIEW 3

PARALLAX BARRIER23 1

1 4 44 11

SCREENSlide 19

18.03.2010 Image Source: De Montfort University, UK

Kars

ten

Mül

ler

3D Video Color only: Assumption 1

• Multi-view Displays require N views (e.g. 5,9,22,50)• Why not transmitting N color views with MVC?

Slide 20


Kars

ten

Mül

ler

MVV and MVC Restrictions

• Pure coding of multi-view video

• Still linear dependency of bit rate from number of views

• Number and position of views fixed over the whole 3D chain

• No baseline adaptation for different displays

• Number of Views fixed in MVV/MVC

Slide 21


Kars

ten

Mül

ler

MVC Restrictions

• Coding experiments on camera density

Original camera distance

Slide 22


Kars

ten

Mül

ler

MVC Restrictions (2)

• Coding experiments on camera density with Rena test sequence

average rate average rate per cameraSlide 23


Kars

ten

Mül

ler

3D Video Color only: Assumption 2

• Multi-view Displays require N views (e.g. 5,9,22,50)• Why not transmitting 2 or 3 color views with MVC

and reconstructing N views at the receiver?

Slide 24


Color Only Intermediate ViewKa

rste

n M

ülle

r

Slide 25

18.03.2010

Original Image Source: Interactive Visual Media Group of Microsoft Research

Depth mapKa

rste

n M

ülle

r

Slide 26

Color information 256 depth layers



Example Reconstruction

Pure image blending Depth image basedrendering


Kars

ten

Mül

ler

Slide 27


Kars

ten

Mül

ler

Advanced 3D Video Solutions Required

• Color-only multi-view video too restricted– High number of views cannot be efficiently

compressed and transmitted– View synthesis of new views at the receiver

side from color-only data gives badreconstruction quality

• Additional Scene Geometry information required, e.g. per-pixel depth/disparity data


Slide 28


Folie 29

Kars

ten

Mül

ler

Depth-enhanced 3D Video

• Stereo / multiview capturing, depth provision

• Production format specification (e.g. SMPTE)

• Rectification, color correction, format conversion

• 3D format specification

• 3D video coding (e.g. MPEG)

• High quality intermediate view synthesis (e.g. 2 view + 2 depth N view)

• High-resolution stereo/multiview display

Capture

Post-processing

Coding

View Synthesis

ProductionFormat

TransportFormat

Transport Format

Display

Display Format

Kars

ten

Mül

ler

3D Video Formats

• Conventional stereo video (CSV)• Mixed resolution stereo (MRS)• Video plus depth (V+D)

• Multiview video (MVV)• Multiview video + depth (MVD)• Layered depth video (LDV)

• Depth-enhanced stereo (DES)

Slide 30


assuming multiple views with color & depth… Ka

rste

n M

ülle

r

Slide 31


Stereo video (CSV)

CSV

Kars

ten

Mül

ler

Slide 32


Mixed resolution stereo (MRS)

MRS

Kars

ten

Mül

ler

Slide 33


Video plus depth (V+D)

V+D

Kars

ten

Mül

ler

Slide 34


MVV

Multiview video (MVV)Ka

rste

n M

ülle

r

Slide 35


MVD

Multiview video plus depth (MVD)Ka

rste

n M

ülle

r

Slide 36


LDV

Layered depth video (LDV)Ka

rste

n M

ülle

r

Slide 37


DES

Depth-enhance Stereo (DES)Ka

rste

n M

ülle

r

Slide 38


Kars

ten

Mül

ler

Towards a new 3D Video Coding Standard

• MPEG develops a new 3D video coding standard

• Motivation for new standard:– Decouple production from coding format– MPEG-4 AVC/H.264 only optimized for 2D color video, but not for

depth information

Slide 39


Advanced 3DTV concept based on MVDKa

rste

n M

ülle

r

Slide 40


3D components under consideration:– Transport format– 3D video coding methods– Intermediate view synthesis


Folie 41

Kars

ten

Mül

ler

Coverage of 3D Video Coding

Capture

Post-processing

Coding

View Synthesis

ProductionFormat

Transport Format

Transport Format

Display

Display Format

Kars

ten

Mül

ler

Challenges for 3DVC

• Consider capturing technology, i.e. maximal 2-3 recorded views

• Break linear dependency of coding bit rate from number of target views (e.g. MVC)

• Provide scene geometry data in general form, i.e. pixel-wise depth data

• Consider statistical properties of depth (and supplementary) data

• Consider new quality evaluation methods for intermediate views

• Provide high-quality view synthesis for continuous viewing range

• Decrease depth and coding errors

Slide 42


Kars

ten

Mül

ler

Coding Experiments MVD with MVC

• Joint color and depth coding for optimal intermediate view quality

• Evaluation of total bit rate (e.g. of 2 color and 2 depth maps) vs. quality of decoded synthesized views

• View synthesis is part of coding optimization

• Reduction of coding and interpolation artifacts through high-quality view synthesis

Slide 43


Kars

ten

Mül

ler

Coding Results Intermediate View

• Ballet (JMVM 7.0.1, GOP16, κ=½)

Slide 44


Kars

ten

Mül

ler


• Breakdancers (JMVM 7.0.1, GOP16, κ=½)

Slide 45


Kars

ten

Mül

ler

Color Depth Bit Rate Distribution

• Objective and subjective examples for optimal and non-optimal bit rate distribution

• Variation of color and depth quality

• Tradeoff between synthesized views and views at original positions

Slide 46


Kars

ten

Mül

ler


• Ballet (JMVM 7.0.1, GOP16, κ=½)

C30D30

C24D40

Slide 47


Kars

ten

Mül

ler

Coding Results across Viewing Range

• Ballet (JMVM 7.0.1, GOP16)

Slide 48


Kars

ten

Mül

ler


• Breakdancers (JMVM 7.0.1, GOP16, κ=½)

C30D36

C36D30

Slide 49


Kars

ten

Mül

ler

Coding Results across Viewing Range

• Breakdancers (JMVM 7.0.1, GOP16)

Slide 50


View Synthesis for MVD

– High quality view interpolationis essential for 3D video applications with MVD data.

– Processing steps:- Layer Extraction- Layer Projection- Hole Filling + Filtering

Kars

ten

Mül

ler

Slide 51


Kars

ten

Mül

ler

Synthesis Improvements (Ballet)

compressed uncompressed

simple merging

layer-basedview synthesis

Slide 52


International R&D in 3D Media

• ISO/IEC MPEG 3DVC (3D video coding)• SMPTE Task Force of 3D to the home (3D master format)• ITU-R SG6• DVB – TM - ES Stereoscopic (3D) TV• 3D@Home Consortium• Association in Korea, Japan,…• EU 3D, Immersive, Interactive Media Cluster

• International Conferences– 3DTV-CON– 3DPVT

Kars

ten

Mül

ler

Slide 53


3D Video Coding Group in MPEG

• Nagoya Univ. • NTT• UPM• Philips• GIST• ETRI• JVC• Poznan Univ. of

Technology• Thomson• MERL• Peking Universtity• Tsinghua Univ.• Huawei• Xidian University• Telefonica

• Samsung• LG Electronics• Nokia• NICT• Kwangwoon Univ.• Sharp• Fraunhofer• NCTU/ITRI• Motorola• NXP• Ericsson• Logitech• Orange/France

Telecom • FUB• Sony

• Zhejiang Univ.• ASTRI• Qualcomm

• Coordination:• Karsten Müller (HHI)

Anthony Vetro (MERL)

Kars

ten

Mül

ler

Slide 54


Kars

ten

Mül

ler

3D@Home Consortium

• Steering Team 1: Content Creation & ProductionObjective: To focus on projects which improve the quality of 3D content for the home audience.

• Steering Team 2: Content Storage, Transmission & DistributionObjective: To develop useful definitions and guidelines for the successful storage, transmission and distribution of 3D content.

• Steering Team 3: 3D PromotionObjective: To promote the adoption of 3D into the home by promoting 3D format and the 3D@Home Consortium.

• Steering Team 4: 3D DisplaysObjective: To identify issues around 3D Displays and associated hardware.

Slide 55


Kars

ten

Mül

ler

3D@Home Consortium

Slide 56


Kars

ten

Mül

ler

EU 3D, Immersive, Interactive Media Cluster• Currently, a number of EU projects develop parts and products of the 3D

media chain for certain application areas, e.g. 3D home entertainment or mobile services

• Enhanced success of these EC funded projects and subsequent associated commercialization efforts if simultaneous operations of individual projects can be further orchestrated towards a strong mutual presence

• Better highlight of Europe‘s good position in worldwide R&D

• 3D, Immersive, Interactive Media Cluster is an umbrella structure for the projects and provides international contact and information about project activities

• 3D Media, Immersive, Interactive Cluster is one of the main drivers towards "3D Media" in the context of the "Future Internet" activities supported by EU Commission (DG INFSO)

Slide 57


Kars

ten

Mül

ler

3DII Media Cluster – Current Projects

• Leadership: Prof. Levent Onural and Dr. Karsten Müller

• Projects: Status: Framework Program• 3DTV ended 2008 FP6• 3DPHONE ongoing FP7• MOBILE3DTV ongoing FP7• 3D4YOU ongoing FP7• 2020 3D Media ongoing FP7• 3DPresence ongoing FP7• MUTED ended 2008 FP6• HELIUM3D ongoing FP7• Real3D ongoing FP7• i3DPost ongoing FP7• VICTORY ended 2009 FP7

3D for MobileSystems

3D for BroadcastSystems

3D DisplayTechnology

3D ContentDescription

Slide 58

3D Videoconference


Kars

ten

Mül

ler

3DII Media Cluster – New Projects

• Projects: Status: Framework Program• DIOMEDES new 2010 FP7• MUSCADE new 2010 FP7• 3D VIVANT new 2010 FP7• BEAMING new 2010 FP7• FINE new 2010 FP7• SKYMEDIA new 2010 FP7• SALA3D new 2010 FP7• 3DLife new 2010 FP7• FascinatE new 2010 FP7

Slide 59


Kars

ten

Mül

ler

Summary

• 3D technology is maturing due to world wide development from capturing to display

• First systems are in use, mainly using stereoscopic displays and MVC coding

• New challenging research topics in all areas of 3D video:– Reliable capturing devices with synchronized views– Improved time-consistent depth estimation– Geometry-enhanced multi-view coding (e.g. 3DVC)– Robust view synthesis– New high-resolution multi-view displays

(e.g. 50 views, each with HDTV resolution!)

Slide 60
