Upload
donguyet
View
223
Download
2
Embed Size (px)
Citation preview
page
1pa
ge 1
R. Schäfer - Immersive Medien und 3D-Video
page
1
Fraunhofer-InstitutFraunhofer-Institut ffüürrNachrichtentechnikNachrichtentechnik
Heinrich-Hertz-Heinrich-Hertz-InstitutInstitutRalf Ralf SchSchääferfer
[email protected]@hhi.dehttp://ip.hhi.dehttp://ip.hhi.de
Immersive Immersive MedienMedienund 3D-Videound 3D-Video
page
2pa
ge 2
R. Schäfer - Immersive Medien und 3D-Video
page
2
OutlineOutline
Immersive Media Examples Interactive Media and Free Navigation Free Navigation in Video Scenes Examples Free Viewpoint Video Summary
page
3pa
ge 3
R. Schäfer - Immersive Medien und 3D-Video
page
3
ImmersiveImmersive Media Media –– CostsCosts vs. Sensation vs. Sensation
CDTV
Sensation
Costs
ImmersiveMedia
VoD
VCR
Interactive TV
AmericanFootball
Theatre
Rock Concert
Cinema
IMAX
Real Live Sensation
page
4pa
ge 4
R. Schäfer - Immersive Medien und 3D-Video
page
4
ImmersiveImmersive Display (UHDTV, Display (UHDTV, D-CinemaD-Cinema))
page
5pa
ge 5
R. Schäfer - Immersive Medien und 3D-Video
page
5
ExampleExample: : ImmersiveImmersive Video Video ConferencingConferencing –– im.pointim.point
• Seamless transition between the real and virtual world• Head motion parallax• Life-sized upper body images representing gestures and body language• 3D representation of the remote participants including provision of eye contact
Audio Speakers
Semi-circulartable
61” Plasmadisplay
Cameras
page
6pa
ge 6
R. Schäfer - Immersive Medien und 3D-Video
page
6
ExampleExample: : LargestLargest Panorama Panorama WorldwideWorldwide
Gasometerin Leipzig
Printed panorama with 60 m diameterand 36 m hight (weight > 1 ton)
page
7pa
ge 7
R. Schäfer - Immersive Medien und 3D-Video
page
7
page
8pa
ge 8
R. Schäfer - Immersive Medien und 3D-Video
page
8
CylindricalCylindrical ProjectionProjection withwith HHIHHI‘‘ss CineCardCineCardultraultra
υ
υ
Single level Multiple levels
CineCardultra: Horizontal & vertical soft edge blending Warping for cylindrical and parabolic projection
page
9pa
ge 9
R. Schäfer - Immersive Medien und 3D-Video
page
9
InteractiveInteractive Media Media Allow to be active in some way instead of just
being a passive consumer Enabled through convergence on
telecommunications, consumer electronics andcomputer science
Well-known: linking of multimedia data forinteractive access
Multimedia DVD, Internet presentations,Multimedia Home Platform (MHP)
page
10
page
10
R. Schäfer - Immersive Medien und 3D-Video
page
10
Free NavigationFree Navigation Arbitrary selection of viewpoint and direction within real
world audio-visual scenes Well-known from
computer graphics, virtual reality, computer games synthetic scenes, objects or static views of real objects or restricted to predefined viewpoints (e.g. Formula 1
on Premiere)
page
11
page
11
R. Schäfer - Immersive Medien und 3D-Video
page
11
Free Navigation in Real Free Navigation in Real ScenesScenes New technology allows generation of intermediate
views in between real existing camera view Image-Based rendering, 3D reconstruction Allows e.g. viewing sports or theatre from arbitrary
viewpoints New technologies for the whole chain:
Acquisition, signal processing, modelling,representation, compression, transmission,rendering, display
page
12
page
12
R. Schäfer - Immersive Medien und 3D-Video
page
12
SceneScene RepresentationsRepresentationsMore geometry -based More image-based
StaticTexture,SingleGeometry
View-dep.Texture,SingleGeometry
StaticTexture,View-dep.Geometry
View-dep.Texture,View-dep.Geometry
Sprites
Layered -Depthimages
Lumigraph
Light-fieldrendering
Incarnations of 3D model
Conventional graphics pipeline Warping Interpolation
page
13
page
13
R. Schäfer - Immersive Medien und 3D-Video
page
13
Systems with curved mirrors www.remotereality.com
Omni-directional VideoOmni-directional Video
page
14
page
14
R. Schäfer - Immersive Medien und 3D-Video
page
14
Courtesy of Prof. Yokoya, Nara Institute of Scienceand Technology
Omni-directional Video Omni-directional Video –– Example 1 Example 1
page
15
page
15
R. Schäfer - Immersive Medien und 3D-Video
page
15
Omni-directional Video Omni-directional Video –– Example 2 Example 2
page
16
page
16
R. Schäfer - Immersive Medien und 3D-Video
page
16
Patches
InteractiveInteractive Navigation in 3D Panoramas Navigation in 3D Panoramas
page
17
page
17
R. Schäfer - Immersive Medien und 3D-Video
page
17
Creation of 3D PanoramasCreation of 3D Panoramas By using several shots of a still camera and stitching techniques By using video and mosaiquing techniques
page
18
page
18
R. Schäfer - Immersive Medien und 3D-Video
page
18
Current view on screen
Visibility sensor
InteractiveInteractive DecodingDecoding and and RenderingRenderingTiles can be streamed asJPEG or JPEG2000 files
page
19
page
19
R. Schäfer - Immersive Medien und 3D-Video
page
19
Capturing a real dynamic scene with N cameras Transformation into a special data representation Interpolation of arbitrary intermediate views Allows free navigation within the scene (within
practical limits)
Free Viewpoint Video (Free Viewpoint Video (FVVFVV))
page
20
page
20
R. Schäfer - Immersive Medien und 3D-Video
page
20
X
Virtual Camera
Path
Example: Eye Vision using tracking camerasExample: Eye Vision using tracking cameras
page
21
page
21
R. Schäfer - Immersive Medien und 3D-Video
page
21
No reconstruction of 3D geometrie => many cameras No individual interaction possible Live broadcast event: Super Bowl 2001, 30 cameras Courtesy of Prof. Takeyo Kanade, Carnegie Mellon Univ.
Eye VisionEye Vision
page
22
page
22
R. Schäfer - Immersive Medien und 3D-Video
page
22
Visual Hull ReconstructionVisual Hull Reconstruction
Synchronized Synchronized multi-video recordingmulti-video recording
Background subtractionBackground subtractionSilhouette extractionSilhouette extraction
Visual Hull Visual Hull reconstructionreconstruction
page
23
page
23
R. Schäfer - Immersive Medien und 3D-Video
page
23
Example: Temple Acquisition by PhotographsExample: Temple Acquisition by Photographs
page
24
page
24
R. Schäfer - Immersive Medien und 3D-Video
page
24
ProcessingProcessing StepStep: 3D : 3D MeshMesh ExtractionExtraction
page
25
page
25
R. Schäfer - Immersive Medien und 3D-Video
page
25
Processing Step: TexturingProcessing Step: Texturing
page
26
page
26
R. Schäfer - Immersive Medien und 3D-Video
page
26
Courtesy ofStephan Wuermlinet al.,ETH Zuerich,Switzerland
Multi camera video acquisitionMulti camera video acquisition
page
27
page
27
R. Schäfer - Immersive Medien und 3D-Video
page
27
SegmentationSegmentationCourtesy of StephanWuermlin et al.,ETH Zuerich,Switzerland
page
28
page
28
R. Schäfer - Immersive Medien und 3D-Video
page
28
Original sequence (1 camera)Original sequence (1 camera)
Source: ETH Zürich
page
29
page
29
R. Schäfer - Immersive Medien und 3D-Video
page
29
Free Viewpoint Video from 16 CamerasFree Viewpoint Video from 16 Cameras Results produced by HHI Same functionality as CG
objects free navigation, can be
viewed from anyviewpoint/direction
Integration into completescenes(virtual/augmented/real)
But: depict appearance,motion, deformation of realworld objects
page
30
page
30
R. Schäfer - Immersive Medien und 3D-Video
page
30
Courtesy of Microsoft Research, Redmont, WA, USA
Example: Depth-based RenderingExample: Depth-based Rendering
page
31
page
31
R. Schäfer - Immersive Medien und 3D-Video
page
31
ConclusionConclusion
Immersive Media is a challengig and excitingfield of research & development
There are many approaches to allowinteractivity and free navigation in video scenes
Free Viewpoint Video (FVV) is a new conceptfor video entertainment which enables the userto navigate in real scenes in the same way asin virtual worlds
page
32
page
32
R. Schäfer - Immersive Medien und 3D-Video
page
32
Fraunhofer Heinrich-Hertz-Institut
We put science into action
Thank you