M. Neri, A. Campi, R. Suffritti, F. Grimaccia, P. Sinogas, O. Guye, C. Papin, T. Michalareas, L. Gazdag, I, Rakkolainen

SKYMEDIA – UAV-BASED CAPTURING OF HD/3D CONTENT WITH WSNAUGMENTATION FOR IMMERSIVE MEDIA

EXPERIENCES

FP7 IST-SKYMEDIA

Skymedia goals

Aims at exploring, designing, and demonstrating a novel multimedia end-to-end architecture focused on immersive media experiences to audience during live events

Multimedia Service Platform real-time HD/3D processing augmented HD/3D video

Unmanned Aerial Vehicle (UAV) and Wireless Sensors Network (WSN) are enabling technologies UAVs capture HD and 3D videos and images from the sky Wireless sensors are able to acquires side-information be

fused with the main audiovisual contents

2

FP7 IST-SKYMEDIA

Key technologies

Innovative 3D coding methods with MVD (Multi-view plus Depth), to be executed in real-time

Advanced transcoding methods that work in real-time to filter and adapt multimedia content to different distribution networks

Mid-air screens materializing high resolution screens By means of vaporized water

UAV to capture HD video and images from the sky

Wireless Sensors to form an Augmentation Network (WSAN) deployed on ground to enrich the multimedia content with side-information To be fused with the main audiovisual content

3

FP7 IST-SKYMEDIA

Conceptual Architecture4

FP7 IST-SKYMEDIA

Level Sub-System Architecture5

FP7 IST-SKYMEDIA

UAV-Based Content Capturing

Technical challenges an unobtrusive and silent video/image capturing data from a ground sensor network need to be collected

and aggregated

Progress beyond the state-of-the-art carries one or more cameras

Vibrations, 3D/HD real time video images (H.264/AVC) kinds of sensor devices

, payload size, limited resources, power consumption

6

FP7 IST-SKYMEDIA

Wireless Sensor AugmentationNetworks and Data Fusion

Technical challenges different types of sensors in dynamic environment with

different requirements sport games like soccer, rugby, athletics, or motor sports, etc.

aggregate data from both ground sensors and airborne sensors

With user generated content

Progress beyond the state-of-the-art wireless multi-sensor network

multi hop network with either a single or a multi sink structure energy management algorithms improved network protocols

data fusion development of tools capable of quickly managing the various

datasets and of fusing data

7

FP7 IST-SKYMEDIA

Multimedia Service Platform

Technical challenges Provide real–time immersive services to end-users platforms

different constraints, heterogeneous input/output computing power, storage, screen resolution, user interface, sensors and actuators

Progress beyond the state-of-the-art Collects pre-formatted multimedia data

view point selection and stereoscopic video enrichment and personalization

adaptation according to device (smartphone, HD/3D TV screen) real-time management of the HD/3D (MPEG4 MVC, JPEG2000)

processing in relation with WSAN data with user interactivity to choose a point of view metadata that are displayed on the HD/3D screen

Video-on-Demand is a mandatory multimedia service

8

FP7 IST-SKYMEDIA

Data path for live and VoD services9

FP7 IST-SKYMEDIA

Enabled Services (1/2)

Real time HD video content captured from the UAV platfor over the live scene Special real-time transcoding methods is needed to

enrich and adapt multimedia content focused on bit-rate reduction, format conversion and

resolution optimization

Telemetry Data remote measurements regarding live event players

parameters delivered to a number of different users

Location Based Services Ex. touristic information

10

FP7 IST-SKYMEDIA

Enabled Services (2/2)

Augmented Reality Live capturing, time constraints are particularly

stringent Processing center should be coupled with efficent data

gathering network

Immersive framework for mobile applications Spatial immersion: utilizing the knowledge of users Psychological immersion: offering the user an

experience that is updated and related to events Sensory immersion: utilizing state of the art available

sensors on the handset (cameras, GPS, gyroscopes, microphones, tough UI, etc.)

11

FP7 IST-SKYMEDIA

Applications and Use Cases

Marathon race different kinds of sensors (biosensor)

runner’s shoes or suit, and fixed sensors and cameras placed along the marathon path

athlete health status, his relative position in the race, shifting from his best previous performance, deviation from the race record and so on

Pre-Test Turin Marathon, November 2011 “Tutta dritta”, April 2012

Final demonstration Turin Marathon, November 2012

12

FP7 IST-SKYMEDIA

Marathon Use Case13

FP7 IST-SKYMEDIA

Applications and Use Cases

Outdoor music concert The active part of a concert is a small perimeter event

with little movement. The audience of a big concert can, however, cover quite large an area. A freely flying UAV gives more flexibility for filming and other sensors than e.g., cameras on static balloons or on moving wire systems, which can move only very limitedly

Motorized sport augment the experience of motorized sports fans avoiding them to become bored during the race

14

FP7 IST-SKYMEDIA

Conclusion

Multimedia architecture able to create immersive media experiences www.ict-skymedia.eu

Key roles UAV platform, enabling content capturing from the sky WSAN to collect sensor data advanced Multimedia Service Platform able to manage in

real-time HD/3D processing

The project is still ongoing Main test event planned in Turin Mrathon 2012 Integration process has been started with preliminary

encouraging results

15

FP7 IST-SKYMEDIA

Thanks

16