Programmable edge-to-cloud virtualization fabric for the 5G ......2020/04/05 · This is achieved through real-time volumetric capturing that 3D reconstructs the appearance of the

Project co-funded by the European Commission under the Horizon 2020 Programme.

Programmable edge-to-cloud virtualization fabric for

the 5G Media industry

D6.2: 5G-MEDIA Immersive Media Pilot

Work Package: WP6: 5G-MEDIA Use Case Scenarios and Validation

Lead partner: CERTH

Authors: Nikolaos Zioulis, Alexandros Doumanoglou, Kyriaki Christaki, Emmanouil Christakis, Prodromos Boutis, Dimitrios Zarpalas [CERTH] Panagiotis Athanasoulis, Stamatia Rizou [SILO] George Agapiou [OTE] David Griffin, Khoa Phan, Morteza Kheirkhah, Miguel Rio [UCL] David Jiménez, Federico Alvarez, Javier Serrano, José Manuel Menéndez [UPM] Alberto Florez, Rocío Ortiz [TID] David Breitgand, Avi Weit [IBM] Kourtis Michail-Alexandros [NCSRD]

Delivery date (DoA): 29 February 2020

Actual delivery date: 11 March 2020

Dissemination level: Public

Version number: 1.0 Status: Final

Grant Agreement N°: 761699

Project Acronym: 5G-MEDIA

Project Title: Programmable edge-to-cloud virtualisation fabric for the 5G Media industry

Instrument: IA

Call identifier: H2020-ICT-2016-2

Topic: ICT-08-2017, 5G PPP Convergent Technologies, Strand 2: Flexible network applications

Start date of the project: 01 June, 2017

Duration: 33 months

Project co-funded by the European Commission under the Horizon 2020 Programme.

Revision History

Revision Date Who Description

0.1 28/01/2020 CERTH Initial draft, table of contents

0.2 18/02/2020 CERTH First round of inputs

0.3 25/02/2020 CERTH Second round of inputs

0.4 26/02/2020 CERTH Internal review

0.5 27/02/2020 NCSRD Testbed description

0.6 02/03/2020 CERTH Document consolidation

0.7 06/03/2020 OTE Testbed description

0.8 10/03/2020 CERTH Final document consolidation

Quality Control

Role Date Who Approved/Comment

Internal Reviewer

02/03/2020 BIT Minor changes, comments and suggestions for improvement

Internal Reviewer

03/03/2020 IRT Minor changes, comments and suggestions for improvement

5G-MEDIA - Grant Agreement number: 761699 D6.2 5G-MEDIA Immersive Media Pilot

Page 3 / 29

Disclaimer

This document may contain material that is copyright of certain 5G-MEDIA project beneficiaries and may not be reproduced or copied without permission. The commercial use of any information contained in this document may require a license from the proprietor of that information. The 5G-MEDIA project is part of the European Community's Horizon 2020 Program for research and development and is as such funded by the European Commission. All information in this document is provided "as is" and no guarantee or warranty is given that the information is fit for any particular purpose. The user thereof uses the information at its sole risk and liability. For the avoidance of all doubts, the European Commission has no liability with respect to this document, which is merely representing the authors’ view.

The 5G-MEDIA Consortium consists of the following organisations:

Participant number

Participant organisation name Short name

Country

01 ENGINEERING – INGEGNERIA INFORMATICA SPA ENG Italy

02 IBM ISRAEL - SCIENCE AND TECHNOLOGY LTD IBM Israel

03 SINGULARLOGIC ANONYMI ETAIREIA PLIROFORIAKON SYSTIMATON KAI EFARMOGON PLIROFORIKIS

SILO Greece

04 HELLENIC TELECOMMUNICATIONS ORGANIZATION S.A. - OTE AE (ORGANISMOS TILEPIKOINONION TIS ELLADOS OTE AE)

OTE Greece

05 CORPORACION DE RADIO Y TELEVISION ESPANOLA SA RTVE Spain

06 UNIVERSITY COLLEGE LONDON UCL United Kingdom

07 TELEFONICA INVESTIGACION Y DESARROLLO SA TID Spain

08 UNIVERSIDAD POLITECNICA DE MADRID UPM Spain

09 INSTITUT FUER RUNDFUNKTECHNIK GMBH IRT Germany

10 NEXTWORKS NXW Italy

11 ETHNIKO KENTRO EREVNAS KAI TECHNOLOGIKIS ANAPTYXIS

CERTH Greece

12 NETAS TELEKOMUNIKASYON ANONIM SIRKETI NET Turkey

13 INTERINNOV SAS IINV France

14 BITTUBES GMBH BIT Germany

15 NATIONAL CENTER FOR SCIENTIFIC RESEARCH NCSRD Greece


Page 4 / 29

Table of Contents

EXECUTIVE SUMMARY ...................................................................................................... 8

1. SCOPE ....................................................................................................................... 9

1.1. STORYLINE .............................................................................................................................................. 9

1.2. 5G-MEDIA REQUIREMENTS .................................................................................................................... 10

1.3. KPIS .................................................................................................................................................... 11

2. TECHNICAL DETAILS ................................................................................................ 13

2.1. COMPONENTS & WORKFLOWS ................................................................................................................. 13

2.1.1. Service Components ...................................................................................................................... 13

2.1.2. Application Components ............................................................................................................... 14

2.1.3. Replay Workflow ........................................................................................................................... 15

2.1.4. On-demand Instantiation of VNFs Workflow ................................................................................ 16

2.2. FAAS .................................................................................................................................................... 17

2.3. SERVICE OPTIMIZATION ........................................................................................................................... 19

2.3.1. Cognitive Network Optimizer ........................................................................................................ 19

3. VALIDATION ............................................................................................................ 21

3.1. TESTBEDS .............................................................................................................................................. 21

3.1.1. SDN Spine - Leaf Network ............................................................................................................. 21

3.1.2. Core Network Gateway ................................................................................................................. 21

3.1.3. OTE labs infrastructure ................................................................................................................. 23

3.2. DEPLOYMENTS ....................................................................................................................................... 23

3.3. RESULTS ............................................................................................................................................... 24

4. CONCLUSION .......................................................................................................... 28

4.1. MAIN ACHIEVEMENTS ............................................................................................................................. 28

4.2. LESSONS LEARNED .................................................................................................................................. 28

4.3. SUMMARY............................................................................................................................................. 28

REFERENCES ................................................................................................................... 29


Page 5 / 29

List of Figures

Figure 1: Use Case 1 - Example of application components and VNFs structure ..................... 14

Figure 2: Step 1 - Game Highlight detection and initiation of replay flow .............................. 15

Figure 3: Step 2 - Instantiation and process of vReplay .......................................................... 16

Figure 4: Step 3 - Spectator notification about available replay ............................................. 16

Figure 5: A network service life-cycle paradigm that demonstrates the use of on-demand VNFs. ...................................................................................................................... 17

Figure 6: NCSRD spine-leaf network topology .......................................................................... 21

Figure 7: NCSRD Data Center .................................................................................................... 22

Figure 8 OTE labs infrastructure ............................................................................................... 23

Figure 9: Classic vs Serverless in terms of QoE and Cost .......................................................... 24

Figure 10: Distribution of Spectators for scenario 1 ................................................................. 24

Figure 11: Produced Qualities for scenario 1. Green squares indicate the produced qualities. Each row corresponds to the stream of each player. .......................................... 25










Page 6 / 29

List of Tables

Table 1: Use Case 1 - KPIs ......................................................................................................... 11

Table 2 NCSRD testbed infrastructure. ..................................................................................... 22


Page 7 / 29

Definitions and acronyms

CNO Cognitive Network Optimiser

FaaS Function as a Service

GPU Graphical Processing Unit

KPI Key Performance Indicator

NS Network Service

OSM Open Source MANO

QoE Quality of Experience

SDK Service Development Kit

OW OpenWhisk

SVP Service Virtualisation Platform

TI Tele-immersive

TVM Time-Varying Mesh

VIM Virtualised Infrastructure Manager

VM Virtual Machine

VNF Virtual Network Function

VNF-FG VNF-Forwarding Graph


Page 8 / 29

Executive Summary

The 5G-MEDIA project has created a platform for flexible and facile deployment, complete lifecycle management and infrastructure usage optimization of next-generation media services. Most of the details of this platform can be found in the other deliverables, with D2.4 Final Report on Architecture, Requirements and Specification, describing the overall architecture and its objectives.

More specifically, this deliverable is a follow up of D6.1 5G-MEDIA Use Case Scenarios and Testbed and reports the progress and maturation of the 5G-MEDIA immersive media pilot. The first section focuses on this use case’s scope by presenting the storyline around a next-generation volumetric media application and how its requirements mapped to the 5G-MEDIA platform, as well as the relevant KPIs. Overall, this use case’s developments during the lifetime of 5G-MEDIA have aligned with the FaaS (or otherwise, serverless) components of the 5G-MEDIA platform.

Following, in section 2, the technical details of the immersive media pilot service are presented that outline the transition to a pure serverless service that capitalizes on the 5G-MEDIA platform’s FaaS offering.

Finally, section 3 presents the testbeds used for deploying and piloting this use case, as well as the results of our experimentation for assessing the efficacy of serverless media service architectures in the context of finite and low user count sessions. We close this document by reiterating our main achievements and lessons learned during the development and deployment of the first 5G-MEDIA use case.


Page 9 / 29

1. Scope

Within 5G-MEDIA, the first use case (UC1) utilizes the 5G- MEDIA platform to deliver a next-generation volumetric media streaming service. The goal is to exploit the advances in the 5G network’s softwarisation and virtualization to deploy an efficient, effective and self-managing media streaming service for emerging types of media that will leverage emerging network service deployment, management, and operational paradigms.

1.1. Storyline

In summary, the UC1 context is a new generation gaming experience that immerses playing users (denoted as players for the rest of this document) within the game virtual environment through their digitised, virtual representations. This is achieved through real-time volumetric capturing that 3D reconstructs the appearance of the players and, generates a live 3D media stream for each player. In more detail, volumetric immersive media is a new form of media that represents the textured 3D appearance of humans, commonly captured by specialized RGB-D (colour and depth) sensors [Alexiadis17, Karakottas18].

This use case demonstrates a real-time interactive immersive media application in which the two (2) players are volumetrically reconstructed and interact with each other in a common virtual gaming environment through their digitized virtual representations (i.e. textured 3D shapes). In addition, this gaming application allows for the live spectating of each gaming session by remote third party users (denoted as spectators for the rest of this document). Given the nature of the media streams (i.e. volumetric/full 3D), spectating heterogeneity is large, as it can comprise both Virtual Reality (VR) and Augmented Reality (AR) spectators, through spectators that use mobile phones, head-mounted displays and headsets, and even traditional desktop PCs. This type of next-generation media applications come with new varying consumer requirements that in turn, necessitate new functionalities and capabilities when being deployed.

Purely from a media perspective, the volumetric appearance representation of live human performances comprises a diverse multimedia stream. On one end, the users’ shape and thus, 3D geometry needs to be streamed. This is usually represented in the form of a 3D triangle mesh, which as a result of modern real-time 3D capturing and reconstruction technologies is of time-varying nature. Further, the 3D media stream is accompanied by a multi-view video stream which is used to reconstruct the coloured appearance of the users through multi-view mesh texturing.

Apart from the general concept of volumetric-media streaming, UC1 also wants to offer game replay highlights to each game session’s spectators. The replay clips are composed of three separate synchronized streams. The first two streams are the volumetric 3D media streams of the players while the third stream contains game state metadata. Replay clips are offered to spectators whenever a game highlight occurs during the game session, like a player score or a player hit event. Replay clips are managed by the game service and are offered to spectators to watch them on demand and at will, as soon as they are processed and become available.


Page 10 / 29

Finally, it is important to clarify that while UC1 involves both players and spectators as end-users, in the scope of 5G-MEDIA, we assume a fixed quality of experience and service cost for the players, while we try to optimize the experience for the spectators under cost constraints, by making proper 5G-MEDIA platform design decisions and taking advantage of them.

1.2. 5G-MEDIA Requirements

The limited progress in efficient and effective inter-frame coding of the 3D time-varying meshes [Doumanoglou19] in addition to the increased payload of the colour multi-view stream used to texture the 3D mesh, realizes the first requirement of UC1, which is high bandwidth due to the large volume of data to be transmitted. Additionally, to support live broadcasting of the session to spectators, there is a need that the streaming pipeline exhibits low latency timings. Further, supporting heterogeneous spectator clients (i.e. clients of varying processing power, screen resolution, and network conditions) is an additional challenge, as the service needs to support a variety of heterogeneous spectating users simultaneously. The aforementioned requirements for high bandwidth, low latency and heterogeneity in network conditions, processing power and screen resolution at the consumer’s side, implies that there is no one-size-fits-all encoding profile for the immersive media streams. Thus, with current technology and in 5G-MEDIA, fulfilling these demands can only be achieved through an efficient implementation of adaptive streaming.

Based on statistics, in UC1, the interactive sessions between the participants are expected to be short in duration, in the order of minutes. Additionally, a small but respectable amount of spectators (in the order of 10 – 50) are expected to join each session to watch the live session. As described in the previous paragraph, to tackle the heterogeneity of the spectators we would need to have varying simultaneous transcoding functions that would transcode the immersive media streams to various qualities. Different encodings require different processing power and different network conditions for optimal end-to-end pipeline performance, in terms of visual quality and latency.

Given the fact that the sessions are expected to be short in duration, it is implied that the transcoding functions are of short duration. Thus, the first requirement that UC1 imposes to the 5G-MEDIA platform is to support Serverless, which is an ideal paradigm for minimizing costs. Additionally, Serverless allows efficient and cost-effective scaling, since supporting a new group of spectators of similar requirements is as simple as spawning a new transcoding function with different encoding parameters. Additionally, Serverless exhibits low latency function instantiation which makes it suitable for transcoding a live stream.

Apart from instantiating regular Serverless functions on session start, adapting the game service to the time-varying spectator joins and leaves, requires from the 5G-MEDIA platform to support on-demand instantiation of transcoding functions during the game session.

In addition, to better support live streaming, in UC1 we can utilize algorithms that use GPU acceleration to achieve real-time performance with low latency. Thus, different transcoder profiles have different hardware requirements. This implies that the Serverless infrastructure of 5G-MEDIA is required to be able to deploy Serverless functions on nodes with specific hardware (i.e. nodes with GPU acceleration).


Page 11 / 29

Further, synchronizing the immersive media streams with the game state metadata to support replays, requires an increased amount of memory which does not scale well when many gaming sessions are expected to run in parallel. Further, the number of highlights, in fixed duration game sessions, cannot be easily predicted in advance nor the frequency of the events. This means that the memory requirements and processing power to process the media replay clips are highly unpredictable and varying. Thus, the Serverless architecture with on-demand function instantiation during a game session (apart from session start), that we adopt in 5G-MEDIA, can be easily identified as a cost-effective and scalable solution to this specific problem as well.

1.3. KPIs

The key performance indicators identified in D6.1 5G-MEDIA Use Case Scenarios and Testbed as pertinent to this use case are mostly related to service deployment and the delivered QoE by the UC1 service. They are outlined in the following table along with their resulting status:

Table 1: Use Case 1 - KPIs

5G-MEDIA KPI Definition Specific Targets Measurement/Assessment

Service deployment time:

Programmable networks and multi-tenant capability in 5G will ensure speedy deployment of services.

<= 5min

Elapsed time between the instantiation REST call and the retrieval of the transcoder IPs as

measured by the game server.

Result: Lightweight virtualization technologies provided by the integration of FaaS / Serverless within the 5G-MEDIA platform have enabled a service deployment time of 30secs.

Guaranteed user data rate:

Guaranteed bit rate for the application to function correctly. It corresponds to the user experienced data rate as defined by ITU.

Download:

>= 25 Mbit/s

Upload:

>= 25 Mbit/s

As measured by each receiving client (downlink). In addition, the production data

rate (uplink) will also be measured, as well as the transcoding data rates.

Result: Bi-directional data rates as measured by the production and consumption clients were measured at approximately 24-26 Mbps in the various tests conducted using the NCSRD testbed.

Network management OPEX

Sum of all the Operating Expenditure for the 5G Network Management including cost for running the 5G network, form its configuration, maintenance, etc.

Reduction with respect to

current cloud infrastructure

solutions

Sum of all the Operating Expenditure for the 5G Network Management including cost for

running the 5G network, form its configuration, maintenance, etc.

Results: The integration of the CNO and the Serverless deployment (i.e. lightweight virtualization) functionalities offered by the 5G-MEDIA platform have allowed for improved cost optimization, whose efficacy is dependant on specific session characteristics (e.g. population size and networking conditions). Under specific conditions, the improvements have been proved to be significant.

Service Ability to increase or Automatic Through the integration of a Serverless VIM,


Page 12 / 29

5G-MEDIA KPI Definition Specific Targets Measurement/Assessment

elasticity decrease the system capacity (e.g. CPU, storage, RAM, network capacity, service coverage, etc.) on-demand and/or in an automated way.

Service Deployment Availability.

we gain the elasticity offered by the OpenWhisk platform. Detailed measurements

will be offered by the FaaS platform.

Results: The definition of a two-level CNO that involves two interacting agents (i.e. O-CNO and SS-CNO) has allowed for the on-demand resizing and scaling of the application in both directions (i.e. downgrading QoE as imposed by O-CNO, or leveraging Serverless scaling to improve QoE).

Service Monitoring

Ability to define service metrics to monitor performances

Full traceability of the

microservice components throughout

their lifecycle.

Through the integration of a Serverless VIM, we gain the monitoring offered by the

OpenWhisk platform. Detailed measurements will be offered by the FaaS platform.

Results: The 5G-MEDIA platform provides monitoring tools that enable the collection of monitoring data from running in VNFs. In the context of UC1, a Kubernetes publisher was used to collect monitoring data from running containers and publish them into the publish/subscribe broker. Also, application-specific monitoring data, such as monitoring metrics per spectator stream (e.g., quality, transcoder id) are being collected to enable application-centric optimizations.

End-to-end latency

Maximum tolerable elapsed time from the instant a data packet is generated at the source application to the instant it is received by the destination application. With a focus on infrastructure, this includes the time needed for uplink, any necessary routing in the infrastructure, and downlink.

<= 100ms

Will be measured in an end-to-end manner through a testing procedure where the

producer and consumer will be co-located on the same machine. However, this also includes

the processing latency for transcoding the outbound traffic.

Results: The availability of FaaS GPUs have enabled the minimization of latency, averaged as 45-85ms among all profiles, even those demanding in terms of computational power.


Page 13 / 29

2. Technical Details

Based on the requirements described in Subsection 1.2 and the 5G-MEDIA platform architecture described in D2.4 Final Report on Architecture, Requirements and Specification, in this section, we give the technical details of UC1 implementation.

As already described in Section 1, the UC1 service’s backbone is the real-time transcoding of the media streams. To realize this, UC1 combined the efficient transcoding of the immersive media stream with the emerging Serverless paradigm to run transcoding (i.e. stateless) functions on the 5G edge for maximum cost savings without compromising quality and scalability. Moreover, UC1 utilizes the 5G-MEDIA platform’s Service-Specific Cognitive Network Optimizer (SS-CNO) to offer network-centric adaptive streaming to spectators, which is a global optimization method in contrast to traditional and more common, client-based adaptive streaming.

Apart from the live broadcasting of immersive media, UC1 takes advantage of the 5G-MEDIA’s Serverless infrastructure to spawn media processing functions, on-demand and at the 5G edge, in order to synchronize the 3D media stream with the state of the virtual environment. This way it offers on-demand viewing of short replay clips, containing interactive action between the players, to the spectators.

In the following subsections, we describe the service components, application components, workflows, FaaS integration, and UC1’s SS-CNO.

2.1. Components & Workflows

2.1.1. Service Components

UC1 comprises a set of Virtual Network Functions (VNFs) that facilitate its operation. All VNFs in UC1 follow the Serverless paradigm, with some of them marked as “regular”, meaning that they are spawned as Serverless functions on session start, while others are marked as “on-demand”, implying that they can be instantiated at any point in time during the session.

A summary of the UC1 VNFs (which are described in more detail in D4.2 5G-MEDIA Catalogue Portal and Network Apps) are given below:

vTranscoder3D (CPU/GPU) (on-demand): This VNF live re-encodes incoming immersive media traffic. The profiles have been solidified to be five, two produced on CPU that encode textures as still jpeg images and three that can be produced only on a GPU-accelerated node that encodes textures as HEVC-H.265 video sequences. Note that geometry coding’s fidelity also varies with different profiles. The output of vTranscoder3D VNFs is published in specific topics on vBroker to be consumed by other components.

vBroker (regular): A message broker VNF which is used to facilitate inter-component communication in UC1.

vReplay (on-demand), the VNF responsible for synchronizing the 3D media streams with the Game State stream to produce a game replay clip. vReplay produces a


Page 14 / 29

message with the required replay synchronization parameters to be consumed by the game application for replay playback.

vBuffer (regular), the VNF for buffering the 3D media streams to facilitate the replay functionality.

Figure 1: Use Case 1 - Example of application components and VNFs structure

2.1.2. Application Components

As UC1 revolves around a tele-immersive (TI) game, the game application comprises of 3 types of components:

Game Server: The Game Server acts as the central managing node of the TI application responsible for instantiating and controlling the required features such as automating service deployment and replay functionality. Upon launching, the Game Server handles the service deployment by instantiating the network service in OSM and configures the VNF instances when the required parameters become available. It is the main module communicating with the OSM and passing generated parameters between the OSM, the instantiated VNFs, and Player Clients and Spectator Clients. Additionally, the Game Server is responsible for orchestrating the workflow for the Replay Functionality.

Player Clients: The player clients is the application component used by the players in order to join a game hosted by the game server. They are responsible for handling the players’ movements, produce 3D media streams of their appearance and visualize the game state in order for the players to be able to play.

Spectator Clients: The spectator clients are the application components used by spectators to spectate a game session. Spectator clients can support heterogeneous running conditions


Page 15 / 29

such as different types of devices (mobile, desktop, Virtual Reality), consuming transcoded 3D media streams of different qualities and consuming the media streams both live or on-demand, in case of a replay clip. Live spectating is implemented in the same manner implemented for the player clients, consuming the produced media streams via the vBroker in real-time. On-demand consumption is implemented through the replay functionality. When a replay is available (commanded and orchestrated by the Game Server) the spectator clients can reproduce on-demand a past game session sequence (a game state and 3D media stream sequence).

The application components and the VNF instances comprising UC1 are illustrated in Figure 1.

2.1.3. Replay Workflow

The replay functionality is a procedural asynchronous process orchestrated by the Game Server. The main modules participating in the replay process are the Game Server, vBroker, vBuffer, vReplay and Spectator Clients. The Game Server is capturing the game highlight and orchestrates the replay workflow. vBroker is the communication node between the modules participating in the replay process. vBuffer buffers and provides the players’ 3D media streams, while vReplay synchronizes the 3D media streams with the game state stream. Finally, Spectator Clients, reproduce and playback the acquired replay event.

Replay workflow is described in three sequential steps:

Step 1: The Game Server detects a game highlight in the running game session and initiates a new replay session. It proceeds to a) flushing the game state stream to a game state topic in the vBroker and b) commands vBuffer to flush the two 3D media streams of the players to two separate topics in the vBroker along with their timestamps to a timestamps topic. The four mentioned vBroker topics are unique per replay session and are generated by the Game Server. Upon vBuffer finishing its process, it notifies the Game Server in a dedicated topic in vBroker. Step 1 procedure is given in Figure 2.

Figure 2: Step 1 - Game Highlight detection and initiation of replay flow

Step 2: When the Game Server receives the notification message from vBuffer it proceeds in requesting the instantiation of a new vReplay VNF instance using the REST API provided by the vBootstraper VNF of the service. When vReplay is up, it consumes the game state


Page 16 / 29

and timestamps topics generated in the previous step by the Game Server and proceeds in syncing the 3D media streams and the game state stream, based on timestamps embedded in the streams. The output of vReplay is a message, in a dedicated vBroker topic, containing, the replay session parameters, namely, replay session ID, the game state stream start and stop offsets and 3D media streams start offsets. Essentially, these comprise the synchronization parameters required for replay playback. The Game Server receives the vReplay output message and proceeds with step 3. The Step 2 procedure is given in Figure 3.

Figure 3: Step 2 - Instantiation and process of vReplay

Step 3: When the Game Server receives the replay session parameters produced by vReplay, it notifies the Spectator Clients that a new replay session is available and provides them with the required session parameters, as generated by vReplay. In case the spectators choose to view the replay, their clients consume the required streams from vBroker and produce a faithful replay playback based on the received parameters. The Step 3 procedure is illustrated in Figure 4.

Figure 4: Step 3 - Spectator notification about available replay

2.1.4. On-demand Instantiation of VNFs Workflow

In this section, we give a detailed description of how the 5G-MEDIA platform’s API was used in UC1 to instantiate VNF, on-demand. For clarity and simplicity, we will follow the example visualized in Figure 5.


Page 17 / 29

To begin with, the session begins with a REST command issued by the game server application that instantiates the network service (t1). As can be seen from the chart, only the bootstrap and the regular-type VNFs are being instantiated.

From this moment and until the end of the session, all parties interested in the exposed information regarding this running instance of the service, such as the IPs and connection ports of the various pods hosting the VNFs or the IngressURL discussed in 0, can poll iteratively at any time to acquire that information. Supposedly, the service-specific SS-CNO and the game server application have been polling and acquired the IngressURL, to be able to issue on-demand VNF spawning and termination commands.

In a later-moment, the SS-CNO issues a spawning REST command for a CPU transcoder, preconfigured to produce the profile with ID 1. Note that, each spawning and termination command is identified by a per-session-unique event_uuid, 111 in this case.

Afterward, the game server application detects a replay-worthy event, so in response, it issues a spawning REST command for the vReplay function to process the demand.

Some moments later, an abrupt change in networking conditions drives the CNO to decide the termination of the CPU transcoder spawned earlier with the event_uuid 111, and spawn two new instances of the GPU transcoder, producing the more bandwidth-efficient profiles with IDs 4 and 5 respectively.

More spawning and termination commands can be executed alike, during the service’s lifetime.

Figure 5: A network service life-cycle paradigm that demonstrates the use of on-demand VNFs.

2.2. FaaS

The highlight of the network virtualization mechanism developed as part of this project is the automatic deployment of “micro-services” supporting FaaS functionality. Their purpose


Page 18 / 29

is to support scalability and elasticity in finer granularities, allowing the service to adapt to changes in demand and service needs in a more responsive and Serverless manner while minimizing the running costs. For UC1, this translates to a) instantiating one 3D-transcoding service for each separate session that is being initiated by the end-users and b) instantiating replay related services on-demand when such need occurs.

In all previous iterations of the use-case scenario, components of the network service were instantiated and de-instantiated together, as this self-contained type of session was deemed to be adequately effective and appropriate for the intended use.

The primary target of this latest iteration of the network service is to utilize the VNFs defined in section 7.2.1 of D4.2 5G-MEDIA Catalogue Portal and Network Apps in a way better aligned with the pure-Serverless model. A pivotal element of such an approach is the compliance with the Single Responsibility Principle1, implying that every module or function should have responsibility for a single part of the functionality provided and that the services should be narrowly aligned with that responsibility.

In contrast to the previous NS incarnation where a limited number of VNF instances bore extensive functionality, such as to produce multiple quality-profile streams concurrently - in the case of vTranscoder3D - new design’s VNFs are only responsible for delivering one profile per instance. As soon as a profile is needed to be produced, a limited-scope VNF instance should be able to be spawned on-demand, to deliver just that. When its respective outcome is no longer needed, e.g. per CNO decision, the infrastructure should be able to terminate the unused instances on the fly, optimizing resource utilization.

To facilitate this functionality, in addition to the regular-type VNFs we introduce two new types; the “on-demand” (or “event-based”) and the “bootstrap”.

The on-demand / event-based VNF will not start with the main service instantiation but will be available to be instantiated later on-demand. Each such instance can have its unique day-0 parameters, passed to it upon instantiation. In the context of this newly designed setup, the spawning and termination actions correspond directly to the previous setup’s day-1…N reconfiguration, as they serve the same purpose; add or remove functionality to/from the service on the fly.

An example of a spawning command:

curl -d '{"osm_ip": "10.100.176.66", "event_uuid": "<spawn_event_uuid>", "osm_ns": "sky_balls", "operation":

"spawn_transcoder", "player_index": "<1 or 2>", "vnfd_name": "<transcoder_2_9_0_gpu_vnfd or transcoder_2_9_0_cpu_vnfd>",

"vnfd_index": "<2 or 3>", "gpu_node": "<1 or 0>", "produce_profile": "<profile_id>", "metrics_broker_ip": "<ip>", "metrics_broker_port":

"<port>"}' -H "Content-Type: application/json" -X POST <IngressUrl>/handlerequest

And an example of the respective termination command:

1 https://en.wikipedia.org/wiki/Single_responsibility_principle as seen on 25/2/2020

https://en.wikipedia.org/wiki/Single_responsibility_principle


Page 19 / 29

curl -d '{"osm_ip": "10.100.176.66", "event_uuid": "<termination_event_uuid>", "osm_ns": "sky_balls", "operation":

"terminate_transcoder", "uuid": "<spawn_event_uuid>"}' -H "Content-Type: application/json" -X POST <IngressUrl>/handlerequest

The other aforementioned new type of VNF is the bootstrap VNF. It is a necessary piece of the on-demand mechanism that holds the needed logic for adding/removing event based VNFs and connecting/deleting them to/from the existing network service. On NS instantiation, it deploys assets into the general k8s eventing system and holds the Ingress-URL of this NS instance. Upon deletion, it will make sure to remove all resources created.

Note that this newly developed functionality is incremental to the one previously in place, as already developed mechanisms for reconfiguration and migration work properly for the regular (non-on-demand) kind of VNFs.

2.3. Service Optimization

2.3.1. Cognitive Network Optimizer

For this second period of the project, following the 5G-MEDIA platform developments, two variants of UC1 service optimization have been developed. The first focused on optimizing the original version of the UC1 service that comprised 2 multi-profile producing transcoding functions and focused on fine-tuning their resource consumption in terms of profile selection. This employed the online function reconfiguration mechanism and its results were published in [Athanasoulis20] with more details available in section 3.4 of D3.4 5G-MEDIA Operations and Configuration Platform.

However, current billing practices would not allow for finer-grained service cost optimization using this multi-responsibility based Serverless service design. Further, the design itself detracts away from the Serverless service development best practices that rely on modular functions under the single responsibility principle. Consequently, UC1 transitioned to a true Serverless design where each transcoding function is responsible for the production of a single profile. This opened up new opportunities for service optimization, and more specifically a cost-oriented approach. Also, the updated 5G- MEDIA platform architecture separated the resource allocation orchestration and service-specific optimization by introducing an overarching CNO (O-CNO, or otherwise denoted as a high-level and infrastructure-wide CNO) and various service-specific CNOs (SS-CNO, otherwise denoted as low-level CNOs).

Thus, an updated variant of UC1 optimization was developed as an SS-CNO and was designed to fulfil a three-fold role. First, to collect application-specific metrics from spectator clients. Second, to process those metrics globally for each standalone session, and, in cooperation with O-CNO, spawn new transcoder function instances and terminate existing ones, based on a pre-specified Quality-of-Experience (QoE) – cost trade-off policy. Third, the UC1 SS-CNO will assign transcoding profiles to spectators, realizing real-time network-centric adaptive streaming (see D2.4 Final Report on Architecture, Requirements and Specification).


Page 20 / 29

In summary, the objective of the UC1 service-specific CNO is to optimize the cost efficiency of each session to eventually maximize profits, while also retaining the serviced QoE levels. To this end, we need to consider revenues and costs. Costs are comprised of fixed production costs and variable costs. Variable costs in our case involve the deployment cost of the transcoders. These vary, with the most important discrepancy being the cost of the HW that the transcoders get deployed at, and more specifically the deployment at either a CPU or a GPU node as it is a scaling factor on top of the finer-grained billing of Serverless functions (typically billed on milliseconds of use). The revenues can be considered an indirect function of the user QoE through the reasonable assumption that (more) satisfied users will in-turn provide (more) revenue to the service provider.

For the UC1 SS-CNO to be able to balance costs and QoE, it has to know certain system parameters such as the number of available GPU nodes, the cost per node type, in addition to media session-specific parameters like the production framerate, and service-specific parameters like the function that connects revenue and QoE and the parameters that describe each of the delivered profiles/qualities that can be produced by the transcoders. These include the bitrate, processing time (average encoding time per frame), visual quality (i.e. average texture PSNR) and qualitative/categorical characteristics like inter- or intra-frame coding.

Given all the above data the UC1 SS-CNO will be able to select which profiles will be transcoded on a per-session basis. In this way, the SS-CNO will mandate to each spectator which profile they will be consuming while also taking into account the cost optimization of the service itself. Thus, the SS-CNO will initially consider the resulting QoE each spectator would experience if he was to transition to any of the possible profiles/qualities. For the profile to QoE mapping, we leverage recent work on the literature that describes the QoE as a function of the received PSNR and framerate for gaming scenarios [Zadtootaghaj18].

Subsequently, the SS-CNO solves an integer programming problem in order to select which qualities will be produced by also taking into account that session’s service costs. Under the Serverless paradigm, each produced quality requires a dedicated transcoder. Each quality can be considered on or off and this way allows us to model hardware resource constraints as well (i.e. GPU availability constraints). The incoming stream profile (as produced by the volumetric capture stations) is always available since it does not require any transcoding and thus no cost is associated with it. Then each combination is scored in terms of the average QoE it would offer to the spectators and its transcoding cost. The combination that best balances these two factors are chosen by the CNO.

Following that, the UC1 SS-CNO communicates with the overarching CNO to request resources and then acts to either spawn new transcoding functions, destroy existing ones or re-direct spectators to consume other profiles given its selected combination. In this way, the service seeks to achieve minimal costs while preserving certain QoE levels, aiming for maximal QoE to all spectators. This is controlled by a (service-specific) ratio factor between QoE and costs under a linear modelling assumption. The aforementioned profile selection process is repeated at regular intervals to take into networking conditions variations, changes in resource availability and the varying nature of the spectator population.


Page 21 / 29

3. Validation

3.1. Testbeds

For this pilot, the testbed in ENG has been used to host the SVP components (public Catalogue, AAA, MAPE, Orchestration) as described in D2.4 Final Report on Architecture, Requirements and Specification. The service itself was deployed at the combined NCSRD/OTE testbeds, with their detailed description following.

3.1.1. SDN Spine - Leaf Network

The WAN backbone network on the NCSRD site is composed of several physical SDN Switches forming a spine-leaf architecture. All the switches are OpenFlow enabled and support OpenFlow protocol version 1.3. They are controlled by a centralized OpenDayLight (ODL) SDN controller, which is responsible for installing forwarding rules (flows) on each switch.

Figure 6 presents the NCSRD spine-leaf network topology of Site 1 of the Athens platform. Every lower-tier switch (leaf layer) is connected to each of the top-tier switches (spine layer) in a full-mesh topology. The leaf layer consists of access switches that connect to any physical or virtual device located on the NCSRD site, while the spine layer is the backbone of the network and is responsible for interconnecting all leaf switches and establish connectivity with the Internet and the other sites of the 5G-MEDIA platform. SDN backbone network can offer isolation and QoS policies for each network slice instantiated on the platform.

Figure 6: NCSRD spine-leaf network topology

3.1.2. Core Network Gateway

An Integrated Services Router (ISR) by Cisco, alongside a Firewall (i.e. Cisco ASA 5510), are used for the realization of the core network gateway on the NCSRD site. Through these nodes, the NCSRD core network is connected to the Internet, via the access provided by Greek Academic network provider (GRNET). Moreover, it is also used as the endpoint for the interconnection between NCSRD and OTE sites using the QinQ Ethernet transport (see section 3.1.3). Finally, a VPN concentrator server allows remote users to connect to the


Page 22 / 29

NCSRD testbed via VPN offering all the standard tunnel types (i.e. OpenVPN, IPSec, Anyconnect).

The core Data-Center Domain is physically located at the NCSRD campus, at the city of Athens. The DC domain is implementing two different services. Firstly, it offers resources for the deployment of 5G-MEDIA virtualized components (OSM, OpenWhisk). Secondly, it offers computing resources to be used by the NFV Orchestrator for the deployment of Network Services and VNFs (i.e. NFVI-PoP). Furthermore, the Core DC domain also supports the Kubernetes cluster either for Cloud-Native NFV service deployment (i.e. container-based VNFs) or for other types of applications related to 5GMEDIA.

The overview of the testbed infrastructure is presented in detail in the table below:

Table 2 NCSRD testbed infrastructure.

CPU RAM GPU

HP ProLiant DL380 Gen9 Intel(R) Xeon(R) CPU E5-2637 v3 @

3.50GHz 100GB Generic

HP Z230 Tower Workstation

Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz

50GB Quadro M4000

Alienware Aurora R7 Intel(R) Core(TM) i7-8700K CPU @

3.70GHz 32GB

GeForce GTX 1080 Ti

Alienware Aurora R7

Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz

32GB GeForce

GTX 1080 Ti

Figure 7: NCSRD Data Center

The first NFV Infrastructure uses OpenStack “Queens” version as a cloud Operating System and NFVI enabler. Currently, it is deployed over two physical HP servers. The dedicated management network for this NFVI is 10.100.176.0/24. Components from the MANO layers are deployed as VMs on this cloud.

The second 5G-MEDIA Infrastructure is a GPU - Kubernetes Cluster, enabling the container-based deployment of VNFs. The management and external network of the cluster is 10.30.2.0/24.


Page 23 / 29

3.1.3. OTE labs infrastructure

Additional resources are offered by the OTE testbed which is connected with the NCSRD testbed with a 1Gbps/10 Gbps line as depicted in Figure 8.

Figure 8 OTE labs infrastructure

The overview of the components of the testbed is shown in the table below:

System CPU RAM GPU

Power edge R640 server Intel(R) Xeon Gold 6138 2.0G,

20C/40T 384GB

Quadro M4000

Power edge R640 server Intel(R) Xeon Gold 6138 2.0G,

20C/40T 384GB

Quadro M4000

Alienware Aurora R7 Intel(R) Core(TM) i7-8700K CPU @

3.70GHz 32GB

GeForce GTX 1080 Ti

Alienware Aurora R8


32GB GeForce

GTX 2080 Ti

Alienware Aurora R8


32GB GeForce

GTX 2080 Ti

The NFV infrastructure is using the Open Stack ‘’Queens’’ version as a cloud operating system that is deployed on top of two Dell servers. The set of the additional GPU resources are also available to enable UC1 service scaling. The management of the testbed is done through the IP 195.167.80.32/27.

3.2. Deployments

Details about the deployment at the NCSRD testbed can be found in D6.1 5G-MEDIA Use Case Scenarios and Testbed, as no updates were made during the period elapsed and the additional resources are seamlessly available to the service providers through the 5G-MEDIA platform’s abstractions.


Page 24 / 29

3.3. Results

To illustrate the advantages of the Serverless approach in terms of cost optimization we run simulated experiments with simulated transcoders and spectators which report semi-randomized metrics. In the figures below we demonstrate the performance of the UC1 SS-CNO against a traditional VM-based approach in terms of QoE and cost for various scenarios where the number, the available bandwidth and the processing capabilities of the spectators vary. Neglecting deployment time and billing method differences, the VM-based approach would naively transcode all profiles all the time, achieving maximal QoE for each session, but at the cost of increased expenses. Overall, under all scenarios, we observe that the Serverless approach can approximate the theoretical maximal QoE while typically greatly outperforming the VM-based approach in terms of service costs (Figure 9).

Figure 9: Classic vs Serverless in terms of QoE and Cost

Scenario 1: Low number of spectators with their distribution presented in Figure 10. The UC1 SS-CNO selects a single profile, i.e. a single transcoding function (Figure 11).

Figure 10: Distribution of Spectators for scenario 1


Page 25 / 29

Figure 11: Produced Qualities for scenario 1. Green squares indicate the produced qualities. Each row corresponds to the stream of each player.

Scenario 2: Additional spectators enter the session (Figure 12). An additional quality is now produced (Figure 13) which is a low video quality that is deployed on high-cost resources (i.e. GPUs) increasing the service’s costs in order to retain the QoE level.



Scenario 3: The spectator population further increases with higher bandwidth and high-end device spectators (Figure 14), which are capable to play high-quality video. As a result the UC1 SS-CNO now deploys another high-cost function (Figure 15).


Page 26 / 29



Scenario 4: Same scenario as #3 (Figure 16) but with a constraint on the GPU resources which forces the SS-CNO to drop the higher video quality ()Figure 17, thus minimizing costs against QoE.



Page 27 / 29


Scenario 5: Again, same as scenario #3 (Figure 18) but with further GPU restrictions imposed, resulting in only a single stream switched to video quality (Figure 19).




Page 28 / 29

4. Conclusion

4.1. Main Achievements

Within 5G-MEDIA, apart from the media format novelty, this use case has designed and developed the first – to our knowledge – Serverless real-time adaptive streaming service, where each transcoded profile is serviced by a single stateless serverless function. To achieve this, OpenWhisk has been extended with real-time streamed network traffic enabled functions, a functionality not seen in other Serverless frameworks. In addition, given the stringent latency requirements that manifest in computational capacity, OpenWhisk has also been extended with serverless function GPU deployment capabilities, another functionality not seen in other Serverless frameworks. In addition, the UC1 service is a hybrid service that comprises rigid and dynamic components, with rigid components to be instantiated upon session start and lasting until the end of the session, while dynamic functions are deployed on-demand in unpredictable intervals and last a short interval, less than the session’s duration.

Apart from the implicit scalability and flexibility, a Serverless design allows for finer-grained service optimization that can take the service costs into account. To that end, the UC1 service optimization has focused on the costs aspect and delivered a Service-Specific CNO that can greatly optimize the service’s costs compared to traditional VM-based approaches, while also preserving the delivered QoE.

4.2. Lessons Learned

Even though session-based optimization proved to be highly suitable for small (i.e. finite and low user count) sessions, many parallel sessions may compete over the same resources, necessitating separation of session-specific and service-specific optimization.

Additionally, rarely will services, especially media streaming ones, be comprised solely of Serverless functions. However, specific cases like the one demonstrated in this UC may greatly benefit from a Serverless transformation, especially on their resource expensive parts and components.

4.3. Summary

In this deliverable, we described specific details regarding the scenario, requirements to the 5G-MEDIA platform and implementation details of the 5G-MEDIA project’s first use case. Apart from the technicalities of the use case, we also described the novelties that this use case rose to the OpenWhisk Serverless framework used in the project, while we provided a concrete proof of concept implementation of how an emerging media application can leverage virtualization and softwarisation capabilities of the new 5G networks to optimize quality of experience for its consumers while minimizing costs through the Serverless paradigm. We hope that the 5G-MEDIA’s UC1 can serve as a reference baseline for future, similar applications that want to take advantage of emerging technologies in 5G networks.


Page 29 / 29

References

[Zadtootaghaj18] Zadtootaghaj, S., Schmidt, S. and Möller, S., 2018, May. “Modeling gaming QoE: Towards the impact of frame rate and bit rate on cloud gaming.” In 2018 Tenth International Conference on Quality of Multimedia Experience (QoMEX) (pp. 1-6). IEEE.

[Athanasoulis20] Athanasoulis, Panagiotis and Christakis, Emmanouil and Konstantoudakis, Konstantinos and Drakoulis, Petros and Rizou, Stamatia and Weit, Avi and Doumanoglou, Alexandros and Zioulis, Nikolaos and Zarpalas, Dimitrios, “Optimizing QoE and Cost in a 3D Immersive Media Platform: A Reinforcement Learning Approach”, MMEDIA 2020: The Twelfth International Conference on Advances in Multimedia (MMEDIA).

[Alexiadis17] Alexiadis, D. S., Chatzitofis, A., Zioulis, N., Zoidi, O., Louizis, G., Zarpalas, D., & Daras, P. (2017). “An Integrated Platform for Live 3D Human Reconstruction and Motion Capturing.” IEEE Transactions on Circuits and Systems for Video Technology.

[Karakottas18] Karakottas, A., Papachristou, A., Doumanoglou, A., Zioulis, N., Zarpalas, D., & Daras, P. (2018). “Augmented VR.” IEEE VR.

[Doumanoglou19] Doumanoglou, A., Drakoulis, P., Zioulis, N., Zarpalas, D., Daras, P., "Benchmarking Open-Source Static 3D Mesh Codecs for Immersive Media Interactive Live Streaming", IEEE Journal on Emerging and Selected Topics in Circuits and Systems, Feb 2019.

Documents

Programmable edge-to-cloud virtualization fabric for the 5G ......2020/04/05 · This is achieved through real-time volumetric capturing that 3D reconstructs the appearance of the