Beyond the MCU

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED.

D3-4 Media Services -Beyond the MCU

Chad Hart

WebRTC Expo V

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 2Image source: http://thewrestlinglifeupdate.tumblr.com/

Multi-PARTY! Video Conferencing

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 3

Easy & cheap approach to multi-party: Mesh

3

Full Mesh

Works for a few parties

No server cost

Lowest latency

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 4

Mesh does not scale for many video conferees

4

Full Mesh

Clients get overloadedEncode costs more than decode

Limited uplink bandwidth

Inconsistent performance across participants

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 5

Traditional Telephony Approach: MCU

5

Multipoint Control Unit (MCU)Good at manipulating all media

Transcoding

Transizing

Transrating

Interworking

Server-side CPU intensive

Client can request different conference mixes

Usually client friendly

Downsizing & mixing can reduce bandwidth

Fewer client streams to process

MCU

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 6

Traditional MCU resource usage

6

Worst-case process:1. Each stream decoded

at the MCU

2. MCU mixes each conferee

3. MCU encodes individual mix for each conferee

MCU

1

1 22

3

3

44

5

5

6

6

Ingress streams 6

Egress streams 6

MCU decodes 6

MCU encodes 6

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 7

Efficient MCU resource usage

7

Efficient process:1. Each stream decoded

at the MCU

2. MCU mixes each conferee

3. MCU shares same encoder for all conferees

MCU

12

3

45

6

Ingress streams 6

Egress streams 6

MCU decodes 6

MCU encodes 1

1

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 8

Resource efficient MCU with encoder sharing

30%-50% reduction in compute resources

0%

20%

40%

60%

80%

100%

1 2 3 4 5 6 7 8 9 10 11 15 17 20

Syst

em

CP

U U

tiliz

atio

n

Conferees

Encoder sharing vs. multi-encode

Traditional MCU Dialogic Evolved MCU

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 9

Newer approach: SFU

9

SFU

Selective Forwarding Unit (SFU) routing

Clients send one & receive many

Client can instruct SFU which streams to send

High throughput

Can be lots of downlink bandwidth

Low latency

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 10

Newer approach: SFU

10

SFU

Selective Forwarding Unit (SFU) routing

Clients send one & receive many

Client can instruct SFU which streams to send

High throughput

Can be lots of downlink bandwidth

Low latency

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 11

New WebRTC approach: Simulcast

11

SFU

High bitrate

Low bitrate

Selective Forwarding Unit (SFU) with Simulcast

Clients send multiple streams to SFU

one high-bit rate

one or more lower-bit

Client directs SFU which streams to receive

Reduces bandwidth vs. SFU

Simulcast in WebRTC coming

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 12

A look at Google Hangouts with Simulcast

http://webrtchacks.com/hangout-analysis-philipp-hancke/

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 13

Future Approach with VP9 – SVC?

13

SFU

MediumHigh LowLayered bitrates:

Selective Forwarding Unit (SFU) with SVC

Clients send layered stream to SFU

Layers have varying bitrates –time, size, quality

SFU directs who gets what

Coming to WebRTC eventually

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 14

Multi-party video summary comparison

http://networkfuel.dialogic.com/webrtc-whitepaper

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 15

Building a compelling UX with a MCU

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 16

Video UX: Traditional method

MCU1 2

3 4

Mixed stream 1 2

3 4

3G Device

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 17

Video UX: Traditional method

MCU1 2

3 4

Mixed stream 1 23

3G DeviceActive Talker

Media Server Chooses Layout

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 18

Client appApp Server2

Modern UX with MCU mixing, HTML5 & JavaScript

Browser

MCU1 2

3 4

HTML5 CanvasJavascript

Mixed stream

Application-defined UX

Layout

"Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum."

Benefits:

Application defined UX

User specific & customizable layout

Lower client BW & CPU vs. mesh

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 19

What do you care more about?

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 20

CPU & processing power is expensive

Client view Provider View

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 21

Bandwidth is not always ubiquitous or free

Client-side Server-side

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 22

Why do you need a media server?

Multi-party conferencing InterworkingTranscoding

Stream processing Person-to-machineRecording

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 23

Case Study: Video Recording Service

Application highlightsRecord live events – E.g. Kids soccer gameShare event recording with family/friends – Multi-party stream

Supported Recording Formats

.3GP

.mp4

.wav

CodecsG.711, Opus, AMR-NB and AMR-WB (voice)VP8, H.264 (video)

Message StorageLocal storageRemote message deposit via MSRP

HTTP/MSRP

REST (HTTP)

Application Server

Media Server

Message Storage

RTP (media)

Solution HighlightsSoftware-based Media Server

Virtualized (VM) and COTS deployment

High Availability and Geographical Redundancy

Scalability via Media Resource Broker

Architected to support 3.5M Subscriber Network

Recording

Remote Message Deposit

Key Functions

Transcoding

WebRTC Signaling

WAN

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 24

The future: gesture controls

https://github.com/mtschirs/js-objectdetect/

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 25

The future: machine vision

COMPANY CONFIDENTIAL © COPYRIGHT 2014 DIALOGIC INC. ALL RIGHTS RESERVED. 26

The future: augmented reality

http://jooink.blogspot.com/2012/10/gwt-augmented-reality-howto-step-1.html