Introduce native html5 streaming player

HOW DID I CREATE A HTML5 VIDEO STREAMING PLAYER

and what I’ve learned

KKBOXVideo陳建辰 jessechen

AGENDA• Some beforehand knowhow

• Why developing this player

• HTML5 extensions

• Introduce our player

• Future of this player

SOME BEFOREHAND KNOWHOW

analog-to-digital adapter

digital raw data

PROCESS OF PLAYING VIDEO1. Make it digital

analog signal

H.264 / H.265Encode

PROCESS OF PLAYING VIDEO2. Encode video file

Raw data .mp4 / .ts file

Easier for storage and transference

PROCESS OF PLAYING VIDEO3. Play it !

.mp4 / .ts filea/v sync

codec decode and

display

extract contentfrom container

WAIT, WHERE IS PLAYER?

WHAT PLAYER DOES

.mp4 / .ts filea/v sync

codec decode and

display

HERE

extract contentfrom container

THEY ARE ALREADY HANDLED

.mp4 / .ts file

HTML5

a/v sync

codec decode and

display

extract contentfrom container

SO WHY DEVELOPING THIS PLAYER?

OUR PLAYER FOCUS ON (1)

extract content and a/v info

from container

.mp4 / .ts file

get filesin smart way

a/v sync

codec decode and

display

HOW PLAYER GET FILE ?

server holds .mp4

content

VINTAGE WAY

Player get it and play

single file

slow and inflexible

server holds .mp4

content

EFFICIENT WAY

Player get part of video and

start play

progressive download

Keep requesting rest part of video during

playback

still inflexible

prepare different qualities

of contenton server

MODERN WAYadaptive streaming

Player playsbest quality depends on

network status

Keep requesting rest part of video during

playback

HOW ABOUT WE SPLIT FILE• split file into segments, described by manifest

• manifest tells

- available bitrates

- how file is fragmented

- other information, e.g encryption

+

+

knows how toget fragments

ADAPTIVE STREAMING IN SHORT

fragments

manifestfile

ADAPTIVE STREAMING TYPES

SSSmooth Streaming

HLSHTTP Live Streaming

HDS

INTRODUCE DASHDynamic Adaptive Streaming through HTTP

PROS OF DASH

• open source, with industry standard, which means more universal usage

• support 2 types of containers, the MPEG-4 file format or the MPEG-2 Transport Stream

GET FILES IN SMART WAY

• manifest parser - able to read different type of manifest

• adaptive bitrate - able to decide which quality to load

OUR PLAYER FOCUS ON (2)

.mp4 / .ts file

get filesin smart way

protectionlogic

extract contentfrom container

a/v sync

codec decode and

display

DRM ?(DIGITAL RIGHTS MANAGEMENT)

Content provider / DRM provider likes it

For end user

CLIENT SIDE PROTECTION• Client get a license instead of a key from DRM

service

• A blackbox or sandbox get the key by processing license

• That blackbox / sandbox decrypt content and directly output it to display

H.264 / H.265Encode

HOW TO PROTECT CONTENT

Raw data .mp4 / .ts file

DRM implemented

here

PROTECTION LOGIC

• give what DRM server needs and retrieve license

• negotiate with browser in order to implement protection on client side

• deal with different browsers

knows how to

NOW, LET’S START TALK ABOUT BROWSER

<video>, MSE and EME

HTML5 VIDEO ELEMENT• <video></video> support progressive download

• For adaptive streaming, MSE is required

extract content from container

WHAT ROLE MSE PLAYS

.mp4 / .ts file

MSE handles

a/v sync

codec decode and

display

MSE handles

INTRODUCE MSE

• Given different bitrate segments and it helpshtml5 video element to play smoothly

• SourceBuffer provide video and audio buffer condition

Media Source Extension

EME IS ABOUT PROTECTION

HOW BROWSER ADOPT DRM

• For browser, that blackbox called CDM (Content Decrypt Module)

• Each browser support different DRM

context - “a blackbox or sandbox get the key by processing license”

DRM ON BROWSER

Widevine FairplayPlayready Primetime

INTRODUCE EME

• Even though browser support its own DRM, W3C defines a EME spec, in order to expose same api for client

• prefixed api was implemented on earlier version of chrome(smart tv)

Encrypted Media Extension

EMECDM

provide context from encrypted content

get ‘challenge’

DRM license server

request with challenge

get license

provide license for CDM to decrypt content

player

EMECDM

PROTECTION LOGIC FLOW

WHAT ROLE EME PLAYS

extract contentfrom container

.mp4 / .ts file

a/v synccodec decode

anddisplay

decrypt content in blackbox

INTRODUCE OUR PLAYER

OUR GOAL

• play not single file but sequence of segments, with different bitrate, a.k.a adaptive streaming

• play protected stuffs, for content providers’ goods

a player able to

影片ya pi

yapi.js

DEVELOP PROCESS

make it work stable refactor

MAKE IT WORK FIRST

only 2 files in the very beginning

BE STABLE

• well structured

• modularized

• dependency management (dijon.js)

• consistent code style

MODULARIZED

api

adaptive

extension

fragment

manifest

protection

stats

stream

utils

UTILS• Capabilities

• Debug

• ErrorHandler

• EventBus

• IntervalBus

• UrlModel

api

adaptive

extension

fragment

manifest

protection

stats

stream

utils

EXTENSION

• MediaSourceExtension

• SourceBufferExtension

• VideoModel

api

adaptive

extension

fragment

manifest

protection

stats

stream

utils

MANIFEST

• ManifestLoader

• ManifestExtension

• ManifestModel

api

adaptive

extension

fragment

manifest

protection

stats

stream

utils

STREAM

• StreamCtrl

• BufferCtrlapi

adaptive

extension

fragment

manifest

protection

stats

stream

utils

FRAGMENT

• FragmentCtrl

• FragmentLoader

• SegmentTemplate

api

adaptive

extension

fragment

manifest

protection

stats

stream

utils

ADAPTIVE

• AbrCtrl

• BandwidthRecorderapi

adaptive

extension

fragment

manifest

protection

stats

stream

utils

PROTECTION

• ProtectionCtrl

• ProtectionModel

• ProtectionRequest

• Playready / Widevine

api

adaptive

extension

fragment

manifest

protection

stats

stream

utils

STATS

• MetricsCtrl

• Metrics

• StatsCtrl

• BitrateHistory

api

adaptive

extension

fragment

manifest

protection

stats

stream

utils

REFACTOR

• review flow

• redefine api

APPLICATION FLOW

REDEFINE API

• use jsdoc to generate spec document

• define api, event and vo(value object)

• spec

• result

DEMOSample player

HOW TO MANAGE THOSE MODULES

DEPENDENCY INJECTION

• not necessary in the beginning, but became very important after it went much more complex

• use dijon.js as di framework

INTRODUCE DIJON.JS

• a system holds all dependencies, after mapping modules to it

• an object get dependencies after injection

DEFINE DEPENDENCIES

// A depfunction A() {}

// B requires Afunction B() { this.a = undefined; // how dijon knows where to inject}

// instantiate dijonvar system = new dijon.System();

3 KINDS OF MAPPING

var a = new A();system.mapValue(‘aValue’, a);// system.getObject(‘a’) would return an ‘a’ instance

system.mapSingleton(aSingleton, A);// system.getObject(‘aSingleton’) would return a singleton a

system.mapClass(bClass, B);// system.getObject(‘bClass’) would return a new inited B

OUTLET MAPPING

// map outlet to make dependency work// @param sourceKey {String} - the key mapped to system would be injected// @param targetKey {String|'global'} - the key outlet is assigned to// @param outletName {String|sourceKey} - the name of property used as an outletsystem.mapOutlet('aSingleton', 'bClass', ‘a');

var b = system.getObject(‘bClass’);// b.a is aSingleton

DIRECTLY INJECTION

var b = new B();system.injectInto(b);

// b WOULDN’T have dependency singleton A// b/c we only mapOutlet to key bClass of system

// map outlet to globalsystem.mapOutlet(‘aSingleton’, ‘global’, ‘a’); system.injectInto(b); // this would work

AUTO MAP OUTLET

function B(){this.aSingleton = undefined;this.c = undefined;

}function C(){}

// auto mapped outlet would assign to ‘global’// and outlet name is the same as mapped key (before mapping)system.autoMapOutlets = true;

system.mapClass(‘bClass’, B); // map again b/c B is changedsystem.mapClass(‘c’, C);// system.getObject(‘b’) would have c dep

COMMUNICATE IN BETWEEN

Now B has dependencies of A and C

How would you do in this scenario:

A is doing something when it’s done, invoke a method of B

–this is a very common situation in player

MAP HANDLER AND NOTIFYfunction A() {

this.say = function() {system.notify(‘aDoneSaying’);

};}

function B() {this.a = undefined;this.afterAsays = function(){

// do something};

}

system.mapSingleton(‘a’, A); system.mapSingleton(‘b’, B);

system.mapHandler(‘aDoneSaying’, ’b’, ‘afterAsays’);// system.getObject(‘b’).a.say() would invoke b.afterAsays

NOTIFYING CLASSsystem.mapSingleton(‘a’, A); system.mapClass(‘b’, B); // map class here

var b1 = system.getObject(‘b’);// b1.say() would invoke a newly instantiated b.afterAsays// instead of b1.afterAsays

system.mapValue(‘b1’, b1);system.unmapHandler(‘aDoneSaying’, ‘b’, ‘afterAsays’) system.mapHandler(‘aDoneSaying’, ‘b1’, ‘afterAsays’)// b1.say() would invoke b1.afterAsays

function E() { this.a = undefined; this.setup = function () {};}

var e = new E();

system.injectInto(e);// e.setup invoked

CONVENTIONsetup method of module would be invoked

after getObject or injectInto

INJECT SYSTEM ITSELF

system.mapValue('system', system);

function D() { this.system = undefined;}

system.mapSingleton('d', D);

// system.getObject(‘d’) has system as dependency

function A() {}; function B() {}

function Dep() { return { system: undefined, setup: function () { this.system.autoMapOutlets = true; // map dep here this.system.mapSingleton('a', A); this.system.mapClass('b', B); } };}

function App() { var b; var system = new dijon.System(); system.mapValue('system', system); system.mapOutlet('system'); var dep = new Dep(); system.injectInto(dep); // inject system to dep and invoke setup function, which map all deps

return { system: undefined, a: undefined, setup: function () { b = system.getObject('b'); },

init: function () { system.injectInto(this); // after init, app.a exists } };}// after new App().init(), app get all dep setup in Dep

SUMMARY OF DIJONJS• 3kb

• very little convention

• focus on di and notify / handler

BUILD PROJECT

• list all source files in index.html of testing page

• grunt as task manager

• concat and minify scripts with grunt-usemin

• export to dist/player.min.js

ONE LAST NOTICE

• ui should be totally separated

• ui <-> player -> videoElement

• ui order player by api, respond to player behavior by event

playervideoElement

ui

FUTURE OF YAPI.JS

• firefox / safari drm support

• live

• ads

• smart tv / chromecast / nexus player

THANK YOUquestions ?