Creating rich WebRTC Applications with Kurento

IETF 96 Hackathon17 July 2016, Berlin

Real-Time media Communications

Capture Encode Cipher Transport Transport Decipher Decode RenderNetwork

The media plane

The signaling planeI wan to call you, do you accept?

Yes, do it in this way

WebRTC to the rescue.

Before WebRTCFirst wave of

WebRTC technologies

Begin End• APIs• Standards• FOSS

Development experience

when working with real-time media

Common WebRTC application (p2p communications)

WebRTC video stream

4

WebRTC infrastructures

Peer-to-Peer WebRTC Application (no media infrastructure)WebRTC video stream

WebRTC Application based on media infrastructuremedia infrastructure

5

WWW VS (Web)RTC apps

Control

Multimedia Application logic

(developers’ code)

Multimedia CapabilitiesRecording

TranscodingRoutingMixing

AnalyzingEtc.

MediaTraffic

RTC Media APIs

Multimedia Clients

Events

Application Signaling

Control

WWW Application logic

(developers’ code)

Database CapabilitiesData storageData recoveryData querying

Data processingEtc.

DD.BB. API

WWW Clients

Events

Application Signaling

6

7

Control

Multimedia Application logic

(developers’ code)

Multimedia CapabilitiesRecording

TranscodingRoutingMixing

AnalyzingEtc.

RTC Media APIs

Events

8

Kurento: a WebRTC infrastructure and its APIs

Control

Multimedia Application logic

(developers’ code)

RecordingTranscoding

RoutingMixing

AnalyzingAdapting

MediaTraffic

RTC Media APIs

Multimedia Clients

Events

Application Signaling

Computervision

Augmentedreality

BlendingEtc.

Multimedia Capabilities

Kurento Media Server(KMS)

Kurento Client APIKurento Room APIKurento Tree API

9

Cooking Kurento

9

10

The Kurento FOSS Community

Community support WebRTC worldwide reference

International awards More than 300 companies

Very active mailing list

First result in Google

11

The Kurento FOSS project− http://www.kurento.org

• Main web site

− https://www.twitter.com/kurentoms • Main social channel

− https://groups.google.com/forum/#!forum/kurento• Main mailing list

− https://github.com/kurento • Main repository

− https://www.youtube.com/channel/UCFtGhWYqahVlzMgGNtEmKug• Kurento Youtube channel

12

Developing with KMS:The Kurento Client API

Media Element• Provides a specific media

functionality

› Send/receive media

› Process media

› Transform media• Exchange media through› Sources

› Sinks

Media pipeline• Chain of media elements

implementing the desired media logic.

• The Media API provides the capability of creating media pipelines by joining media elements of the toolbox

Media Element

Sin

k

SR

C

13

A modular API− Modular in the sense of modularity*

• Isolationo Internal states of a module don’t affect the rest

• Abstraciono Internal states are hidden behind an interface

• Composabilityo Interface must enable module recombination and assembly

• Reusabilityo If a module has it, you can use it

• Extensibilityo If a module does not have it, you can add it.

* Baldwin, Carliss Young, and Kim B. Clark. Design rules: The power of modularity. Vol. 1. MIT press, 2000

14

The paradigm of modularity

15

The Kurento Client API in one word

sourceElement.connect(sinkElement)

connect

16

Kurento hello world

17

//MediaPipeline is a holder of media elementsMediaPipeline pipeline = kurento.createMediaPipeline();

//Create your media elementsWebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();

//Connect your media elementswebRtcEndpoint.connect(webRtcEndpoint);

Control

Multimedia Application logic

(developers’ code)

RecordingTranscoding

RoutingMixing

AnalyzingAdapting

Kurento Media APIs

Multimedia Clients

Events

Application Signaling

Computervision

Augmentedreality

BlendingEtc.

Multimedia Capabilities

//Where to render the local and remote streamsvar options = {

localVideo : videoInput, remoteVideo : videoOutput, onicecandidate : onIceCandidate }

//Start media capture and communicationswebRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(

options, function(error) { if (error) return console.error(error); webRtcPeer.generateOffer(onOffer); });

Application Server Code (Java)

Client Server Code

Application source for Spring (Java EE)Application source for Node.js (JavaScript)

18

Recording

Sin

kS

RC

Sin

k

//MediaPipeline is a holder of media elementsMediaPipeline pipeline = kurento.createMediaPipeline();

//Create your media elementsWebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build(); RecorderEndpoint recorderEndpoint = new RecorderEndpoint.Builder(pipeline).build(“file:///myfolder/myfile.mp4”);

//Connect your media elementswebRtcEndpoint.connect(webRtcEndpoint);webRtcEndpoint.connect(recorderEndpoint);

Application source for Spring (Java EE)Application source for Node.js (JavaScript)

19

Playing

Sin

kS

RC SR

C

//MediaPipeline is a holder of media elementsMediaPipeline pipeline = kurento.createMediaPipeline();

//Create your media elementsWebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build(); PlayerEndpoint playerEndpoint = new PlayerEndpoint.Buildr(pipeline).build(“file:///myfolder/myfile.mp4”); //RSTP and HTTP URIs also supported

//Connect your media elementsplayerEndpoint.connect(webRtcEndpoint);

Application source for Spring (Java EE)Application source for Node.js (JavaScript)

Interoperating

20

Media Pipeline

WebRTC audiostreaming

Sin

kS

RC

SR

CS

ink

Media PipelineS

ink

SR

C

SR

CS

ink

WebRTC videostreaming

RTP audiostreaming

RTP videostreaming

Transparent transcoding!!

21

Some Media processing modules− Augmented reality

• MarkerArModule• MarkerlessArModule

− Computer vision• FaceDetector• NoseDetector• EyeDetector• CrowdDetector• PointerDetector• MotionDetector• VirtualFence• Etc.

Sin

k

SR

C

Sin

k

SR

C

Sin

k

SR

C

Sin

k

SR

C

Sin

k

SR

C

22

Using media processing modules

Sin

kS

RC S

ink

SR

C

CrowdEventsS

RC

Sin

k

SR

C

CrowdDetector

RtspPlayer

Sin

kS

RC

See demo

See demo

23

Implementing group communications: just connect

Media Pipeline

WebRTCstreaming

Sin

kS

RC

WebRTCstreaming

SR

CS

ink

SR

CS

ink

WebRTCstreaming

WebRtcEndpoint 1

WebRtcEndpoint 3

WebRtcEndpoint 2

SR

CS

ink

WebRTCstreaming

WebRtcEndpoint 4

User 1 User 2

User 3

User 4

24

Group communications: RTP topologies

− Media Mixing Mixer• Composite

− Media Switching Mixer• Connect primitive• Natural topology

− SFU• WebRtcSfu

o Simulcast

Sin

kS

RC

SRC Sink

SinkSRC

SR

CS

ink

25

Creating your own modules

KurentoIDL Compiler

JSON

Module IDL Description

Java Java-Script

cpphpp

Media API code Toolchain

Auto-generated code

ModuleImplementation

cmake

Interface

Developer

(1)

(2)

(3)

26

Going up the API stack: the room API

createRoomdeleteRoom

joinRoomleaveRoom

publisMediaunpublishMedia

onParticipantJoinedonParticipantLeft

onMediaPublishedonMediaUnpublished

onMessage

See demo

27

Going up the API stack: the tree API

createTreedeleteTree

joinAsPresenterjoinAsViewer

leave

28

What’s next

29

Where to start

− http://www.kurento.org/documentation • Installation and administration guide• Java tutorials for Spring and Java EE

developers• JavaScript tutorials for Node.js developers• JSON-RPC protocol documentation

− https://github.com/kurento • Main repository

30

Thanks