FOSDEM 2016 - Creating rich WebRTC Applications with Kurento

Creating rich WebRTC Applications with Kurento

Luis [email protected]

http://www.kurento.org - [email protected]

Who I amAssociate Professor

Escuela Técnica Superior de Ingenieros de TelecomunicaciónUniversidad Rey Juan Carlos (south Madrid)

Researcher in the area of RTC

• http://www.nubomedia.eu• http://www.fiware.org

FOSS enthusiast

• Kurento project lead: http://www.kurento.org • NUBOMEDIA Community lead: http://www.nubomedia.eu

http://www.nubomedia.eu/

http://www.fiware.org/

http://www.kurento.org/

http://www.nubomedia.eu/


Real-Time media Communications

Capture Encode Cipher Transport Transport Decipher Decode RenderNetwork

The media plane

The signaling plane

I wan to call you, do you accept?

Yes, do it in this way


WebRTC to the rescue.

Before WebRTCFirst wave of

WebRTC technologies

Begin End• APIs• Standards• FOSS

Development experience

when working with real-time media

Common WebRTC application (p2p communications)

WebRTC video stream


WebRTC infrastructures

Peer-to-Peer WebRTC Application (without media infrastructure)

WebRTC video stream

WebRTC Application based on media infrastructuremedia infrastructure


WWW VS (Web)RTC applications

Control

Multimedia Application logic

(developers’ code)

Multimedia Capabilities

RecordingTranscoding

RoutingMixing

AnalyzingEtc.

MediaTraffic

RTC Media APIs

Multimedia Clients

Events

Application Signaling

Control

WWW Application logic


Database Capabilities

Data storageData recoveryData querying

Data processingEtc.

DD.BB. API

WWW Clients

Events




Control





RoutingMixing

AnalyzingEtc.

RTC Media APIs

Events


Kurento: a WebRTC infrastructure an its APIs

Control




RoutingMixing

AnalyzingAdapting

MediaTraffic

RTC Media APIs

Multimedia Clients

Events


Computervision

Augmentedreality

BlendingEtc.


Kurento Media Server(KMS)

Kurento Client APIKurento Room APIKurento Tree API


10

Cooking Kurento

10


The Kurento FOSS Community

Community support WebRTC worldwide reference

International awards More than 300 companies

Very active mailing list

First result in Google


The Kurento FOSS project• http://www.kurento.org

– Main web site

• https://www.twitter.com/kurentoms – Main social channel

• https://groups.google.com/forum/#!forum/ kurento– Main mailing list

• https://github.com/kurento – Main repository

• https://www.youtube.com/channel/UCFtGhWYqahVlzMgGNtEmKug– Kurento Youtube channel

http://www.kurento.org/

https://www.twitter.com/kurentoms

https://groups.google.com/forum/%23!forum/kurento



https://github.com/kurento

https://www.youtube.com/channel/UCFtGhWYqahVlzMgGNtEmKug




Developing with KMS:The Kurento Client API

Media Element• Provides a specific media

functionality

› Send/receive media

› Process media

› Transform media• Exchange media through› Sources

› Sinks

Media pipeline• Chain of media elements

implementing the desired media logic.

• The Media API provides the capability of creating media pipelines by joining media elements of the toolbox

Media Element

Sink

SRC


A modular API

• Modular in the sense of modularity*– Isolation

• Internal states of a module don’t affect the rest– Abstracion

• Internal states are hidden behind an interface– Composability

• Interface must enable module recombination and assembly– Reusability

• If a module has it, you can use it– Extensibility

• If a module does not have it, you can add it.

* Baldwin, Carliss Young, and Kim B. Clark. Design rules: The power of modularity. Vol. 1. MIT press, 2000


The paradigm of modularity


The Kurento Client API in one word

sourceElement.connect(sinkElement)

connect


Kurento hello world


//MediaPipeline is a holder of media elementsMediaPipeline pipeline = kurento.createMediaPipeline();

//Create your media elementsWebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();

//Connect your media elementswebRtcEndpoint.connect(webRtcEndpoint);

Control




RoutingMixing

AnalyzingAdapting

Kurento Media APIs

Multimedia Clients

Events


Computervision

Augmentedreality

BlendingEtc.


//Where to render the local and remote streamsvar options = {

localVideo : videoInput, remoteVideo : videoOutput, onicecandidate : onIceCandidate }

//Start media capture and communicationswebRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(

options, function(error) { if (error) return console.error(error); webRtcPeer.generateOffer(onOffer); });

Application Server Code (Java)

Client Server Code

Application source for Spring (Java EE)Application source for Node.js (JavaScript)

https://github.com/Kurento/kurento-tutorial-java

https://github.com/Kurento/kurento-tutorial-node




Recording

Sink

SRC

Sink


//Create your media elementsWebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build(); RecorderEndpoint recorderEndpoint = new RecorderEndpoint.Builder(pipeline).build(“file:///myfolder/myfile.mp4”);

//Connect your media elementswebRtcEndpoint.connect(webRtcEndpoint);webRtcEndpoint.connect(recorderEndpoint);







Playing

Sink

SRC SR

C


//Create your media elementsWebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build(); PlayerEndpoint playerEndpoint = new PlayerEndpoint.Buildr(pipeline).build(“file:///myfolder/myfile.mp4”); //RSTP and HTTP URIs also supported

//Connect your media elementsplayerEndpoint.connect(webRtcEndpoint);







Interoperating

Media Pipeline

WebRTC audiostreaming

Sink

SRC

SRC

Sink

Media Pipeline

Sink

SRC

SRC

Sink

WebRTC videostreaming

RTP audiostreaming

RTP videostreaming

Transparent transcoding!!


Some Media processing modules• Augmented reality

– MarkerArModule– MarkerlessArModule

• Computer vision– FaceDetector– NoseDetector– EyeDetector– CrowdDetector– PointerDetector– MotionDetector– VirtualFence– Etc.

Sink

SRC

Sink

SRC

Sink

SRC

Sink

SRC

Sink

SRC


Using media processing modules

Sink

SRC

Sink

SRC

CrowdEventsSR

C

Sink

SRC

CrowdDetector

RtspPlayer

Sink

SRC

See demo

See demo

https://www.youtube.com/watch?v=S6iWSCysgT0

https://www.youtube.com/watch?v=JlRg4PzeRKQ


Implementing group communications: just connect

Media Pipeline

WebRTCstreaming

Sink

SRC

WebRTCstreaming

SRC

Sink

SRC

Sink

WebRTCstreaming

WebRtcEndpoint 1

WebRtcEndpoint 3

WebRtcEndpoint 2

SRC

Sink

WebRTCstreaming

WebRtcEndpoint 4

User 1 User 2

User 3

User 4


Group communications: RTP topologies

Sink

SRC

SRC Sink

SinkSRC

SRC

Sink

• Media Mixing Mixer– Composite

• Media Switching Mixer– Connect primitive– Natural topology

• SFU– WebRtcSfu

• Simulcast


Creating your own modules

KurentoIDL Compiler

JSON

Module IDL Description

Java Java-Script

cpphpp

Media API code Toolchain

Auto-generated code

ModuleImplementation

cmake

Interface

Developer

(1)

(2)

(3)


Going up the API stack: the room API

createRoomdeleteRoom

joinRoomleaveRoom

publisMediaunpublishMedia

onParticipantJoinedonParticipantLeft

onMediaPublishedonMediaUnpublished

onMessage

See demo

https://www.youtube.com/watch?v=hkT8fLROdwo


Going up the API stack: the tree API

createTreedeleteTree

joinAsPresenterjoinAsViewer

leave


What’s next


Where to start

• http://www.kurento.org/documentation – Installation and administration guide– Java tutorials for Spring and Java EE developers– JavaScript tutorials for Node.js developers– JSON-RPC protocol documentation

• https://github.com/kurento – Main repository

http://www.kurento.org/documentation

https://github.com/kurento


Thanks

Luis [email protected]

Internet

FOSDEM 2016 - Creating rich WebRTC Applications with Kurento