WebRTC... GWT & Browser Performance

Alberto Mancini, Francesca TosiJooinK.com

WebRTCPlug-in free realtime communication …

WebRTC is a free, open project that enables web browsers with Real-Time Communications (RTC) capabilities via simple Javascript APIs.

source: webrtc.org

WebRTC

Just another JS API

or

WebRTC is a new front in the long war for an open and unencumbered webBrendan Eich (Mozilla CTO and inventor of JavaScript)

?

Browser PiercingOnce upon a time ...

LDP: Firewall Piercing mini-HOWTOFrançois-René Rideauv0.97, 24 November 2001

Browser Piercing

a Javascript Application cannot

➔ access the filesystem and then File API

➔ open full-duplex socket connection and then WebSockets

Browser Piercing

a Javascript Application cannot

➔ use graphics accelerator and then WebGL

➔ span multiple threads and then WebWorkers

Browser Piercing

a Javascript Application cannot

➔ acquire audio and video and then WebRTC

➔ communicate P2P and then WebRTC

WebRTC across platforms● Chrome (mobile too)● Firefox (mobile too)● Opera● Native Java and Obj-C

WebRTC JS-APIAcquiring video and audio

navigator.getUserMedia(constraints,

successCallback,

errorCallback);

WebRTC JS-APIAcquiring video and audio

navigator.getUserMedia(constraints, successCallback,

errorCallback);

WebRTC JS-APIConstraints

● Controls the contents of the MediaStream● Media type, resolution, frame rate

var constraints = {video: true};

video: { mandatory: { ... }, optional [{... }]}

WebRTC JS-APIConstraints

● Controls the contents of the MediaStream● Media type, resolution, frame rate

var constraints = {video: true};

video: { mandatory: { chromeMediaSource: ‘screen’ }, optional [{... }]}

WebRTC JS-APIAcquiring video and audio

navigator.getUserMedia(constraints,

successCallback,

errorCallback);

WebRTC JS-APIerrorCallback

function errorCallback(error) { console.log("error: ", error);}

;-)

WebRTC JS-APIAcquiring video and audio

navigator.getUserMedia(constraints,

successCallback,errorCallback);

WebRTC JS-APIsuccessCallback

function successCallback(stream) { ...}

stream: MediaStream; a flux of audio- or video-related data.

WebRTC JS-APIsuccessCallback

function successCallback(stream) { var video = ... video.src =

window.URL.createObjectURL(stream);

}

(W3C File API)

WebRTC JS-APIfull sample

var constraints = {video: true};

function successCallback(stream) { var video = document.querySelector("video"); video.src = window.URL.createObjectURL(stream);}

function errorCallback(error) { console.log("navigator.getUserMedia error: ", error);}

navigator.getUserMedia(constraints, successCallback, errorCallback);

WebRTC JS-APIDo not forget

● to play the video

● to polyfill getUserMedia (moz*,webkit*,ms*)

WebRTC JS-APIRTCPeerConnection

→getUserMedia

+RTCPeerConnection

←

WebRTC JS-APIRTCDataChannel

Bidirectional communication of arbitrary data

● low latency● API similar to WebSockets

RTCPeerConnection samplepc = new RTCPeerConnection(null);pc.onaddstream = gotRemoteStream;pc.addStream(localStream);pc.createOffer(gotOffer);

function gotOffer(desc) { pc.setLocalDescription(desc); sendOffer(desc);}

function gotAnswer(desc) { pc.setRemoteDescription(desc);}

function gotRemoteStream(e) { attachMediaStream(remoteVideo, e.stream);}

RTCPeerConnection

RTCPeerConnection

.

RTCPeerConnection

.

STUN

.

TURN

.

Yes BUTVideoChat already exists and we are too late to build skype without skype ...

● Video chat:○ SimpleWebRTC○ easyRTC○ webRTC.io

● Peer-to-peer data:○ PeerJS○ Sharefest

Yes BUT

● getUserMedia: gives us real time stream from the cam

● we can draw the <video/> into a <canvas/>

● from the canvas we can extract pixel data getImageData()

and then elaborate.

Computing (GWT 4 us)goals- Almost native performance- Manage complex algorithms and applications

Computing (GWT 4 us)goals- Almost native performance- Manage complex algorithms and applications

our approach- javascript as a target language (GWT) - hand written optimized pieces of code

JooinK’s WebRTC sample

Sample

http://picshare.jooink.com

Demo …

PicShareTecnologies

● WebGL for 3D graphics● WebRTC for MediaSteams● ImgUR for pictures store● AppEngine as a glue

FOSS

gwt-nyartoolkit.googlecode.com

gwt-webgl.googlecode.com

The strategy: compile a java-lib with GWT into javascript

What’s nextChallanges

● real-time on mobile apps● real world recognition: computer vision

(BoofCV)

What’s nextOur approach

● use TypedArrays everywhere ● offload computation to the graphic

accelerator through WebGL● help the JIT compiler/optimizer by asm.js

That’s all folks!!!

Alberto Mancinialberto@jooink.com

Markers