The State of WebRTC

WEBRTCT H E S T A T E O F

@ROBHAWKES

Hi, I’m Rob Hawkes and I’m here today to talk about the state of WebRTC.

ROB HAWKESWHO IS

A little about me…

- Digital Tinkerer (I play with new technologies and see how far I can push them)- Former Technical Evangelist at Mozilla (worked on HTML5 games, Firefox OS, and Web tech in general)- Author of multiple books, such as Foundation HTML5 Canvas- Currently taking time out to work on personal projects, such as…

VIZICITIES.COM

ViziCities, an experimental project that uses WebGL to bring real-world cities to life in the browser… in 3D!

http://vizicities.com

@ROBHAWKES

WHAT IS WEBRTC?

What is WebRTC?

Peer-to-peer communication technology for the browser.

@ROBHAWKES

W E B R T C I S A F R E E , O P E N PROJECT THAT ENABLES WEB BROWSERS WITH REAL-TIME C O M M U N I C A T I O N S ( R T C ) C A P A B I L I T I E S V I A S I M P L E JAVASCRIPT APIS.

WEBRTC.ORG

“WebRTC is a free, open project that enables web browsers with Real-Time Communications (RTC) capabilities via simple Javascript APIs.”

http://www.webrtc.org

@ROBHAWKES

AT A BASIC LEVEL, WEBRTC IS L IKE SKYPE RUNNING IN THE BROWSER. SORT OF…

ROB HAWKES

At a basic level, WebRTC is like Skype running in the browser.

However, this is barely scratching the surface of what it can really do.

@ROBHAWKES

WebRTC is a W3C specification, supported by Google, Mozilla, and Opera, amongst others.

It’s currently in the editorial process and is not yet a full recommendation.

The spec is a fantastic resource for in-depth technical information about WebRTC and how it should be implemented as per the browser. It’s a good read if you want to get right into the API structure.

http://dev.w3.org/2011/webrtc/editor/webrtc.html

@ROBHAWKES

The official WebRTC site is also a fantastic resource for a slightly more human-readable overview of the technology and the underlying architecture supporting it.

The FAQ is also a good place to start when you have questions.

http://www.webrtc.org/

@ROBHAWKES

WEBRTCVS

WEBSOCKETS

Now we already have real-time communication technologies such as WebSockets, so why use WebRTC?

At the the most basic level, WebSockets and WebRTC achieve different goals.

WebSockets is purely about providing a reliable real-time data connection in JavaScript.

On the other hand, WebRTC is about providing a network infrastructure for real-time media communication.

@ROBHAWKES

WEBSOCKETS

Communication methods such as WebSockets require a server to act as a middleman, passing messages back and forth between users.

@ROBHAWKES

WEBSOCKETS

The problem with a middleman approach is that you have to wait for your message to reach the server, be passed onto and reach the recipient, have a response sent back to the server, which in turn is then sent back to you.

If both users had a latency to the server of 20ms, which is conservative, a round trip between the two users would take 80ms. That’s nearly a tenth of a second, just ferrying messages back and forth.

@ROBHAWKES

WEBRTC

On the other hand, WebRTC is a peer-to-peer connection directly between 2 users. No middleman.

@ROBHAWKES

WEBRTC

This means you can send messages directly to the other user without hopping via a server.

If the latency between the users was 20ms then a round-trip would take just 40ms, half the time it would take with something like WebSockets.

@ROBHAWKES

UDPVS

TCP

Another big draw to WebRTC is that it primarily uses UDP, a method for sending data across a network in an unreliable fashion.

In comparison, technologies such as HTTP use TCP which is a reliable method of sending data.

Without going into too much detail, UDP is a lot faster than TCP because it doesn’t worry about order or error correction. Because of this you wouldn’t want to use it to send important data that needs to be received in order and whole.

Instead, UDP is commonly used in situations where you only care about the very latest piece of data and have no need to wait for data sent in the past to be received correctly. VoIP and multiplayer games are perfect examples of this.

WebRTC is the first time UDP has been available in the browser without plugins and because of this a lot of developers are excited to get their hands on it. I know I am!

http://www.skullbox.net/tcpudp.phphttp://gafferongames.com/networking-for-game-programmers/udp-vs-tcp/

@ROBHAWKES

WEBRTC USE CASES

The ability to connect peer-to-peer with another user certainly sounds cool, but what exactly is it useful for?

@ROBHAWKES

VIDEO AND AUDIO CALLS

The most common use-case is video and audio calls, a la. Skype.

This seems to be the approach most people are taking with WebRTC, which is fair considering that’s what it has been marketed as.

But WebRTC is more than that…

@ROBHAWKES

SCREEN SHARING

Screensharing is another approach, similar to the idea of sharing video.

@ROBHAWKES

COLLABORATIVE ACTIVITIES

There are also collaborative activities like writing or coding, this is another area which is gaining a lot of interest.

@ROBHAWKES

MULTIPLAYER GAMES

Multiplayer games are an area that will benefit hugely from WebRTC.

This is mainly because of UDP and the ability to have unreliable network data connections, an approach used by the vast majority of multiplayer games outside of the Web.

We also shouldn’t forget that the peer-to-peer nature of WebRTC will also allow for a new breed of Web games that don’t need servers to be played multiplayer. That will be very interesting to see progress.

@ROBHAWKES

P2P FILE SHARING

A slightly more interesting use of the technology is peer-to-peer file sharing.

The ability to quickly send a file directly to a friend without leaving the browser is certainly an appealing one.

Also, imagine the possibility for something like BitTorrent in the browser!

@ROBHAWKES

P2P CDN

Something I’d never considered before is the idea of using WebRTC to create a peer-to-peer CDN amongst your website visitors.

The idea of share the serving of assets amongst a network of peer-to-peer connections is very intriguing.

@ROBHAWKES

REMOTE CONTROL

Something that flips the common approach of WebRTC is the idea of remote control of devices from other devices.

In this case, instead of connecting to another browser controlled by another person you are connecting two devices owned by the same person.

This way you can use one device to control the other without having to worry about a server or Internet connection. Theoretically this could work over the local network.

One approach where this would be useful is for using mobile phones to act as a gamepad for games playing on a desktop or TV.

@ROBHAWKES

WEBRTC APIS

There are 3 main APIs involved in WebRTC.

Let’s quickly take a look at each of them.

@ROBHAWKES

MEDIASTREAM

MediaStream API, commonly referred to as getUserMedia.

This API gives you access to media data streams from devices such as webcams and microphones.

http://dev.w3.org/2011/webrtc/editor/getusermedia.html

@ROBHAWKES

GETUSERMEDIAIS NOT

“WEBRTC SUPPORT”

getUserMedia is not WebRTC.

@ROBHAWKES

ASCII Webcam

@ROBHAWKES

PEERCONNECTION

PeerConnection API.

This API lets you make peer-to-peer connections and attach media streams like video and audio.

http://dev.w3.org/2011/webrtc/editor/webrtc.html#peer-to-peer-connections

@ROBHAWKES

DATACHANNEL

DataChannel API.

This API lets you send and receive arbitrary data across a peer-to-peer connection.

http://dev.w3.org/2011/webrtc/editor/webrtc.html#peer-to-peer-data-api

@ROBHAWKES

DESKTOPSUPPORT

We’ve seen what WebRTC offers, but what is the real-world support like?

It’s still a very new technology but things are looking good, at least on desktop.

@ROBHAWKES

Firefox supports WebRTC by default in version 22, which is currently going through the testing stages in the Nightly and Aurora versions.

If all goes well, functional WebRTC support will land in the public release of Firefox by the end of June.

https://hacks.mozilla.org/2013/04/webrtc-update-our-first-implementation-will-be-in-release-soon-welcome-to-the-party-but-please-watch-your-head/

@ROBHAWKES

The current public release of Chrome has support for basic audio and video WebRTC calls enabled by default.

DataChannel is supported in Chrome beta and is currently behind a flag.

http://www.html5rocks.com/en/tutorials/webrtc/basics/http://updates.html5rocks.com/2013/02/WebRTC-data-channels-API-changes-and-Chrome-talks-to-Firefox

@ROBHAWKES

Opera only supports the ability to get video and audio via getUserMedia.

This is not really WebRTC support.

@ROBHAWKES

IE has no WebRTC support natively but support can be added if you use Chrome Frame.

Gotta love Chrome Frame!

http://www.google.com/chromeframe

@ROBHAWKES

M O B I L ESUPPORT

While desktop support is pretty good, considering, mobile support isn’t so good.

@ROBHAWKES

Firefox on Android has WebRTC support behind a config flag.

It’s very early days and your mileage may vary while using it as the mobile implementation is changing quite rapidly.

Mozilla plan to firm up their mobile WebRTC support in the coming months.

http://www.morbo.org/2013/04/webrtc-support-on-android.html

@ROBHAWKES

Chrome on Android also has WebRTC support behind a config flag.

It seems to be quite reliable and you can certainly make calls between Chrome mobile and a desktop browser. Very cool!

https://groups.google.com/d/msg/discuss-webrtc/uFOMhd-AG0A/81p3dE_5peYJ

@ROBHAWKES

Opera Mobile only has support for getUserMedia.

Again, this is not really WebRTC support.

@ROBHAWKES

INTEROPERABILITY

Browser support for the APIs is all well and good, but what is the interoperability like?

It’s not very useful to have WebRTC support if you can’t make a call between different browsers and platforms.

Up until recently the situation wasn’t very promising.

http://www.webrtc.org/interop

However, earlier this year it was announced that a WebRTC connection can be made between public builds of Firefox and Chrome.

This was pretty big news and was co-announced on both the Chrome dev blog and the Mozilla Hacks blog. It was one of the rare occurrences where the browser wars were put aside and true cooperation was embraced.

http://blog.chromium.org/2013/02/hello-firefox-this-is-chrome-calling.htmlhttps://hacks.mozilla.org/2013/02/hello-chrome-its-firefox-calling/https://apprtc.appspot.com/

@ROBHAWKES

WEBRTC DEMO

If all goes well, I’m going to show you a quick demo of WebRTC.

http://freshtilledsoil.com/the-future-of-web/webrtc-video/https://apprtc.appspot.com

@ROBHAWKES

USING WEBRTC

Let’s take a look at how to actually use WebRTC.

@ROBHAWKES

MEDIASTREAM

The most basic API to look at first is the MediaStream API, specifically the getUserMedia method.

@ROBHAWKES

var constraints = {video: true};

function successCallback(localMediaStream) { var video = document.querySelector("video"); video.src = window.URL.createObjectURL(localMediaStream); video.play();}

function errorCallback(error){ console.log("getUserMedia error: ", error);}

getUserMedia(constraints, successCallback, errorCallback);

HTML5ROCKS.COMGETUSERMEDIA DEMO

In this example we have an implementation for accessing the webcam.

It’s pretty basic but it gets the job done. The only other thing you need is a HTML video element to attach the webcam stream to.

It’s worth pointing out that, although you can’t see it, we’re making use of a simple polyfill to remove the (very few) browser inconsistencies in the API calls.

You also need to run this through a domain, not the local filesystem, otherwise security restrictions will prevent you from accessing the media devices.

http://www.simpl.info/getusermedia/https://code.google.com/p/webrtc-samples/source/browse/trunk/apprtc/js/adapter.js

@ROBHAWKES

PEERCONNECTION

Things get a little more complicated when we look at the PeerConnection API, but once you understand the process it’s not so bad.

We’ll first look at an example that makes a local PeerConnection call within a single browser page.

It’s a good way to look at the code before adding in the extra functionality to make remote calls.

@ROBHAWKES

function start() { getUserMedia({audio:true, video:true}, gotStream, function() {});}

function gotStream(stream){ vid1.src = window.URL.createObjectURL(stream); localstream = stream;}

...

HTML5ROCKS.COMPEERCONNECTION DEMO (1/4)

In this example we’re again making use of the polyfill to avoid the browser prefix dance. We’re also missing the HTML as it’s most important to understand the JavaScript.

1. The first step is to call getUserMedia2. Next, we connect the local stream to a HTML video element to you can see yourself3. Next, we store a reference to the local stream so we can access it later on

http://www.simpl.info/rtcpeerconnection/https://code.google.com/p/webrtc-samples/source/browse/trunk/apprtc/js/adapter.js

@ROBHAWKES

...

function call() { var servers = null;

window.pc1 = new RTCPeerConnection(servers); pc1.onicecandidate = iceCallback1;

window.pc2 = new RTCPeerConnection(servers); pc2.onicecandidate = iceCallback2; pc2.onaddstream = gotRemoteStream;

pc1.addStream(localstream); pc1.createOffer(gotDescription1);}

...


At this point we haven’t achieved anything more then the previous getUserMedia example.

Once the webcam and audio is set up we can begin the process of making a PeerConnection.

1. Firstly, we call the RTCPeerConnection method and pass it an empty variable - This variable can be used to define TURN and STUN servers, which can help when routing through firewalls2. The next step is to set up an event handler for onicecandidate - This is used to work out which methods work best for making a connection3. Next, we call RTCPeerConnection again and use this as our ‘remote’ connection4. We set up the onicecandidate handler for the remote connection5. And also set up the onaddstream handler for the remote connection - This lets you know when a remote stream has been received by PeerConnection6. Now we add the local media stream to the local PeerConnection7. Finally, we call the createOffer method of PeerConnection and pass a handler to deal with the returned session description. - We’ll look at the handler in more detail in a moment

@ROBHAWKES

v=0o=- 5948426442422644553 2 IN IP4 127.0.0.1s=-t=0 0a=group:BUNDLE audio videoa=msid-semantic: WMS qcBqgcQpLthJWqiwDV4Hbujhpj0rTn5vvUZRm=audio 1 RTP/SAVPF 111 103 104 0 8 107 106 105 13 126

...

a=ssrc:2770545510 label:qcBqgcQpLthJWqiwDV4Hbujhpj0rTn5vvUZRv0

SESSION DESCRIPTION EXAMPLE

The session description we get returned by createOffer is a method of describing the streaming media connection that we want to make between the two peers.

This is just an example session description; what you see will vary depending on what you’re trying to do and your network situation.

http://www.html5rocks.com/en/tutorials/webrtc/basics/#toc-signalinghttp://en.wikipedia.org/wiki/Session_Description_Protocol

@ROBHAWKES

...

function iceCallback1(event){ if (event.candidate) { pc2.addIceCandidate(new RTCIceCandidate(event.candidate)); }}

function iceCallback2(event){ if (event.candidate) { pc1.addIceCandidate(new RTCIceCandidate(event.candidate)); }}

...


Moving on; this is what the onicecandidate handlers look like, for both the local (top) and remote (bottom) connections.

What’s happening here is that the returned connection methods (candidates) are added to the opposite connection so they both know the best way to connect to the other.

However, we’re still not ready to make the connection.

@ROBHAWKES

...

function gotDescription1(desc){ pc1.setLocalDescription(desc); pc2.setRemoteDescription(desc); pc2.createAnswer(gotDescription2);}

function gotDescription2(desc){ pc2.setLocalDescription(desc); pc1.setRemoteDescription(desc);}

function gotRemoteStream(e){ vid2.src = window.URL.createObjectURL(e.stream);}


The last chunk of code allows us to successfully hook up the PeerConnection and tidy up the loose ends.

1. The first thing we do is set up the gotDescription1 handler, which is called by the createOffer method from earlier - This returns the session description for the local connection2. We then call the setLocalDescription method on the local PeerConnection3. We then call the setRemoteDescriotion method on the ‘remote’ PeerConnection - At this point we’re nearly ready to make the connection, we just need to respond to the offer4. The next step is to call createAnswer on the ‘remote’ PeerConnection and pass a handler for the returned session description.5. From here we jump into the gotDescription2 handler which deals with the session description for the ‘remote’ PeerConnection created by createAnswer - Now it’s just a case of tying up the loose ends6. We call setLocalDescription on the ‘remote’ PeerConnection and pass it the session description7. We then call setRemoteDescription on the local PeerConnection and pass it the same session description - At this point you’ve successfully connected the two peers, but there is one last step…8. The last thing to do is to set up the gotRemoteStream handler - This is called when a media stream is added by a remote peer. Eg, when the connection starts. - All this does is set the stream, in this case a video, to another HTML video element

And you’re done!

@ROBHAWKES

SIGNALLING

The problem with the previous demo is that it only runs within a single browser page.

Most of you who want to use WebRTC will want to make connections between remote browsers.

A full code example of a remote WebRTC demo is beyond the scope of this talk, though I’ve linked to a great example in the slides.

Instead, what we can do is briefly take a look at the concept of signalling… the discovery and connection of remote peers.

http://www.webrtc.org/demohttp://www.html5rocks.com/en/tutorials/webrtc/basics/#toc-signaling

@ROBHAWKES

LOCAL PEERCONNECTION

With the local PeerConnection example the act of peer discovery was not needed as the two peers were running from the same page and as such knew about each other already.

@ROBHAWKES

DISCOVER PEERS VIA SERVER

#ROOM1 #ROOM1

With a full remote PeerConnection, the first step is to use a server to discover other peers (users) wanting to make a WebRTC connection. Ie. We use the server to work out which computers want to connect with each other as we can’t do that locally.

Common techniques include the use of a room name or a unique identifier that two peers can both use so the server knows who to pair.

Once peers are paired a handshake is performed so that each peer knows just enough information to connect to the other one. This is done by sharing ICE information and the session description.

You’ve already done this with the local demo but it becomes a little more complicated when you do it remotely.

Common approaches for this handshake use AJAX or WebSockets to send the ICE candidates and session descriptions between peers via the server.

To be clear the server is purely acting as a messaging proxy.

And this is the only step that requires a server.

http://blog.gingertech.net/2012/06/04/video-conferencing-in-html5-webrtc-via-web-sockets/http://www.html5rocks.com/en/tutorials/webrtc/basics/#toc-signaling-with-channelhttps://sites.google.com/site/muazkh/webrtc-order-the-code

@ROBHAWKES

DIRECTLY CONNECT PEERS

Once the session description handshake has been made the two peers will be connected directly.

At this point the server is no longer needed and you have a WebRTC connection up and running.

It sounds quite simple in theory, and it kind of is thanks to the API introduced with WebRTC. The underlying technology and networking to make this happen is much, much more complicated.

@ROBHAWKES

DATACHANNEL

At this point you may have a PeerConnection set up that is streaming webcam and microphone input between two browsers.

If you want to then send arbitrary data over the PeerConnection you’ll need to set up a DataChannel.

Let’s look at that next.

@ROBHAWKES

...

window.dc1 = pc1.createDataChannel("dataChannel1", {reliable: false});

...

pc2.ondatachannel = function(event) { window.dc2 = event.channel; dc2.onmessage = handleMessage;};

var handleMessage = function(event) { console.log("DataChannel message received from pc1: \n " + event.data);}

...

dc1.send("Hello, world!");

DATACHANNEL DEMO

We’ll keep this one brief as it mostly relies on an existing PeerConnection, which you now know how to set up.

In this example we make a call to the createDataChannel method on the local PeerConnection, passing it a label and some option.

We’re passing the reliable: false property in the options to use unreliable UDP for the DataChannel.

We then set up a handler on the remote PeerConnection to listen for a DataChannel connection.

When the DataChannel is received we store a reference to it and set up a handler to listen for incoming messages.

From there, we can then call the send method on the local DataChannel to send messages to the remote peer.

The DataChannel API is feels very similar to WebSockets.

@ROBHAWKES

WEBRTC IN USE

Now that we’ve seen how WebRTC works, let’s take a look at some of the projects that are using it today.

@ROBHAWKES

TowTruck by Mozilla.

Collaborative browsing, editing, chat, and voice.

Potential demo…

https://towtruck.mozillalabs.com

@ROBHAWKES

Conversat.io

Small video chat rooms.

http://conversat.io/

@ROBHAWKES

PeerJS

Awesome cross-browser DataChannel framework.

They also offer developers free signalling servers for your PeerJS apps, at least up to 50 concurrent connections.

http://peerjs.com/

@ROBHAWKES

Twelephone

Twitter-powered video and text chat.

http://twelephone.com/

@ROBHAWKES

Twilio

Increasingly-popular service that allows you to make and receive real-world phone calls programatically, including from a browser with WebRTC.

http://www.twilio.com/

@ROBHAWKES

PeerCDN

Offloading the serving of website assets to a peer-to-peer network of connected visitors.

https://peercdn.com/

@ROBHAWKES

Peerkit

Another approach at a peer-to-peer CDN.

Potential demo…

http://peerkit.com/

@ROBHAWKES

BananaBread

Peer-to-peer multiplayer in a 3D WebGL game.

https://hacks.mozilla.org/2013/03/webrtc-data-channels-for-great-multiplayer/

@ROBHAWKES

Experimental sharing of browser tabs in Chrome.

http://updates.html5rocks.com/2012/12/Screensharing-with-WebRTC

@ROBHAWKES

Phono

A jQuery plugin for making phone calls from the browser.

http://phono.com/webrtc

@ROBHAWKES

Att.js

Effectively powered by Phono, with a few extra bits and pieces.

https://att.io/

@ROBHAWKES

OpenTok

Another platform for creating audio and video calls within the browser using WebRTC.

http://www.tokbox.com/

@ROBHAWKES

File Hangout

Experiment with DataChannels for sharing files amongst multiple peers.

https://webrtc-experiment.appspot.com/file-hangout/

@ROBHAWKES

Sqwiggle

Collaborative dashboard for remote working, allowing you to see each other and share resources throughout the day.

https://www.sqwiggle.com/

@ROBHAWKES

Ericsson Labs 3D WebRTC demo

Using output from a depth sensor synchronised with a peer-to-peer video call.

https://labs.ericsson.com/blog/3d-webrtc-3d-video-communication-in-a-browser

@ROBHAWKES

Codassium

Collaborative code editor, marketed towards programming interviews.

http://codassium.com/

@ROBHAWKES

Grimewire ‘OS’

A crazy project aimed at creating a browser OS that uses WebRTC. I’m not even sure how yet, but it certainly sounds cool!

http://blog.grimwire.com/#2013-04-04-grimwire.md

@ROBHAWKES

G O F O R T HAND WEBRTC!

I hope that I’ve managed to pique your interest in WebRTC enough to start experimenting with it.

There is so much that the technology can do that people just aren’t exploring yet, particularly around to use of mobile devices.

I’d love to see what you make with it so make sure to ping me on Twitter if you do have a play.

Tweetmap.itReal-time visualisation of tweets

ViziCities.com3D visualisation of real-world cities

Slidesslideshare.net/robhawkes

Rawkes.comPersonal website and blog

RECENT PROJECTS MORE COOL STUFF

@robhawkesRob Hawkes

Get in touch with me on Twitter: @robhawkes

Follow my blog (Rawkes) to keep up to date with stuff that I’m working on: http://rawkes.com

ViziCities http://vizicities.com

Tweetmap http://tweetmap.it

These slides are online at http://slideshare.net/robhawkes

Technology

The State of WebRTC