A Practical Guide to Building WebRTC Apps

Ben Strongben@vline.comhttps://vline.com@vlineinc

vLine

10/09/2013

8/2011

My Background with WebRTC

Founded vLine in early 2011 to build tools and infrastructure for WebRTC

EDGE LOCATIONS ON 5 CONTINENTS

Global Infrastructure

Developer SDKs

Built what was probably the first WebRTC App

8/2013

vLine

vLine 3

Topics

1. Making Mobile Work Well

2. Choosing a Strategy for Multi-Party Conferencing

3. UI Considerations

vLine 4

Mobile: What is Possible?

[DEMO]

vLine 5

Mobile: The Basics

Android

• Works in Chrome 29+ and Firefox 24+

• Google Maintains Java PeerConnection APIs

• Works pretty well

iOS

• Not supported in any browser (thanks to Apple’s browser engine policies)

• Google Maintains Obj-C APIs

• Voice works well

• Video can be made to work well (this will get easier)

vLine 6

Android: Browser vs. Native

Browser Pros Native App Pros

• Re-use app built for desktop browsers.

• Supports the most powerful part of WebRTC: Links

• Push notifications!!!

• Other native APIs like address book and background services

• Easier to tune for particular devices

• Slightly better performance

vLine 7

Android: Browser vs Native

Our Answer: Both

• Make your web app work well on mobile, so that users without native app have low-friction experience

• Run the same app in PhoneGap with push notifications, address book integration, etc.

vLine 8

Mobile: Video Quality

• Mobile devices are less powerful than laptops and desktops(duh)

• Wide range of capabilities among mobile devices(smaller gap between high-end devices and low-end laptops than between low-end and high-end mobile devices)

• WebRTC methods for adapting to device capabilities don’t work well (yet)

vLine 9

Mobile: How WebRTC Adapts

Two Methods:

• Video Adapter reduces encode resolution if CPU is overloaded

• Remote Bitrate Estimator tells other side to reduce encode bitrate if decoder isn’t keeping up

Problems

• Only kick in after quality is visibly degraded

• Encoder and decoder compete for CPU, with unpredictable interactions

• Result is often bitrates of 50kbps on devices and networks that could support much better.

vLine 10

Mobile: Device Adaptation

The unfortunate reality

You’ll get much better results by imposing limits on resolution, framerate, and bitrate than by relying on the built-in adaptation methods.

vLine 11

Device Adaptation: The Knobs

Framerate and resolution are controlled by MediaConstraints passed to getUserMedia().function getStream(width, height, framerate) { var constraints = { video: { mandatory:{ minWidth:width, maxWidth:width, maxFramerate:framerate } } }; navigator.webkitGetUserMedia(constraints, onStream1, onFailure);}

Note: On Chrome, if another LocalMediaStream is active, the constraints will not take effect. Stop all other streams first!

vLine 12

Device Adaptation: The Knobs

Max bitrate and quantization set by adding VP8 fmtp parameters to SDP (Chrome only)

...a=rtpmap:100 VP8/9000a=fmtp:100 x-google-max-bitrate=700; x-google-max-quantization=20...

vLine 13

Mobile: Choosing Parameters

Two Strategies:

1. If at least one party is on mobile, set very low limits on resolution, framerate, and bitrate (e.g., 320x240 at 15fps and 400kbps)

2.Detect device capabilities, signal them to peers, and adapt encoding parameters appropriately

• Set encoding parameters to minimum of what the local device can encode and the remote device can decode

vLine 14

Mobile: Choosing Parameters

Detecting Device Capabilities

1. For native apps, Android and iOS APIs will give you fine-grained capabilities

2.For browser-based apps, run a javascript benchmark

• Run it once in a single web worker.

• Run it in parallel on 4 web workers.

• Infer CPU clock rate and number of cores.

3.Never encode at a higher resolution than the remote device’s screen!

15

Mass-Market TelepresenceWith HDMI outputs, 1080p-capable hardware codecs, and support for bluetooth conference mics, Android devices make the perfect embedded telepresence systems, starting at $100 per room.

A Mobile Bonus

HP Slate 21 - $400Tegra 4 processor and 21” screen. Perfect on a desktop or wall-mounted in a small conference room.

Galaxy NX - $1600Exnyos-5 Octa processor plus APS-C sensor. Mount it on a wall and plug in TV for a room system with broadcast-quality video.

HDMI Dongle - $50Rk3188 processor. Plug in a USB camera and a TV for a 1080p capable room system.

Note 10 - $500Exnyos-5 Octa processor. Put it on a conference table and plug in a wall-mounted TV and USB camera to power a larger room.

vLine 16

Multi-Party: Two Strategies

Mesh Topology Star Topology

vLine 17

Multi-Party: What is Possible?

[DEMO]

vLine 18

Multi-Party: Mesh Topology

Mesh Pros

• No server (simpler and cheaper infrastructure)

Mesh Cons

• More complex session management

• Only scales to about 4 participants

• Poor results on mobile devices (which are often hard-pressed to support a single peer)

vLine 19

Multi-Party: Star Topology

Star Pros

• Less processing and network load on clients

• Higher quality on low-end devices

• Potentially unlimited number of participants.

• Good place to record, do speaker detection, etc.

Star Cons

• Have to run lots of servers (complex and expensive)

• For fast machines on fast network connections, may be worse quality (mostly remedied with geo-distributed servers)

vLine 20

Multi-Party: Star Topology

How the Star Topology Works

• All clients connect to central server (usually called MCU or Media Router)

• Clients encode and send one stream. MCU fans it out.

vLine 21

Multi-Party: Star Topology

MCU/Router Strategies

• Mix/composite once and re-encode a single stream for each client (best quality. lowest load on client. very little flexibility on client. high load on server)

• Re-encode each stream for each client (high quality. lots of flexibility on client. more load on client. very high load on server)

• Restrict bandwidth, resolution, framerate, to lowest common denominator and just forward (low load, potentially poor quality)

• Advanced strategies: temporal scaling and simulcast

vLine 22

Multi-Party: Star Topology

Mixed approach is probably best

• Send two streams to server from each client (low and high res)

• For small differences, adapt to lowest-common denominator and forward.

• For large differences, re-encode.

• Use temporal scaling to fine-tune (if you control the encoder)

vLine 23

UI Considerations

For Video, Bigger is Better

• Use full window

• Encourage full-screen (especially on mobile)

• HD is best of all, but make sure devices on both ends can handle the CPU load, or you’ll get a worse experience.

vLine 24

UI Considerations

No one notices the GetUserMedia permissions UI!

• Show an arrow to point at it and don’t let them do anything else before accepting

• Be aware:• In Chrome, button positions vary by platform. • Firefox is different• On Android Phone devices, buttons are at bottom.

vLine 25

UI Considerations

Feedback for states and errors

• If you have a mute button, make it obvious when you're muted• Give the user feedback that you are trying to connect (or reconnect)

• If a session ends, distinguish between the other party hanging up and losing connection

• Use stats api to detect poor network conditions and provide feedback. User can then do something about it (e.g., move closer to wifi access point)

• If video bitrate stays at 50kbps, encourage (or force) user to switch to voice only