WebRTC 1.0Real-Time Communications in the Browser

Huib KleinhoutProduct Manager, Google Stockholm

@hkleinhout

2011

2018

>1.8BWeekly Chrome audio/video minutes, 3X from last year

>1300WebRTC-based companies and projects

>80%Of all installed browsers have now WebRTC built-in

>10kDownloads of our AAR lib last 30 days

>6k+Downloads of our Cocoapod last 30 days

WebRTC in YouTube Live

WebRTC in YouTube Live

● Works in browser and mobile

● Opus + H.264

Replaces RTMP

● Decreased buffer rate

● Increased quality

WebRTC 1.0

Specs Compliance Tests Reliability

Well documented Consistent across browsers

Can be fully tested

Works every time

Completing version 1.0

Version 0.9 Version 1.0

WebRTC pipeline and standards

capture

codingprocessing

decodingprocessing

session

session

playback

transport

transport

Javascript client

Javascript client

MediaCapture and StreamsWebRTC 1.0 API

Audio & Video codecsVP8, H264, OPUS

Session managementData channelBandwidth estimationJSEP, SDP, FEC...

PacketisationSecurityNAT transversalSTUN,TURN, ICE, RTP, SCTP, DTLS...

STA

NDA

RDS

Standardization of WebRTC 1.0

W3C IETF

WebRTC 1.0 CR JSEP ~RFC

Media Capture and Streams CR Data Channel ~RFC

Identifiers for WebRTC's Statistics CR RTP Usage ~RFC

Transports ~RFC

Audio/Video codecs RFC

Requirements RFC

StandardsCompliance

Making Chrome fully compliant with the current standards

WebRTC SDP based

2011 2014

ORTC Object based API’s

WebRTC 1.0SDP based with some object API’s

2016 2018

SDP Munging

v=0o=- 8858844963286989377 3 IN IP4 127.0.0.1s=-t=0 0a=group:BUNDLE audio video dataa=msid-semantic: WMSm=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126c=IN IP4 0.0.0.0a=rtcp:9 IN IP4 0.0.0.0a=ice-ufrag:NkPRa=ice-pwd:ywKe3pGpkiUGGuJ4lImufBcsa=ice-options:tricklea=fingerprint:sha-256 F7:EA:F9:45:48:A6:F8:2D:DD:BF:26:E0:47:A0:A9:57:00:D6:27:A9:BA:FE:CD:C0:10:0E:DB:BA:05:92:8E:01a=setup:activea=mid:audioa=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-levela=recvonlya=rtcp-muxa=rtpmap:111 opus/48000/2a=rtcp-fb:111 transport-cca=fmtp:111 minptime=10;useinbandfec=1m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102 122 127 121 125 107 108 109 124 120 123 119 114c=IN IP4 0.0.0.0

a=rtcp-fb:100 nack plia=fmtp:100 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001fa=rtpmap:101 rtx/90000a=fmtp:101 apt=100a=rtpmap:102 H264/90000a=rtcp-fb:102 goog-remba=rtcp-fb:102 transport-cca=rtcp-fb:102 ccm fira=rtcp-fb:102 nacka=rtcp-fb:102 nack plia=fmtp:102 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001fa=rtpmap:122 rtx/90000a=fmtp:122 apt=102a=rtpmap:127 H264/90000a=rtcp-fb:127 goog-remba=rtcp-fb:127 transport-cca=rtcp-fb:127 ccm fira=rtcp-fb:127 nacka=rtcp-fb:127 nack plia=fmtp:127 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f

SDP “API”

a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d0033a=rtpmap:109 rtx/90000a=fmtp:109 apt=108a=rtpmap:124 H264/90000a=rtcp-fb:124 goog-remba=rtcp-fb:124 transport-cca=rtcp-fb:124 ccm fira=rtcp-fb:124 nacka=rtcp-fb:124 nack plia=fmtp:124 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=640033a=rtpmap:120 rtx/90000a=fmtp:120 apt=124a=rtpmap:123 red/90000a=rtpmap:119 rtx/90000a=fmtp:119 apt=123a=rtpmap:114 ulpfec/90000m=application 9 DTLS/SCTP 5000c=IN IP4 0.0.0.0b=AS:30a=ice-ufrag:NkPRa=ice-pwd:ywKe3pGpkiUGGuJ4lImufBcsa=ice-options:trickle

RTC Transceivers and Unified Plan in Chrome

Interface Version

RTCRtpReceiver and RTCRtpContributingSource support 59

RTCRtpSender (addTrack) 64

RTCRtpSender.replaceTrack 65

RTCRtpReceiver 65

Unified Plan SDP (optional) 68

RTCRtpTransceiver 69

RTCRtpEncodingParameters 70

sendEncodings 71

Unified Plan default 72 (feb 2019)Current Chrome version: 70

Voice Engine Video Engine Transport

Audio capturing/rendering Video capturing/rendering Network I/O

Echo cancellation, noise suppression, gain control

NetEQ

Opus, ISAC, G.711

Bandwidth Estimation, error correction

Video jitter buffer

VP8, VP9, H.264

P2PSTUN/TURN/ICE

Multiplexing

SRTP

Session management / abstraction

WebRTC C++ API (PeerConnection)

W3C Web APIs iOS Objective-C APIs Android Java APIs

Web

RTC

libr

ary

AP

IA

pps

Browser support

Chrome Firefox Edge Safari

Library webrtc.org webrtc.org proprietary webrtc.org

Version current 1-6 months behind 1-8 months behind

JS layer Follows C++ layer Custom ORTC &WebRTC

Follows C++ layer

Testing

Testing WebRTC - wpt.fyi

Need more green!

● Test automation

● Spec compliance work

Web Platform Test results (Oct 2018)

Chrome Firefox Edge Safari

Media Capture 83% 74% 12% 80%

Peer Connection 44% 55% 16% 45%

Testing WebRTC - Interoperability

javascript

HTTP

SRTP/SCTP

STUN/TURN

javascript javascript

Web Platform Tests

KITE

Testing WebRTC - KITE

Hardware, VMs, hosted VMs...

AppRTC tests

Selenium Grid

KITE dashboard

WebDriv

er

github.com/webrtc/KITE

Reliability

Reliability in Chrome

Common reliability problems

● Incorrect device selection

● Muted, volume problems

● Exclusive locks on devices

● Audio driver instability

● Forwarding problems

Signals

Device API’s

● Name/label/HID

Events

● Trackend

● Muted

Metrics

● Audio energy

Issue Status Version

Audio glitches on Linux, CrOS Fix in Chrome POSIX layer 60

No audio from microphone on Mac OS X Workaround for Core Audio issue 63

Long delays when screen sharing Rework of pacing algorithm 63

Audio echo New AEC3 algorithm 69

Crashes in camera subsystem in OS Chrome Video Process 62

Crashes in audio subsystem in OS Chrome Audio Process 70

Audio driver hangs Hang detection & restart 71

Reliability in Chrome

Current Chrome version: 70

No mic audio on Mac OS X

{TLDR}

Deep deep investigation of MacOS CoreAudio behavior revealed that sleeping and waking the computer can cause a race condition that results in the mic being disabled.

A fix for this issue was added in Chrome M63

Audio Capture failure rate on Mac OS X

8.5%

0.4%

M63

AEC3

● Redesigned WebRTC echo canceller solution for desktop platforms

● Addresses fundamental existing AEC issues

● Experimentation with built-in AEC in Windows and Mac

Screen sharing latency improvements

Interframe spacing on 200kbps link and 5% packet loss, 5fps

Chrome Audio Capture Process

Renderer

IPC

Browser

Renderer

IPC

Renderer

IPC

Platform audio

Problem:

● Recover from audio crashes

● Fix “no audio” caused by system layer

Solution:

● Audio Service

● Runs platform audio in a separate process

● Auto-restart on failure

Chrome Video Capture Process

● Video capture moved to separate process

● Driver crashes do not bring down the entire

browser.

● Process/capture can now be restarted in JS

● Simplifies IPC, data ownership and sharing

between processes.

● Released in M62

Coming soon

● VP9 provides ~35% better visual quality than VP8 at equivalent bitrate● HD at 700kbps up/down

VP9 SVC

Screen Capture

Browser Current getDisplayMedia()

Firefox getUserMedia() Under development

Edge getDisplayMedia() Supported

Chrome desktopCapture/tabCapture (extension API) Under development

Safari none Under development

● Different API implementations● Convergence towards extension-less getDisplayMedia()

WebRTC 1.0● 6+ years of engineering● Fully open source● Open and free to use standards● Implemented in all major browsers

High quality real time communication in the browser

WebRTC + Browser tech = fun!

WebRTC + WebGL = AR, VR, shader based video processing

WebRTC + TensorFlow.js = AI in communications

WebRTC + Screen capture = Game streaming

WebRTC + Canvas capture = Video Mixing

WebRTC + PWA = Installable Web Apps

+ cross browser / instant updates / fully sandboxed / open standards!

Thank you@hkleinhout