WebRTC XamU [Autosaved] · The Basics of WebRTC Web Real-Time-Communication New(ish) open-sourced...

Cross-PlatformWebRTCALEXDUNN

TWITTER:@SUAVE_PIRATE

BLOG:HTTPS://ALEXDUNN.ORG

AgendaBriefIntroductiontoWebRTC

HowWebRTC Works

AdaptingtoNativePlatforms

ToolsandResources

LookatSomeCode

2

ResourcesFinishedsource:https://github.com/SuavePirate/Xamarin.WebRTC

WebRTC docs:https://webrtc.org

IceLink docs:http://docs.frozenmountain.com/icelink2/

Nativelibraries:https://webrtc.org/native-code/native-apis/

3

Whatis

4

TheBasicsofWebRTCWebReal-Time-Communication

New(ish)open-sourcedstandardfortransmittingdatafromonepeertoanother

Asetofframeworksandtoolsforcreatingconnectionsandsendingdata

Datatransmittedoverpeer-to-peerconnections

Agnostictotypesofdatasent◦ Audiobuffers◦ Videoframes◦ Rawdata◦ Files

5

ImportantComponentsandTermsPeerConnections– theactualconnectionbetweentwoclients

6

ImportantComponentsandTermsPeerConnections– theactualconnectionbetweentwoclients

STUN- SessionTraversalofUserDatagramProtocol[UDP]ThroughNetworkAddressTranslators[NATs]

7

ImportantComponentsandTermsPeerConnections– theactualconnectionbetweentwoclients

STUN- SessionTraversalofUserDatagramProtocol[UDP]ThroughNetworkAddressTranslators[NATs]

TURN– TraversalUsingRelaysaroundNAT

8

ImportantComponentsandTermsPeerConnections– theactualconnectionbetweentwoclients

STUN- SessionTraversalofUserDatagramProtocol[UDP]ThroughNetworkAddressTranslators[NATs]

TURN– TraversalUsingRelaysaroundNAT

ICE- InteractiveConnectivityEstablishment

9

VanillaIce

ImportantComponentsandTermsPeerConnections– theactualconnectionbetweentwoclients

STUN- SessionTraversalofUserDatagramProtocol[UDP]ThroughNetworkAddressTranslators[NATs]

TURN– TraversalUsingRelaysaroundNAT

ICE- InteractiveConnectivityEstablishment

Signaling– ameansofcommunicatingconnectioninfobetweenclients

10

ImportantComponentsandTermsPeerConnections– theactualconnectionbetweentwoclients

STUN- SessionTraversalofUserDatagramProtocol[UDP]ThroughNetworkAddressTranslators[NATs]

TURN– TraversalUsingRelaysaroundNAT

ICE- InteractiveConnectivityEstablishment

Signaling– ameansofcommunicatingconnectioninfobetweenclients

Codecs◦ Video

◦ VP8

◦ Audio◦ Opus

11

MakingPeerConnections

CreatingaP2PConnectionviaSTUN

13

CreatingaP2PConnectionviaSTUNCoversmostscenarios

Failswhenpeershave2incompatibleNATtypesorpeersrunthroughheavyfirewalls

14

CreatingaP2PConnectionviaSTUN+TURN

15

CreatingaP2PConnectionviaSTUN+TURNTURNserverrelaysdatabetweentwopeers

Meansnodirectpeerconnectionismade

Canhaveperformanceimplications

Canbeexpensive

Covers99.999%ofcases

16

Multi-connectionMeshModelEachclientcreatesaconnectiontoeveryotherclient

Cancreateunreliableconnections

Individualconnectionscanbedroppedwithoutdroppingthemall

Requiresextendedsignalingandaccountability

17

Multi-connectionMixerModelEachclientconnectstoasingleserver

Theservermixesaudioandvideostreamsperpeer

Theserverrelaysthemixedstreamstoeachpeer

Heavyliftingputonserverratherthandistributed

Expensive

Hardtoscale

18

UsingaConnection

19

SendingAudioandVideoDataGetUserMedia◦ Getsaccesstodevicehardware(micandcamera)

20

SendingAudioandVideoDataGetUserMedia◦ Getsaccesstodevicehardware(micandcamera)

Encodethemedia◦ Pickacodec◦ Encodethedatawiththatcodec

21

SendingAudioandVideoDataGetUserMedia◦ Getsaccesstodevicehardware(micandcamera)

Encodethemedia◦ Pickacodec◦ Encodethedatawiththatcodec

Sendencodedmediaoverconnection

22

SendingAudioandVideoDataGetUserMedia◦ Getsaccesstodevicehardware(micandcamera)

Encodethemedia◦ Pickacodec◦ Encodethedatawiththatcodec

Sendencodedmediaoverconnection

Switchmediasources(switchcamera,switchmic)

23

AdvancedFunctionalityScreenstreaming

Multi-sourcestreaming◦ Streamingmultiplevideostreamsfromoneclient◦ Streamingmultipleaudiostreamsfromoneclient

AudioMixing

Non-TURNrelaying

Recordingstreams

Distributedstreaming

Streamsignatures

24

GoingCross-Platform

25

PlatformsSupportedWeb*Windows◦ WPF◦ WindowsForms◦ Win8◦ UWP*

WindowsPhone◦ 8.0Silverlight+◦ 8.1RT*

AndroidiOSMac

26

WebSupport

27

28

WebWorkaroundsNpapi plugins

Chrome+Firefoxextensions

ActiveXplugins

JavaAppletplugins

Stilldoesn’tcatchall◦ EdgedoesnotsupportWebRTC (theyarepushingORTC)◦ EdgedoesnotsupportActiveXplugins,JavaApplets,ornpapi

29

NativePlatformSupportandIssuesWebRTC isbuiltonanopensourcedC++platform-agnosticframework◦ Youtheoreticallycanrunitanywhere!

Noteveryplatformsupportsgivencodecs

VP8isnotnativetoanyWindowsSDK!

WeneedtogetacustombuildofthenativeVP8(C++)codec

CompiletheVP8codecagainstVisualC++perplatform

SamewithOpus

UnlikewebbrowsersthatshipwithVP8andOpuscodecsaspartoftheapplication,weneedtoincludethelibrariesseparately

30

BuildingWindows◦ BuildVP8codec◦ BuildOpuscodec◦ WrapcallstoC++library

Android◦ IncludereferencetonativecodecsbuiltintoOS

iOS+Mac◦ Addofficiallibrariesfornativecodecs

Xamarin◦ Addbindingstonativelibrariesforcodecsandwrapcalls

31

IceLinkAlibraryandframeworkwrappingalotofnativeWebRTC

Languagesandplatformssupported◦ Android◦ iOS◦ Windows(WinForms,WPF,Win8,UWP)◦ WindowsPhone(8,8.1)◦ Web

◦ ActiveXcontrol◦ JavaApplet◦ Npapi intheworks

◦ Mac◦ Xamarin (Android,iOS,Mac)◦ Java

32

Demo