10
Uncovering Twilio: Insights into Cloud Communication Services Ramnatthan Alagappan [email protected] Sourav Das [email protected] 1 Abstract Cloud communication service (CCS) with its simplicity and lower investment cost is becoming increasingly pop- ular. In contrast to its growing popularity, very little is known about the internals of CCS with respect to its ar- chitecture and protocols. To gain insights into CCS, we study a popular cloud communication service ”Twilio” us- ing gray box techniques. In our study, we provide insights into the Twilio ecosystem, its components, the interaction among components and the protocols. We also measure some guarantees provided by Twilio and show how the measurements fair against what is promised. Our analysis un-veils a number of interesting aspects about the Twilio ecosystem and have strong implications for developers who want to build applications on top of Twilio APIs. 2 Introduction CCS is an upcoming service model which provides so- phisticated APIs for enterprises to develop applications and offload communication related tasks from enterprise applications. All the communication services are offered through simple REST APIs for the applications to make use of them. Cloud communication services have a lot of advantages compared to on-premise hosted commu- nication infrastructure. Firstly, it offloads the burden of communication from the applications and separates it as a separate service. The applications can seamlessly inter- act with the communication APIs to accomplish complex communication tasks like sending promotional messages to customers, providing critical real-time information to mobile phones, etc. Secondly, the development effort in- volved in integrating an application with CCS is very less compared to developing and maintaining a home-brewed communication infrastructure. Thirdly, enterprises can build communication applications in a cost effective way because of the pay-per-use model provided by most the CC services. Our work focuses on study of such cloud communi- cation services. There are a lot of players in the market which provides CC services like Avaya, Clickatell, Plivo, etc. and we chose a popular CCS Twilio for our study pur- pose. Twilio is one of the prominent players in the CCS space and has a huge customer base which includes popu- lar enterprises like Coca-Cola, WalmartLabs, Intuit, Box. These enterprises use Twilio to accomplish a wide variety of tasks ranging from two-step authentication, powering lending machines with music, secure file sharing, deliver- ing deals and ads to mobile phones, etc. Twilio enables lot of new scenarios for businesses that were rather cumber- some to implement in the past. The APIs are simple and intuitive to understand and develop. Twilio also provides rich documentation and code examples to build simple ap- plications like VoIP communication. For our study, we developed a simple VoIP service atop Twilio APIs which we call as VoT (VoIP on Twilio). Cou- pled with logs collected at several places in the system we present a detail study of Twilio which we believe is some- what representative of CC service in general and is first of its kind. We give insights into the Twilio ecosystem, the high level as well as packet level protocol details and mea- surement of some guarantees that Twilio provides. We also discuss some interesting oddities that we found dur- ing the course of our study and point to some possible enhancements in the system. The reminder of the paper is organized as follows. In section 3, we motivate our study. In section 4, we provide a detailed analysis of the Twilio ecosystem, architecture of the system, our experimental setup, some of the scenar- ios that Twilio supports, high level protocols and packet level analysis for some key scenarios. We give detailed measurements with respect to call and message dequeu- ing rates in section 5. Then, we present some oddities that we found in the ecosystem in section 6. In section 7, we discuss some of the possible enhancements to the system. Finally we conclude by presenting the implications of our study and our future work. 3 Motivation and Related Work Businesses want to delegate communication from their applications and services because of the advantages pro- vided by CCS. The number of enterprises using CCS has seen a steady increase since its advent. Though there are not enough evidences that this pattern is going to continue, but because cloud communication services pro- vide an attractive cost-model and easy-to-use APIs, we strongly believe that this trend is going to continue in the future. As more and more enterprises start using CC ser- vices like Twilio, there is a good chance that this traffic may contribute to a good fraction of Internet traffic in the future. There have been lot of studies on VoIP services like Skype in the past [4]. There have been recent studies 1

Uncovering Twilio: Insights into Cloud Communication Servicespages.cs.wisc.edu/~ra/Twilio.pdf · Uncovering Twilio: Insights into Cloud Communication Services Ramnatthan Alagappan

Embed Size (px)

Citation preview

Page 1: Uncovering Twilio: Insights into Cloud Communication Servicespages.cs.wisc.edu/~ra/Twilio.pdf · Uncovering Twilio: Insights into Cloud Communication Services Ramnatthan Alagappan

Uncovering Twilio: Insights into Cloud Communication ServicesRamnatthan [email protected]

Sourav [email protected]

1 AbstractCloud communication service (CCS) with its simplicityand lower investment cost is becoming increasingly pop-ular. In contrast to its growing popularity, very little isknown about the internals of CCS with respect to its ar-chitecture and protocols. To gain insights into CCS, westudy a popular cloud communication service ”Twilio” us-ing gray box techniques. In our study, we provide insightsinto the Twilio ecosystem, its components, the interactionamong components and the protocols. We also measuresome guarantees provided by Twilio and show how themeasurements fair against what is promised. Our analysisun-veils a number of interesting aspects about the Twilioecosystem and have strong implications for developerswho want to build applications on top of Twilio APIs.

2 IntroductionCCS is an upcoming service model which provides so-phisticated APIs for enterprises to develop applicationsand offload communication related tasks from enterpriseapplications. All the communication services are offeredthrough simple REST APIs for the applications to makeuse of them. Cloud communication services have a lotof advantages compared to on-premise hosted commu-nication infrastructure. Firstly, it offloads the burden ofcommunication from the applications and separates it asa separate service. The applications can seamlessly inter-act with the communication APIs to accomplish complexcommunication tasks like sending promotional messagesto customers, providing critical real-time information tomobile phones, etc. Secondly, the development effort in-volved in integrating an application with CCS is very lesscompared to developing and maintaining a home-brewedcommunication infrastructure. Thirdly, enterprises canbuild communication applications in a cost effective waybecause of the pay-per-use model provided by most theCC services.

Our work focuses on study of such cloud communi-cation services. There are a lot of players in the marketwhich provides CC services like Avaya, Clickatell, Plivo,etc. and we chose a popular CCS Twilio for our study pur-pose. Twilio is one of the prominent players in the CCSspace and has a huge customer base which includes popu-lar enterprises like Coca-Cola, WalmartLabs, Intuit, Box.These enterprises use Twilio to accomplish a wide varietyof tasks ranging from two-step authentication, powering

lending machines with music, secure file sharing, deliver-ing deals and ads to mobile phones, etc. Twilio enables lotof new scenarios for businesses that were rather cumber-some to implement in the past. The APIs are simple andintuitive to understand and develop. Twilio also providesrich documentation and code examples to build simple ap-plications like VoIP communication.

For our study, we developed a simple VoIP service atopTwilio APIs which we call as VoT (VoIP on Twilio). Cou-pled with logs collected at several places in the system wepresent a detail study of Twilio which we believe is some-what representative of CC service in general and is first ofits kind. We give insights into the Twilio ecosystem, thehigh level as well as packet level protocol details and mea-surement of some guarantees that Twilio provides. Wealso discuss some interesting oddities that we found dur-ing the course of our study and point to some possibleenhancements in the system.

The reminder of the paper is organized as follows. Insection 3, we motivate our study. In section 4, we providea detailed analysis of the Twilio ecosystem, architecture ofthe system, our experimental setup, some of the scenar-ios that Twilio supports, high level protocols and packetlevel analysis for some key scenarios. We give detailedmeasurements with respect to call and message dequeu-ing rates in section 5. Then, we present some oddities thatwe found in the ecosystem in section 6. In section 7, wediscuss some of the possible enhancements to the system.Finally we conclude by presenting the implications of ourstudy and our future work.

3 Motivation and Related WorkBusinesses want to delegate communication from theirapplications and services because of the advantages pro-vided by CCS. The number of enterprises using CCS hasseen a steady increase since its advent. Though thereare not enough evidences that this pattern is going tocontinue, but because cloud communication services pro-vide an attractive cost-model and easy-to-use APIs, westrongly believe that this trend is going to continue in thefuture. As more and more enterprises start using CC ser-vices like Twilio, there is a good chance that this trafficmay contribute to a good fraction of Internet traffic in thefuture.

There have been lot of studies on VoIP services likeSkype in the past [4]. There have been recent studies

1

Page 2: Uncovering Twilio: Insights into Cloud Communication Servicespages.cs.wisc.edu/~ra/Twilio.pdf · Uncovering Twilio: Insights into Cloud Communication Services Ramnatthan Alagappan

on cloud storage services like Dropbox [3]. Best to ourknowledge, we are the first to study cloud communica-tion services. [6] provides a thorough characterizationof the Dropbox protocol and traffic patterns. They pro-vide insights into how traffic to Dropbox varies across fourdifferent networks including home and campus networks.Our study on the other hand does not deal with traffic anal-ysis since we did not have the sophistication of collect-ing the packet traces on the campus network. Our studyaims at studying the architecture and protocols of the en-tire Twilio ecosystem. Driven by our study, we also aim atexploring some possible enhancements to the entire Twilioecosystem. [4] provides a detailed packet level study ofthe Skype protocol. The authors also provide deep insightsinto the Skype architecture. Our study involves studyingthe architecture of the system and the protocols involvedusing a suite of gray box tools that we have developed.

4 Twilio OverviewIn what follows, we describe the Twilio Ecosystem, someof the possible scenarios using the Twilio APIs, the ex-perimental setup that we used for our study, high levelprotocol study and then packet level protocol study.

4.1 Twilio EcosystemThe Twilio ecosystem, as depicted in the Figure 1 can beviewed as a layered architecture. In the bottom most layerlie the Twilio servers. These servers lay the foundationof this ecosystem by exposing a set of data and voice com-munication APIs for sending and receiving voice calls andmessages.

In the middle layer, lie the Application servers.These servers are installed by Twilio customers with ded-icated Twilio accounts. For e.g. These application serversmight belong to some company X that wants to provideVoIP service to its customer. Each Twilio account islinked to 1) one or more Twilio numbers, 2) an AccountSID and 3) an Auth Token. A Twilio number is a tendigit phone number. Account SID is an unique identi-fier for a Twilio account. Auth Token is a token used byTwilio Servers to authenticate an account. The Applica-tion server uses the Account SID and Auth Token to ac-cess the Twilio APIs. The cost model for this second layerApplication servers (or Twilio customers) is typically payper message or pay per call.

In the top most layer, lie the Clients which could bebrowser based clients or phone based clients. An impor-tant thing to note here is that these Clients are not thedirect customers of Twilio, instead they are the customersof the services that are built on top of Twilio. For e.g.These clients could be the customers of company X thatis providing VoIP service. These clients, can be free cus-tomers or paid customers of company X. Also, depend-ing on the type of service that the layer two based Twilio

Figure 1: Twilio Ecosystem

Figure 2: Sample TwiML Snippet

customers wants to provide, it is possible that these layerthree is non-existent. The Clients when present, in orderto use the services and communicate with each other needto register with the layer three Twilio servers. This is doneusing a capability token which is provided by the Appli-cation Server.

So far we have only discussed how the Twilio clientsconnect to each other in the Twilio ecosystem. The Twilioclients and the Application servers can also interact withthe external phones belonging to different carriers via theTwilio servers. Hence, these external phones also form apart of the Twilio ecosystem.

Application Container: Each Twilio number is linkedto an application container. The application containercontains two URLs: V oiceURL and MessageURL.These URLs are configurable and are usually configuredby the Application server administrators (or Twilio cus-tomers). Whenever an incoming call or message for aTwilio number arrives, the Twilio server makes post re-quest to these URLs. The content generated by theseURLs direct Twilio servers to perform the needed actionson the incoming calls or messages. These contents arein form of a special markup language called TwiML de-scribed in the subsequent section. Additionally, Twilioappends the query parameters that it obtains from theend-clients before making the requests to the applicationservers.

TwiML: It is a markup language developed by Twilio.

2

Page 3: Uncovering Twilio: Insights into Cloud Communication Servicespages.cs.wisc.edu/~ra/Twilio.pdf · Uncovering Twilio: Insights into Cloud Communication Services Ramnatthan Alagappan

Script Twilio AppServer Phone

Queue Call (#, url)

Async ACKDequeue and place call

On attend

Fetch TwiML

TwiML response

Parse TwiML and Stream content

Figure 3: Automated call to a phone

It contains a set of simple verbs that can be used by theApplication Servers to direct the Twilio servers about theaction that needs to be taken whenever a call or messageis received to its number. Various kinds of verbs are sup-ported as shown below which can be used to create inter-active applications atop Twilio.

• Say - Read text to the caller

• Play - Play an audio file for the caller

• Dial - Add another party to the call

• Record - Record the caller’s voice

• Gather - Collect digits the caller types on their key-pad

• Sms - Send an SMS message during a phone call

• Hangup - Hang up the call

• Queue - Add the caller to a queue of callers.

• Redirect - Redirect call flow to a different TwiMLdocument

• Pause - Wait before executing more instructions

• Reject - Decline an incoming call without beingbilled

For e.g. The TwiML snippet shown in Figure 2 willsay Hello World (dictated by the verb ”say”) to the callerin female voice (dictated by the attribute ”voice”). Ourexperience with TwiML suggests that it is very simple to

use and powerful with respect to the diversity of verbs thatit supports.

4.2 ScenariosThere are multiple scenarios that one can enable with thehelp of Twilio Apis. Some of them are:

• Automated calls : Twilio APIs can be used to placeautomated calls to phone numbers to deliver a pre-recorded message.

• V oice calls : VoIP applications can be built atopTwilio voice APIs which can be used to place voicecalls. There are three scenarios possible for VoIP ap-plications:

– Call between VoIP Clients

– Call from Phone to VoIP Client

– Call from VoIP Client to Phone

• Messages : Twilio message APIs can be used tosend automated messages.

Voice calls between VoIP clients:

4.3 Experimental SetupFor our study, we developed a simple VoIP service (calledVoT) atop Twilio and deployed it on OpenShift RedHatCloud. The VoT server also hosts a simple web interfacewhich can be used to place voice calls by specifying aphone number or the registered names of VoIP clients. Formaking automated calls and sending automated messages

3

Page 4: Uncovering Twilio: Insights into Cloud Communication Servicespages.cs.wisc.edu/~ra/Twilio.pdf · Uncovering Twilio: Insights into Cloud Communication Services Ramnatthan Alagappan

Mike Twilio AppServer Jenny

Ask for Capability TokenAsk for Capability Token

Capability token Capability Token

Call Jenny

Trigger incoming call

Fetch TwiML(Jenny)

TwiML Response

Parse TwiML - Call Jenny

Actual VoIP communication starts

Reg with capability token Reg with capability token

Figure 4: Call between two VoIP clients

we have developed python scripts using the Twilio clientlibraries which directly calls into the Twilio APIs. Forour study, we extensively use traces collected at differ-ent places. This includes application level trace messagescollected at our VoT server, logs in Twilio user portal,Twilio object store queries and Wireshark traces collectedat client machines.

4.4 High Level Protocol StudyWe now present a high level protocol study of some of theimportant scenarios that Twilio supports.

Automated calls to phones: The protocol diagram forthis scenario is shown in Figure 3. At first, a script (ourpython script) queues a call request specifying the phonenumber to which the call needs to be placed and an url in-dicating the location of the TwiML response. The Twilioserver acknowledges the request and places the call. Oncethe phone attends the call, the Twilio server does a HTTPPOST or HTTP GET to the url in the call queue request tofetch the TwiML response. In our experiments, we hostedthese pre-recorded messages in our application servicer.The application server processes the request and sends aTwiML response in return. The Twilio server parses theTwiML response and finally streams the content to thephone.

Voice calls from phone to VoIP clients: The proto-col diagram for this scenario is shown in Figure 4. Let ussay, Mike and Jenny are the two VoIP clients and Mikewishes to call Jenny. As shown in Section 4.1, for Mikeand Jenny to communicate they need to first register with

the Twilio Server. Hence, Mike and Jenny at first requestthe Application server for capability tokens. Once the Ap-plication server delivers the tokens, Mike and Jenny reg-ister with the Twilio server using the tokens. Mike thenqueues a call to Jenny. An interesting thing to note hereis that the Twilio server does not have information aboutwhat to do with the call. It need to communicate withthe Application server to get this information. For this, ittriggers an incoming call to the Twilio number to whichMike is linked to. As stated in Section 4.1, every incom-ing call to a Twilio number generates a post request on thevoice URL present in the application container linked tothat number. This post request is directed to the Applica-tion server with query parameter as ”Jenny”. The Appli-cation server parses the request and generates the appro-priate TwiML content indicating the action that needs tobe taken. In this case, the application server would servea response that contains a Dial verb and the client for theDial verb would be ”Jenny”. The Twilio Server parses theTwiML and places a call to Jenny. Finally, Mike is noti-fied about this and the voice communication starts. Notethat the actual voice communication is not peer-to-peerand is routed via Twilio servers.

Voice calls from VoIP clients to phone: The proto-col diagram for this scenario is not shown for space con-straints and is similar to that shown in Figure 4 except thatJenny has a dedicated phone number and hence does notneed to register with the Twilio server. Lets say, Mike is aVoIP client and Mike wishes to call Jenny with a dedicatedphone number (may belong to any carrier). Mike will at

4

Page 5: Uncovering Twilio: Insights into Cloud Communication Servicespages.cs.wisc.edu/~ra/Twilio.pdf · Uncovering Twilio: Insights into Cloud Communication Services Ramnatthan Alagappan

Mike Twilio AppServer Phone

Ask for Capability Token

Capability Token

Fetch TwiML

TwiML response

Call Twilio number

Trigger incoming call

Call Mike

Actual communication starts

Reg with Capability Token

Figure 5: Call from phone to a VoIP client

first request the Application server for a capability token.Once the Application server delivers the token, Mike willregister with the Twilio server. Mike then places a callto Jenny’s number. As in the case of two VoIP clients inthe previous section, the Twilio server communicates withthe Application server to get the information about the ac-tion that needs to be taken for the queued call. For this,it triggers an incoming call to the Twilio number linkedwith Mike. This incoming call generates a post requeston the voice URL present in the application container.This post request is directed to the Application server withquery parameter as ”Jenny’s phone number”. The Appli-cation server parses the request, generates the appropriateTwiML content for the action that needs to be taken anddelivers it to the Twilio server. The Twilio Server parsesthe TwiML and places a call to Jenny’s phone. Finally,Mike is notified about this and the actual communicationstarts.

The protocol diagram for this scenario is shown in Fig-ure 5. Lets say, Mike is a VoIP client and a phone wish tocall Mike. Mike will first request the Application serverfor capability token. Once the Application server deliversthe token, Mike registers with the Twilio server. Whenthe phone calls the Twilio number associated with Mike,the Twilio server triggers an incoming call to that Twilionumber. The incoming call generates a post request on thevoice URL present in the application container. This postrequest is directed to the Application server. The Appli-

cation server parses the request, generates the appropriateTwiML content for the action that needs to be taken anddelivers it to the Twilio server. The Twilio Server parsesthe TwiML and places a call to Mike. Finally, the phoneis notified about this and the actual communication starts.

4.5 Packet Level AnalysisIn this subsection, we dig deeper and present the packetlevel analysis for a scenario where a browser makes avoice call to a phone or another browser based client.

Figure 6 shows the sequence of events that happenswhen a browser based client wants to make a VoIP callto another browser based client or a phone. As shown, atfirst the client browser does a HTTP GET request to theapplication server that we have deployed. The applicationserver generates a capability token and passes it onto theclient. The client side javascript includes the twilio js li-brary and does Twilio.Device.Setup to register its capabil-ity with the Twilio servers. The sequence of messages forthe second block is also shown. The client passes the to-ken as HTTP data after establishing a TLS session with theserver. After this step, the client is free to make a call toeither another browser based client or another phone num-ber. The actual voice communication is tunneled throughTwilio servers and it involves RTMP protocol. This se-quence of packet exchanges are not shown due to spaceconstraints.

5

Page 6: Uncovering Twilio: Insights into Cloud Communication Servicespages.cs.wisc.edu/~ra/Twilio.pdf · Uncovering Twilio: Insights into Cloud Communication Services Ramnatthan Alagappan

Clie

nt T

wili

o A

uth

Ser

ver

Client Capability Token Registration

S A CH A A

CKE, CCS, EH D

SA A SH C,SHD CCS, EH

D

LegendS-TCP SYN, A-ACK,CH-TLS Client Hello,CKE-TLS Client Key Exchange, EH-EncryptedHandshake, D-Data, FA-TCP FIN ACK, SH-Server Hello, C-Certificate,SHD-Server Hello Done, CCS-Change Cipher Spec

Http Get to VoT Server

Clie

nt V

oT S

erve

r

S A H/G A

SA A H/R

200

OK,

HTM

L/JS

Htt

p G

et to

VoT

Se

rver

Cap

abili

ty

Toke

n Re

g

Client Browser

Client Browser

VoT Server

Twilio Auth Server

Voi

ce S

essi

on

Client Browser

Twilio Voice Server

Time

Figure 6: Voice Call from Browser Client The figure shows sequence of packet exchanges that happen between the client and other

components of the system. The top diagram shows the high level operations. The participants are shown above and below the top and bottom lines.

The left bottom figure shows how the client interacts with the application server to obtain the capability token and it corresponds to the first block

in the top diagram. The right bottom diagram shows the sequence of messages exchanged between the client and the Twilio server when the client

executes the Twilio.Device.Setup method and corresponds to the second block in the top diagram.

5 MeasurementsIn this section, we provide measurements with respect tocall and message dequeuing rates. Twilio provides guaran-tees that the placed calls and messages will be dequeuedat a rate of 1 per sec and placed onto the phones or thebrowser clients [1]. We measure the degree to which thisguarantee is met. All the experiments were conducted onLenovo W530 laptop with 8 GB RAM running on 4 coresconnected to a 30 Mbps Internet link.

5.1 CallsWe developed scripts that can queue automated calls toUS phones and online browser clients. We queue the firstcall to the browser client and queue a variable numberof calls to a US phone and then finally place one morecall to the browser client. Twilio dequeues the calls in theorder it was placed into the queue. Using Wireshark inthe browser client, we collect the packet traces and thetimestamps of the incoming call connections. We thencan calculate the time difference between the first call andthe second call received by the browser. A careful readerwould note that the call timestamps may not reflect the ac-tual time when the call was dequeued from the queue and

Figure 7: Calls queuing and dequeuing. The figure shows

the queuing and dequeuing rate for calls

it will include the network latency involved in placing thecall to the browser client. We noticed that this latency waslesser than 50 ms and so was discounted for the purposeof our calculations.

Figure 7 shows the expected dequeue rate, observed de-queue rate and queue rate for calls. The horizontal axisshows increasing number of calls that we place to thephone and the vertical axis shows the queue or dequeuerate. As mentioned before Twilio gives a guarantee to de-

6

Page 7: Uncovering Twilio: Insights into Cloud Communication Servicespages.cs.wisc.edu/~ra/Twilio.pdf · Uncovering Twilio: Insights into Cloud Communication Services Ramnatthan Alagappan

Figure 8: SMS queuing and dequeuing. The figure shows

the queuing and dequeuing rate for messages

queue calls at the rate of 1 per second. This is shown bythe flat line. We define the queue rate as the rate at whichthe client library can place calls into the Twilio call queue.It is quite possible to achieve higher values of queue ratewith high speed links. This value does not necessarily im-ply that the Twilio server has a throttling mechanism foradding calls into the queue. The Dequeue rate line showsthe observed dequeue rate in our experiments. We noticethat for a small number of calls such as 100, Twilio de-queuing mechanism performs really well than expected.As the number of calls increases, the dequeue rate dropsslightly below 1 but stays between 0.9 and 0.95. We be-lieve that this behavior is completely acceptable if furtherincreasing the number of calls to say few thousands stilldoes not reduce the dequeue rate.

Summary.Twilio call dequeuing guarantees are met andexceeded for small number of calls like 100. The dequeuerate slightly falls below 1 as the number of calls increasesfrom 100. If applications need very critical and real timedata to be delivered to phones or clients, we believe it isadvisable to use one Twilio number for say around 100outgoing client/phone connections. For applications thatdo not need this level of delivery accuracy, we believe thisbehavior is completely acceptable.

5.2 MessagesWe developed scripts that can queue automated messagesto US phones. We queue 100 messages with varying mes-sage sizes to a sinlge US phone number. Twilio dequeuesthe messages in FIFO order similar to calls. We noticedthat telephone networks take a very variable amount oftime to deliver messages to phones. So, for calculatingthe dequeuing rate, we used the timestamps present in themessage resource in the Twilio object store. We querythese message resources using the REST APIs and arriveat the dequeue rate by looking at the sent timestamp in themessage resource.

Figure 8 shows the expected dequeue rate, observed de-queue rate and queue rate for SMS. The horizontal axisshows increasing size of the messages that we place to the

phone and the vertical axis shows the queue or dequeuerate. As mentioned before Twilio gives a guarantee to de-queue SMS at the rate of 1 per second. As mentionedearlier in 5.1, the queue rate shows the rate at which theclient can push messages into the queue. It is interesting tonote that the expected rate itself drops as the message sizeincreases. Twilio treats a single message as just 160 bytes.If the message contains say 170 bytes, Twilio considersthis as two messages and so it can queue at the rate of onemessage per two seconds and so the expected rate falls to0.5 from 1. Similarly for a 400 byte message, there arethree such chunks and so the expected rate drops to 0.33.It can be noted that the observed rate closely follows theexpected rate.

Summary.Twilio message dequeuing guarantees arewell maintained. An interesting observation is that naivedevelopers who do not notice the size limits of the mes-sages may wrongly believe that Twilio can dequeue at arate of 1 message per second irrespective of the messagesize. We bring this out clearly in our study by showingthat the expected rate itself falls down as the message sizeincreases.

6 OdditiesWe now present some of the oddities in the Twilio ecosys-tem that we discovered during our study. First, as dis-cussed in section 4, Twilio charges twice for making asingle outgoing call. We discovered this from the call logsavailable in the Twilio user portal. On first seeing this, wethought this was a problem in the VoIP service code thatwe developed. But it turns out that even after using thecode examples from Twilio developer forums, we werestill observing this behavior. It should be noted that webuilt our VoIP service on top of a single Twilio number.Second, we noticed that some applications may requireordered message delivery for the messages that they tryto send through Twilio APIs. For example, a messagebased query system will require that the messages sentare delivered to the mobile phone in the same order. Weunderstand that the telephone service provider can also re-order messages when delivering them. We understand thatTwilio does not provide any guarantee as such for orderedmessage delivery. But we wanted to measure to what ex-tent the messages can be reordered.

Figure 9 shows the message re-orderings that can hap-pen. It is understandable that Twilio does not itself pro-vide ordered message delivery. Application developerswho develop on the Twilio platform should be aware ofthis and should use suitable application level techniquesto solve this issue. For example, applications can use a la-bel denoting the position of the message in the sequence.

Third, we injected few faults into some parts of thesystem and observed how Twilio reacts to these cornercases. For example we tested cases where in the TwiML

7

Page 8: Uncovering Twilio: Insights into Cloud Communication Servicespages.cs.wisc.edu/~ra/Twilio.pdf · Uncovering Twilio: Insights into Cloud Communication Services Ramnatthan Alagappan

Exp

ecte

d1

byt

e20

0 b

yte

4th message was received 6th4

6

1 100

1 100

Sent order

Received order

600

byt

e

Figure 9: SMS reordering The figure shows the extent to which messages can get reordered. The top line in each strip denotes the sent

order and the bottom line denotes the received order. The top most strip shows the ideal message delivery behavior. The second strip shows the

re-orderings that happened when 100 one byte messages were sent. The third strip shows the same for 200 byte messages and the last strip for 600

byte messages. Note that the sent time is identified from the sent timestamp from the resource store so it represents the time for nearest upstream

telephone carrier network to acknowledge the message was received and will be sent to the phone.

response returned by our VoIP server was malformed, un-acceptably large, corrupted etc. We noted that the faulthandling mechanisms were not uniform across all errorscenarios. For example, if the TwiML Say verb containedcontent greater than 4096 bytes, the call was placed but anerror message was delivered to the attendee. On a differ-ent scenario when there are nested Say verbs in the TwiMLresponse, then a blank call was placed to the attendee. Weare barely scratching the surface in terms of fault injec-tion. We strongly believe that much better extensive tech-niques can be developed to analyze all the possible errorconditions in the service.

Fourth, during our study we observed a weird behav-ior in the Twilio Python client v3.66 library. We observedthat when a client script tried to queue a call or a mes-sage using the create() API, there was a HTTP level authfailure that was happening before the actual sequence ofpackets that were happening between the client and theTwilio server. On further investigation and debugging, we

found that the client was not passing the Auth Token andAccount SID when the first REST API call is made to theserver. Then the exception is handled and then the re-quired credentials are passed onto the server for the sub-sequent REST API calls. We noted that this unwantedauth failure wastes 6 RTTs. We also believe that this canbe easily optimized in the client library.

Figure 10 explains this client library behavior.

7 DiscussionWe now discuss some possible enhancements to the entireTwilio ecosystem.

Client Library Improvements: We showed that theclient library can be optimized to save few RTTs in theprevious section. It is important to notice that we iden-tified this just by manually studying the behavior of theclient library. We believe that there should be more ro-bust and extensive ways to study the client’s interactionwith the server. We also showed in the previous section

8

Page 9: Uncovering Twilio: Insights into Cloud Communication Servicespages.cs.wisc.edu/~ra/Twilio.pdf · Uncovering Twilio: Insights into Cloud Communication Services Ramnatthan Alagappan

Clie

ntTw

ilio

Serv

erC

lient

Tw

ilio

Ser

ver Aut

h Fa

ilure

Auth Failure

S

SA

A CH

A SH

A

Send SMS

C,SHD

ACKE, CCS, EH

CCS, EH

D

D FA

FA

A

Http

401

S A CH A ACKE, CCS, EH D FA

SA A SH C,SHD CCS, EH D FA

A

LegendS-TCP SYN, A-ACK,CH-TLS Client Hello,CKE-TLS Client Key Exchange, EH-EncryptedHandshake, D-Data, FA-TCP FIN ACK, SH-Server Hello, C-Certificate,SHD-Server Hello Done, CCS-Change Cipher Spec

Time

6 RTTs wasted!

Figure 10: Packet analysis for creating automated SMS The figure shows the sequence of packets exchanged between the client

and the server when the client script tries to create a message.

that fault handling mechanisms are not uniform across thesystem spanning the service and the client library. Also,in most of the cases that we tested by inducing faults fromthe caller side, we found that the callee is notified of theerror conditions instead of caller. We believe that, in suchscenarios it is best to inform caller with consistent errornotifications since it is the caller who can take actionsagainst those error and correct the errors.

Security: We found that presently there is no wayfor Application servers to authenticate the Twilio serverswhen the later makes the HTTP POST requests. In theinterest of securing the application servers from attacks,we believe Twilio may provide IP whitelists which can beused by the Application servers to authenticate requests.Even IP whitelists might not be sufficient for scenariosin which Application servers are located behind proxiesand the IP from which request originates gets masked (e.g.application server hosted behind load balancer in PaaS ar-chitecture). For such scenarios some more robust securitymeasures needs to be provided.

Intelligent TwiML fetching: As we have shown in Sec-tion 4.4, in case of automated calls to phone, Twilio server

fetches the TwiML only after placing the call. We un-derstand that this technique saves extra RTTs in case thecall is rejected by the phone. But from the perspective ofa callee and assuming that most calls will be answered,we believe it is best to prefetch TwiML before placingthe call. Also, we see that in the current automated callmodel the application server is contacted each time a callis placed. One of the most popular use of automated callis to send/broadcast a single message to multiple numbers(e.g. to send promotional messages or alerts). In suchscenarios, it is wasteful to flood the application server tofetch the same content multiple times both from Appli-cation server as well as Twilio server point of view. Tohandle such scenario Twilio might provide some specialoptions using which the application server can indicatethat the same message has to be used for a specified num-ber of calls or for some specified period of time and theTwilio server can then fetch the request once and cache it.

9

Page 10: Uncovering Twilio: Insights into Cloud Communication Servicespages.cs.wisc.edu/~ra/Twilio.pdf · Uncovering Twilio: Insights into Cloud Communication Services Ramnatthan Alagappan

8 Future Work and ConclusionsIn this section we discuss some of the possible avenuesto extend this work, then present the implications of ourstudy and finally conclude. As a future work, we think tostudy other CC services and compare different servicesin terms of traffic characters, usage patterns, user basediversity etc. We also intend to develop a much moregeneric and sophisticated gray-box framework to studyCC services. Our current implementation of the toolboxis highly specific to Twilio. We also started looking at thesecurity aspects of the Twilio servers by doing port scansusing nmap. From our initial analysis, the servers haveonly ports 80 and 443 open. As a future work, we wouldlike to analyse if some impersonation attacks are possiblewith AuthTokens. We also want to study how misbehav-ing clients can attack or cause some form of interferencein the system.

Our study has implications for both application devel-opers and Twilio developers. First, we showed differentcomponents of the Twilio ecosystem and how they inter-act. Our study gives more insights for developers to buildrobust applications atop Twilio platform. We also showedhow Twilio meets its guarantees in terms of call and mes-sage dequeuing rates. This gives the application develop-ers a good sense of what to expect from the Twilio service.Second, we should some possible improvements to Twilioin sections 6 and section 7.

To conclude, we developed a simple VoIP service atopTwilio platform and a graybox toolset to gain insights in tothe protocols and the architecture of the system. We alsoempirically measured call and message queue/dequeuerates. We showed some interesting oddities in the serviceand the client library. We also pointed out to some pos-sible enhancements to the Twilio ecosystem. We showedthat our study has implications for both application devel-opers and Twilio developers. Our study is a small steptowards studying the rapidly growing Cloud Communi-cation Services arena and we strongly believe further re-search is required in exploring this area.

References[1] Twilio Call Rate Limit. https://www.twilio.com/help/faq/twilio-

basics/what-are-the-limits-on-outbound-calls-and-sms-messages-per-second.

[2] Andrea C. Arpaci-Dusseau and Remzi H. Arpaci-Dusseau. Infor-mation and Control in Gray-Box Systems. In Proceedings of the18th ACM Symposium on Operating Systems Principles (SOSP ’01),pages 43–56, Banff, Canada, October 2001.

[3] Paul Barford and Mark Crovella. Critical path analysis of tcp trans-actions. SIGCOMM Comput. Commun. Rev., 31(2 supplement):80–102, April 2001.

[4] Salman A. Baset and Henning Schulzrinne. An analysis of the skypepeer-to-peer internet telephony protocol. 2004.

[5] Nathan C. Burnett, John Bent, Andrea C. Arpaci-Dusseau, andRemzi H. Arpaci-Dusseau. Exploiting Gray-Box Knowledge ofBuffer-Cache Contents. In The Proceedings of the USENIX AnnualTechnical Conference (USENIX ’02), pages 29–44, Monterey, CA,June 2002.

[6] Idilio Drago, Marco Mellia, Maurizio M. Munafo, Anna Sperotto,Ramin Sadre, and Aiko Pras. Inside dropbox: Understanding per-sonal cloud storage services. In Proceedings of the 2012 ACM Con-ference on Internet Measurement Conference, IMC ’12, pages 481–494, New York, NY, USA, 2012. ACM.

[7] James Nugent, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. Controlling your PLACE in the File System with Gray-box Techniques. In Proceedings of the USENIX Annual Techni-cal Conference (USENIX ’03), pages 311–324, San Antonio, Texas,June 2003.

10