1

Next Stop, the Cloud: Understanding Modern Web Service

Deployment in EC2 and Azure

Keqiang He, Alexis Fisher, Liang Wang, Aaron Gember, Aditya Akella, Thomas Ristenpart

University of Wisconsin-Madison

2

PrivateDatacenters

Co-locationCenters

Move to IaaS clouds

3

Incomplete view of cloud use

Want a global, in-depth understanding of IaaS cloud usage patterns

4

Fundamental questions

1. Who is using public IaaS clouds?→Traffic patterns; network design/traffic engineering

2. How are these services using the cloud?→ Impact of failures; ways to improve availability→ Design of new systems/services

3. How can quality of experience be improved?→ Deployment design (e.g., ideal region/zone usage)

5

First in-depth empirical study examining IaaS cloud usage patterns and identifying ways tenants could

better leverage IaaS clouds

6

DNSServer

CDN

VM

IaaS cloud environments

VM

LBProxy

PaaSNode

VMPlatform-as-a-Service

(PaaS)

7

Datasets• University packet capture– Deep, but possibly atypical, perspective

• Alexa subdomains DNS records– Broad, high-level perspective

Top 1 million Subdomains

Zone transferDNS

records

1.4TB Capture

IP RangesConn

recordsCampusTraffic

8

Fundamental questions

1. Who is using public IaaS clouds?→Traffic patterns; network design/traffic engineering

2. How are these services using the cloud?→ Impact of failures; ways to improve availability→ Design of new systems/services

3. How can quality of experience be improved?→ Deployment design (e.g., ideal region/zone usage)

40,333

959,667

Cloud-using domains

Non-cloud-using domains

How many popular web services use the cloud?

9

94.2%

5.1% 0.7%Both

9. 13.29.35.36.

2 of 68

3 of 142

4 of 181

18 of 24

14 of 89

43% in top250K domains

Cloud availability and performance issues may impact over 4%

of popular web services

10

18.3%

81.7%

Traffic Volume (Bytes)

Which (locally) popular web services use the cloud?

1. 2. 3. 4. 5.

68.2%

1.7%

1.1%

0.7%

0.6%

1. 2. 3. 4. 5.

atdmt.com

s-msn.com

msecnd.net

3.1%

2.4%

2.3%

1.5%

1.4%

11

What type of content is served by(locally) popular web services?

24.2%

73.0%

0.1%

2.4% 0.3%Other (TCP) Other (UDP)

DNS (UDP)

HTTPS (TCP)

HTTP (TCP)

1. text/html

2. text/plain

3. image/jpeg

4. app/flash

5. app/octet-stream

24.1%

23.4%

10.6%

8.7%

7.8%

HTTP Content Type

Top 100 cloud-using domains account for

≈80% ≈100%

of HTTP flows

Compression could be employed to save WAN bandwidth

and improve delivery latency

12

Fundamental questions

1. Who is using public IaaS clouds?→Traffic patterns; network design/traffic engineering

2. How are these services using the cloud?→ Impact of failures; ways to improve availability→ Design of new systems/services

3. How can quality of experience be improved?→ Deployment design (e.g., ideal region/zone usage)

13

DNSServer

CDN

VM

IaaS cloud environments

VM

LBProxy

PaaSNode

VM

PaaS

14

VM ?

VM

VMLB

Proxy ?

Which front ends are used?

sub.abc.com A 72.44.32.1

sub.abc.com CNAME abc.elb.aws.comabc.elb.aws.com A 72.44.42.1

sub.abc.com CNAME proxy.heroku.com proxy.heroku.com A 72.44.62.1

?PaaSNode

DNSServer

VM

LBProxy

PaaSNode

String matching

on CNAMEDNS

records

VM front end

LB front end

PaaS front end

0% 20% 40% 60% 80%

72%

4%

8%

50%35%

15%

1 VM2 VMs

3+ VMs

97%

3%Elastic Beanstalk

Which front ends are used?

≈60% of ELB-using subdomains

resolve to 2+ ELB IPs≈4% of ELB IPs are shared

by 10+ subdomains15

There is limited uptake of value-added features

16

Which front ends are used?

9. 13.29.35.36. VM LB

LB PaaS

VM

LB PaaS

CDNFailures of value-added features

will impact the most popular EC2-using subdomains

17

Which locations are used?

DNS records

Match IP against

region ranges

18

How many and which regions are used?

521,681

116,366

40,548 VirginiaIrelandCaliforniaOregonSingaporeTokyoSao PauloSydney

19

How many and which regions are used?862

558

2071

1395

1035

1205

632502 Virginia

CaliforniaIllinoisTexasIrelandNetherlandsSingaporeHong Kong

How many and which regions are used?

92%

8%

1 region

2+ regions

97%

3%

1 region

2+ regions

9. 13.29.35.36.

1

2

1

1

2

7. 18.20.31.

3

5

1

5

Single region failures can take down a large fractionof cloud-using subdomains

21

How many and which availability zones are used?

• Latency measurements– Low latency => instances are in same zone– High coverage, low accuracy (noisy)

• IP address correlation– IPs are in the same /16

subnet => instances are in the same zone

– Higher accuracy, low coverage

22

How many and which availability zones are used?

33.2%

44.5%

22.3%

1 Zone

2 Zones

3+ Zones

9. 13.29.35.36.

4

5

1

3

5

Single zone failures can take down 16% of subdomainsand cripple many others

23

Fundamental questions

1. Who is using public IaaS clouds?→Traffic patterns; network design/traffic engineering

2. How are these services using the cloud?→ Impact of failures; ways to improve availability→ Design of new systems/services

3. How can quality of experience be improved?→ Deployment design (e.g., ideal region/zone usage)

Many popular web services are not deployed optimally for their users

24

Future measurements

• Use of other clouds (e.g., Rackspace)• Use of other EC2 and Azure features• Back end deployments• Packet captures from other vantage points

25

Summary

• Systematic method for studying IaaS cloud use• 4% of popular web services use EC2/Azure• VMs are the most popular EC2 front end, but

value-added features are used by top domains• Limited region and zone use (and diversity)

makes web services vulnerable to failures

26

Are web services deployed near users?

• Compare a domain’s customer country (according to Alexa) and region usage

• 47% of subdomains are not deployed in the same country as users

• 32% subdomains are not deployed on the same continent as users

27

What is the user perceived network performance for different regions/zones?

Regions should be selected based on users’ locations