http://comsys.rwth-aachen.de/

Characterizing aMeta-CDN

Oliver Hohlfeld, Jan Rüth, Konrad Wolsing, Torsten Zimmermann

Berlin / PAM 2018

2Torsten Zimmermann

https://netray.io

Motivation - What is a Meta-CDN?

� Content Delivery Networks� Key component in the Internet, high availability and low latency

� But is one CDN enough?� Varying performance, cost model (quota, time depending)

origin

3Torsten Zimmermann

https://netray.io

Motivation - What is a Meta-CDN?

� Content Multihoming� Distribute content via multiple CDNs� Select CDN with best performance and/or low costs

� Meta-CDN� Multi-CDN, CDN selector or broker� Offers request routing service

¾ E.g., Conviva for video delivery

� User Perspective� Higher availability and (possibly) improved performance

� CDN Perspective� Challenge for business model à traffic harder to predict

C

C

C

C

4Torsten Zimmermann

https://netray.io

Detection - Why Cedexis?

� DNS measurements at our institute� Stability of mappings

dig www.download.windowsupdate.com...;; ANSWER SECTION:www.download.windowsupdate.com. 2853 IN CNAME 2-01-3cf7-0009.cdx.cedexis.net.2-01-3cf7-0009.cdx.cedexis.net. 239 IN CNAME fg.download.windowsupdate.com.c.footprint.net.fg.download.windowsupdate.com.c.footprint.net. 229 IN A 8.253.208.112fg.download.windowsupdate.com.c.footprint.net. 229 IN A 8.248.99.254...;; SERVER: 8.8.8.8#53(8.8.8.8);; WHEN: Thu Mar 08 11:05:08 CET 2018

dig www.download.windowsupdate.com...;; ANSWER SECTION:www.download.windowsupdate.com. 3395 IN CNAME 2-01-3cf7-0009.cdx.cedexis.net.2-01-3cf7-0009.cdx.cedexis.net. 239 IN CNAME download.windowsupdate.com.edgesuite.net.download.windowsupdate.com.edgesuite.net. 831 IN CNAME a767.dspw65.akamai.net.a767.dspw65.akamai.net. 19 IN A 2.21.242.189a767.dspw65.akamai.net. 19 IN A 2.21.242.196...;; SERVER: 8.8.8.8#53(8.8.8.8);; WHEN: Thu Mar 08 11:05:27 CET 2018

� Characterize a Meta-CDN� Operational principle, global view on infrastructure and customers

5Torsten Zimmermann

https://netray.io

Cedexis - Operational Principle

� DNS-based redirection� Transfer requesting user via CNAME to authoritative nameserver

� Cedexis CNAME structure� 2-01-3cf7-0009.cdx.cedexis.net

� Customer can configure different profiles (applications)¾ E.g, images.domain.tld, download.domain.tld

dig www.download.windowsupdate.com...;; ANSWER SECTION:www.download.windowsupdate.com. 3395 IN CNAME 2-01-3cf7-0009.cdx.cedexis.net.2-01-3cf7-0009.cdx.cedexis.net. 239 IN CNAME download.windowsupdate.com.edgesuite.net.download.windowsupdate.com.edgesuite.net. 831 IN CNAME a767.dspw65.akamai.net.a767.dspw65.akamai.net. 19 IN A 2.21.242.189a767.dspw65.akamai.net. 19 IN A 2.21.242.196...

CID APPID

6Torsten Zimmermann

https://netray.io

Cedexis - Operational Principle

� CDN selection options� Cedexis offers three components: Radar, Fusion, Openmix

� Radar� Community-driven measurements via embedded JavaScript

¾ Once page is loaded, requests probe instructions from Cedexis

� Private (own performance) and Community probes (measure others)¾ Results reported back to Cedexis

� Fusion� Gather statistics from CDNs itself, e.g., traffic share, quota, or performance

� Openmix� Enables customers to program DNS resolution � Based on Radar and Fusion measurements

7Torsten Zimmermann

https://netray.io

Cedexis - Operational Principle

� CDN selection options� Cedexis offers three components: Radar, Fusion, Openmix

� Radar� Community-driven measurements via embedded JavaScript

¾ Once page is loaded, requests probe instructions from Cedexis

� Private (own performance) and Community probes (measure others)¾ Results reported back to Cedexis

� Fusion� Gather statistics from CDNs itself, e.g., traffic share, quota, or performance

� Openmix� Enables customers to program DNS resolution � Based on Radar and Fusion measurements

Dynamic traffic routing,thus harder to predict for CDNs

8Torsten Zimmermann

https://netray.io

Cedexis Customers - Measurement

� Via DNS resolutions and Domain lists� Enumerating 2-01-CID-APPID.cdx.cedexis.net (limit APPID to 256 à 16.8M)� Resolve A-records based on domain lists à CNAME *.cedexis.net

¾ E.g., Alexa 1M, .com/.net/.org, .fi/.se/.nu

� (CID,APPID) tuples� 84% via enumeration, 11.2% enum. and domain lists, 4.8% only in lists� 55 (20) customers with 1 (2) apps, 1 customer with 84 apps� 62.7% of (CID,APPID) behind a single domain

¾ 31 (6) APPIDs managing more than 10 (100) domains

2016-0

2

2016-0

4

2016-0

6

2016-0

8

2016-1

0

2016-1

2

2017-0

2

2017-0

4

2017-0

6

2017-0

8

2017-1

0

Time

40

80

120

160

200

240

280

Dom

ains

DDoS Attack, 10th May ’17

Outages of our probe

OpenIntel Own measurements

Domains utilizing Cedexis on the Alexa 1M list

9Torsten Zimmermann

https://netray.io

Cedexis Customers - Classification

� Manual Classification and evolving set of categories� Automated Alexa Web Information Service not practical

¾ Classifications only exist for 17% of domains¾ Errors, e.g., secure-m.ibis.com à …/Materials_and_Supplies/Wafer

Service Share

Web 62.7 %

Unknown 15.6 %

Assets 12.9 %

Media 5.4 %

API 2.3 %

Bulk 1.1 %

Type Share

Business 17.7 %

IT 12.1 %

News 11.3 %

Gambling 11.3 %

Shopping 8.1 %

Games 8.1 %

Unkown 8.1 %

Goods 5.6 %

Automotive 5.6 %

Advertising 3.2 %

Top 10

10Torsten Zimmermann

https://netray.io

Infrastructure - Cedexis DNS Deployment

� Authoritative DNS Deployment� Managing *.cedexis.net� 4 servers configured in .net zone

¾ flip{a,d,g,m}.cdxcn.net

� Measure latency (ping)� ~ 870 RIPE Atlas probes� 4 server locations

¾ US, South America, Europe, Asia¾ Anycast deployment

� Performance of DNS� Register own Cedexis account� Directly return an A-record

<10ms<20ms

<50ms<100ms

<200ms>200ms

<10ms<20ms

<50ms<100ms

<200ms>200ms

a) min ping RTTs to authoritative nameservers

b) Median DNS query time to resolve A record

11Torsten Zimmermann

https://netray.io

Infrastructure - Active Measurements by Cedexis

� Radar Community Probes [1]� Embedded JavaScript� Different metrics and infrastructures

¾ throughput, latency, availability¾ Cloud, CDN, ISP

� Different levels of severity¾ minor, medium, major

� Definition of anomaly and severity by Cedexis� E.g., confirmed CDN medium throughput event

¾ Witnessed by 5 ISPs (AS) for ≥ 5 mins¾ Throughput drop of 75% - 90%, compared to rolling average of last 5h

[1] https://live.cedexis.com

12Torsten Zimmermann

https://netray.io

Infrastructure - Active Measurements by Cedexis

� Analysis of captured events� Capture feed of website� 9th Oct. 2017 – 8th Jan. 2018

� Allows to infer location of visitors� Reported via probes� Hints at user base of customers

¾ Coincides with DNS deployment

� Reported Events (as observed)� Confirmed after ~ 9 mins� Most events concern CDNs

¾ CacheFly outage during study

0 200 400 600 800 1000 1200 1400

a) Locations of reported events

b) Event types, status, and severity

13Torsten Zimmermann

https://netray.io

Measurement Setup

� Probes� 35 PlanetLab nodes (AU, BRA, CA, CHN, CZE, JP, POL)� 6 RPis in 6 distinct German ISPs (DE)

� Measurement� Resolve each domain every 15 mins� In addition, capture (ping) latency

� Capture CNAME redirection to determine used CDN¾ If new domains discovered, include in set

14Torsten Zimmermann

https://netray.io

How customers utilize Cedexis?

� DNS Time-To-Live� CNAME mapping from Cedexis to subsequent entry� No country specific settings

� 67% ≤ 20s, faster reaction to change

� Compare to TTLs set by CDNs

(a) Customer configured TTLfrom Cedexis to the next CDN.

Rank Used by CDN TTL in seconds1 32.9% Akamai2 31.5% CDNetworks3 19.2% DNSDD4 14.8% Edgecast5 13.4% Level36 8.7% Cloudflare7 8.3% CDNWD8 7.3% ChinaCache9 5.8% Cloudfront

10 5.0% Highwinds0 100 200 300

TTL in s3600 3800TTL in s(b) TTLs of A-record for top 10 used CDNs. (Note the gap

in the time scale to display Edgecast using anycast.)

Fig. 5: DNS TTLs experienced among Cedexis-enabled domains. For a) mappings fromCedexis to the subsequent entry and b) the CDNs used for the final delivery.

increase between 200 % and 500 % is considered as medium severity. We find mostof the reported events to concern CDN, followed by ASes. The high amount of majoravailability events concern CacheFly CDN outages during our measurement period.Takeaway. We observe visitors of Cedexis-managed sited from almost every country. Yet,its anycast DNS platform is suggested to be based in the US, Europe, and Asia. Users inother countries can be subject to higher DNS query latencies.

4.2 How Customers utilize CedexisDNS TTL. The DNS Time To Live (TTL) defines the time a record can be cachedby DNS resolver and thus the timespan between Cedexis balancing decisions. A smallTTL allows more frequent switches at the cost of more frequent DNS queries to theCedexis DNS infrastructure. This query latency can be significant, depending on theDNS resolver location.

Figure 5(a) depicts the CDF of the TTLs for the validity of the CNAME-mappingsfrom Cedexis to the subsequent entity (see 2nd CNAME in Figure 1) for all customerdomains. We did not observe country-specific settings. Around 67% of all domains haveconfigured a TTL of at most 20 s, indicating a rather short time scale enabling rapidreactions to changes. The next 30% are within 300 s, denoting an already moderatereaction time while around 3% have configured higher TTLs. Higher TTLs can hint tonon-latency-based, but rather throughput or cost-based optimizations.

To compare these configurations to TTLs deployed by CDNs, we show the A-recordTTLs for the top 10 CDNs in Figure 5(b). To the right of every CDN, the figure showsthe boxplot of TTLs observed for the A-records of all resolutions we performed. We seethat the top 3 CDNs use a short TTL in the range of most Cedexis CNAMEs, whereasEdgecast has a lifetime of one hour (probably due to their use of anycast).DNS Resolution Time. When employing Cedexis, an additional step in DNS resolutionis required to enable CDN balancing. Figure 6(a) compares the latency for resolving(from our Planet Lab sites) a mapping at Cedexis, in case of multi-staged CDNs afurther CNAME redirect (CDN) and the final resolution of the A-record. We observethat Cedexis performs similarly to the other CDNs. However, while this hints at a goodDNS deployment for our vantage points, it also means that using Cedexis inflates thelatency of a DNS lookup. Given the on average short TTLs, users will often incur anadditional added latency when using Cedexis-enabled websites.

0 100 200 300TTL in s

0.0

0.2

0.4

0.6

0.8

1.0

CDF

ofdo

mai

ns

15Torsten Zimmermann

https://netray.io

AU BRA CA CHN CZE DE JP POLCountry

0255075

100

Shar

e

KeyCDNAizonIncapsulaG-CoreChinaCache

CacheflyFastlyMaxCDNLimelightHighwinds

CloudfrontCDNWDCloudflareLevel3

EdgecastDNSDDCDNetworksAkamai

� DNS Resolution Time� Cedexis to CDN to A-Record� Indicates good DNS deployment

¾ Regarding vantage points

� However, adds one additional step

� CDN Usage of Customers� Most customers 1-2 CDNs� Most regions behave the same

¾ Minor exception for CHN

� Contrast to Mukerjee et. al. [2]¾ Analysis of Conviva

� Extend analysis to open resolvers (161 countries)¾ For 67% same CDN in every country, 30% have two, 3% use three

How customers utilize Cedexis?

0.00 0.05 0.10 0.15 0.20Resolution time [s]

0.0

0.2

0.4

0.6

0.8

1.0

CDF

CedexisCDNA-Record

[2] M.K. Mukerjee et. al. The Impact of brokers on the future of content delivery, ACM HotNets 2016

16Torsten Zimmermann

https://netray.io

Latency Perspective

� Evaluate choices made by Cedexis� Compare latency of all configured CDNs to selected CDNs

¾ ICMP ping instead of HTTP requests to not generate costs

� Inflation evaluation� When choice was not optimal

¾ Difficult to assess the quality, routing metric not known

� 50% only small relative distance & only small absolute difference� Apart from BRA (& JP), 90% are within 20ms

0 50 100 150 200 250Relative latency inflation [%]

0.00.20.40.60.81.0

CDF

DEUS

AUBRA

0 50 100 150 200 250Absolute latency inflation [ms]

0.00.20.40.60.81.0

CDF

DEUS

AUBRA

17Torsten Zimmermann

https://netray.io

Conclusion

� Analysis of Meta-CDN concept by the example of Cedexis� Operational principle, customers, infrastructure, and performance

� Potential to steer large portions of traffic� Customers with users across the globe� Enables arbitrary routing decisions

¾ Radar, Fusion, Openmix platform

� Future Work� Quality of Meta-CDN’s own measurement

¾ Do they reflect the actual performance?

� Interested in more stuff that we do?� Visit https://netray.io

Thank you for your attention!