Leveraging Multi-CDN
at Riot Games
Kris BeeversFounder
NS1
Ray PanahonTech Ops
Riot Games
Intelligent DNS & Traffic Management
Leveraging Multi-CDN at Riot Games
Leveraging Multi-CDN at Riot Games
As you scale, one CDN may not fit all your needs, even within the same use case
Distributed usersCDNs do not perform uniformly well for all geographies / networks
Fault toleranceCDNs are built for resilience, but localized outages still happen
Cost managementDiversification enables you to trade off performance and cost
Leveraging Multi-CDN at Riot Games
Make sure you’re introducing CDNs that fit your application, not just your geo/network demands:
• CDNs often specialize by content type (small objects, large objects, video, mobile, ...)
• Using implementation-specific features? Will they translate across CDNs?
• Not just about pushing bits: invalidation, APIs, reporting, etc
Leveraging Multi-CDN at Riot Games
DNS lookup is a good time to address multi-CDN:
• Already part of every application
• CDN typically implemented with CNAME "pointers" in DNS• cdn.example.com IN CNAME
example.somecdn.net
• Opportunity for deep control over CDN selection: cdn.example.com -> ALGORITHM(CDN1, CDN2, CDN3) -> CNAME answer
Leveraging Multi-CDN at Riot Games
Implementing multi-CDN with DNS
Step 1: selecting CDNs
• What are you trying to optimize? Global delivery? Performance in key markets? Reliability against localized network issues? Cost?
• Ensure you select CDNs suitable for your content
• Look for CDNs with some network independence -- cover your gaps. Don’t want CDNs with same upstreams, same datacenters, etc.
Leveraging Multi-CDN at Riot Games
Implementing multi-CDN with DNS
Step 2: burn-in testing
• Don't go full multi-CDN from Day 0 -- big configuration risk
• Canary test: introduce new CDNs and multi-CDN selection with a small % of users
• Modern DNS traffic management platforms support granular weighting for gradual turn-up
Leveraging Multi-CDN at Riot Games
Implementing multi-CDN with DNS
Step 3: traffic management
• Which CDN should we use for this user right now?
• Approach depends on app & optimization needs
• Best technologies make building and evolving complex application-specific algorithms easy
• Key aspect of modern multi-CDN: getting real-time data into traffic management platform
Leveraging Multi-CDN at Riot Games
Example: Local CDNs in Key Markets
• Basic idea: introduce CDNs with connectivity optimized for specific geographies or ISPs
• Global CDNs: great coverage in US, EU, some of APAC
• Smaller/developing markets more challenging, but can be key for a use case like Riot's -- huge active user bases in South America, Eastern Europe, etc.
Leveraging Multi-CDN at Riot Games
Example: Local CDNs in Key Markets
Multi-CDN using a simple algorithm:
Extract GeoIP GEOFENCINGCDN Selection
ConfigurationCDN1South AmericaCDN2TurkeyCDN3AustraliaCDN4Rest of World
cdn.example.com?
Leveraging Multi-CDN at Riot Games
Example: Selecting CDN based on response time
• Basic idea: select from among multiple CDNs globally, based on response time between CDN and ISPs/geographies/etc
• Optimizes delivery across multiple networks using real-time RUM data
• Protects against localized outages or network issues in individual CDNs
Leveraging Multi-CDN at Riot Games
Example: Selecting CDN based on response time
Extract GeoIP + ASN, detect mobile,
etc.RUM Steering
CDN Selection
ConfigurationCDN 1 CDN 2CDN 3CDN 4
cdn.example.com?
RUM Measurement Engine
CDN 1 CDN 2 CDN 3 CDN 4
Response timemeasurements
Leveraging Multi-CDN at Riot Games
Example: selecting CDN based on locality & application-specific performance metrics
• Basic idea: direct traffic in key markets based on a performance metric measured from within the application
• Application specific metrics: e.g., real-world throughput downloading a game patch
• Measure and direct traffic based on metrics that matter most to your application
Leveraging Multi-CDN at Riot Games
CDN 3 CDN 4
Example: selecting CDN based on locality & application-specific performance metrics
cdn.example.com?Extract GeoIP + ASN, detect mobile, etc.
ConfigurationCDN1: S America CDN2: TurkeyCDN3: AustraliaCDN4: Global
GEOFENCING RUM Steering
RUM Measurement Engine
CDN Selection
CDN 1 CDN 2
App-specificmeasurements
Leveraging Multi-CDN at Riot Games
Leveraging Multi-CDN at Riot Games
Patching our games is a huge delivery problem:
• Huge number of global active users• League of Legends: patch/updates
every other Tues/Wed – 350-450Mb to every user across 13 localization regions in 36-48h
• Hot fixes / patches released on the fly• E-sports environments: competitive
play = performance sensitivity
Leveraging Multi-CDN at Riot Games
One CDN is not enough:
• CDN network / geo diversity to deliver to global player base• Redundancy & better performance in key markets
• Niche/regional CDNs can be great in super-specific markets• Tariff free updates
Leveraging Multi-CDN at Riot Games
One CDN is not enough:
• Other motivations for multi-CDN:• Enabling new products to be multi-CDN ready from the start• Increased reliability of front-facing products• Level playing field by driving price transparency of CDN services
Leveraging Multi-CDN at Riot Games
Globally distributed users?24/7? (who isn’t)
Big scale?
You need multi-CDN
Leveraging Multi-CDN at Riot Games
DNS is a great way to introduce multi-CDN
• Changing patcher code — long lead times • Minimal impact on patcher / release teams• Granular control of whom we release – targeting highly impacting
changes is huge
Leveraging Multi-CDN at Riot Games
Today: 8+ CDNs and counting globally
Constant internal analysis & discussions:• Reviewing strengths & weaknesses of
each CDN partner• Which regions / ISPs does each CDN
serve best
Leveraging Multi-CDN at Riot Games
So far: big success in specific tough markets
• Brazil: 60-90% increase in download throughput
• Turkey: similar• Australia: routing ISPs with download
tariffs to specific white-listed CDNs• Empowered local teams to take more
control over regional delivery
Leveraging Multi-CDN at Riot Games
Lots of lessons
Shared origin across many CDNs is a challenge (S3, GCP, roll-your-own)
Leveraging Multi-CDN at Riot Games
Lots of lessons
Need a normalized approach for monitoring/analyzing perf across CDNs
Leveraging Multi-CDN at Riot Games
Lots of lessons
Flexible routing capabilities are key because different strategies work well in different markets
Leveraging Multi-CDN at Riot Games
Lots of lessons
Operating across many CDNs puts serious workload on NOC/support staff:• Which CDN is serving bad/stale data?• Control panel / API / feature deltas
Leveraging Multi-CDN at Riot Games
What's next?
• Performance-based routing• Local patcher software gathers
real-time data perfect for routing, e.g. real patch download throughput metrics
• Combine with NS1’s Pulsar CDN datasets & ASN/IP/geo specific rules to drive granular routing
• Leveraging same data to gain carrier-level insights for NOC/support
Thank You!
@riotgames
@nsoneinc / @beevek