How Verizon's go90 streams the NFL Superbowl with Lightbend
TechnologiesChristopher Webster
Verizon / AOL
Outline● Overview of go90
● Service Architecture
● Traffic Spikes
● Availability
● Caching
Go90 App
What is go90● App
○ Mobile app supporting both video on demand and live event streaming
● Platform
○ APIs for app and website delivery
■ Signup / Registration
■ Video content metadata
■ Social interactions
● Comments / sharing / favorite
○ Management APIs
■ Content restrictions
■ Content metadata - automatic ingestion
■ Merchandising
Client
CDN/WAF
Edge API
Accounts
SocialNotifications
ExperimentsSearch
Discovery2D View
Metadata
Technologies● The edge services all use Scala (2.11) with Java 8
● Cassandra, Elastic Search, Kafka, and some Mongo
● Cloud native application
● Play web services 2.3 (2.4 migration in progress)
Auth
Find Content
Landing Page
End
Get Key Play Content
Video app during a live event
Traffic Spikes
10x
4 min
Traffic Spikes● Timed Scaling
○ Can be used for known events
○ Difficult to predict the actual load, could still be under provisioned
● Autoscaling
○ Helps but takes time
○ Adjust the policies to assume a spike (aggressively adding capacity)
○ Adding capacity around the utilization target
● Load Shedding with Circuit Breaker allows popular experience to be preserved
○ Failure triggers default behavior (serving static initial experience page)
● Use CDN where appropriate in extreme cases
Availability● Microservices help partition outages to specific areas
○ Failure of user login wouldn’t impact video data
● Extensive use of Future recover
● Use of Akka CircuitBreaker in conjunction with failure recovery to further aid
○ Fail requests fast and use recovery if downstream services are in trouble
● Employing load shedding techniques
○ Upstream techniques but also use a filter to keep track of number of requests in flight
■ Shed load as the normal limit is exceeded
● Bounded queues
○ Service calls, database persistence queues
Future Recovery Example
Circuit Breaker Example
Caching - Hints● CDN / app caching help with extreme failures
● Use prewarmed caches
● Use caution around automatic cache expiration
● Separate hot cache from on demand cache
Questions
Recommended
