Lessons Learned from Verizon: Implementing Microservices with go90

  • View
    2.516

  • Download
    2

Embed Size (px)

Text of Lessons Learned from Verizon: Implementing Microservices with go90

  • How Verizon's go90 streams the NFL Superbowl with Lightbend

    TechnologiesChristopher Webster

    Verizon / AOL

  • Outline Overview of go90 Service Architecture Traffic Spikes Availability Caching

  • Go90 App

  • What is go90 App

    Mobile app supporting both video on demand and live event streaming

    Platform APIs for app and website delivery

    Signup / Registration Video content metadata Social interactions

    Comments / sharing / favorite Management APIs

    Content restrictions Content metadata - automatic ingestion Merchandising

  • Client

    CDN/WAF

    Edge API

    Accounts

    SocialNotifications

    ExperimentsSearch

    Discovery2D View

    Metadata

  • Technologies The edge services all use Scala (2.11) with Java 8 Cassandra, Elastic Search, Kafka, and some Mongo Cloud native application Play web services 2.3 (2.4 migration in progress)

  • Auth

    Find Content

    Landing Page

    End

    Get Key Play Content

    Video app during a live event

  • Traffic Spikes

    10x

    4 min

  • Traffic Spikes Timed Scaling

    Can be used for known events Difficult to predict the actual load, could still be under provisioned

    Autoscaling Helps but takes time Adjust the policies to assume a spike (aggressively adding capacity) Adding capacity around the utilization target

    Load Shedding with Circuit Breaker allows popular experience to be preserved Failure triggers default behavior (serving static initial experience page)

    Use CDN where appropriate in extreme cases

  • Availability Microservices help partition outages to specific areas

    Failure of user login wouldnt impact video data

    Extensive use of Future recover Use of Akka CircuitBreaker in conjunction with failure recovery to further aid

    Fail requests fast and use recovery if downstream services are in trouble

    Employing load shedding techniques Upstream techniques but also use a filter to keep track of number of requests in flight

    Shed load as the normal limit is exceeded

    Bounded queues Service calls, database persistence queues

  • Future Recovery Example

  • Circuit Breaker Example

  • Caching - Hints CDN / app caching help with extreme failures Use prewarmed caches Use caution around automatic cache expiration Separate hot cache from on demand cache

  • Questions