Score! Superior Digital Experiences with MLS

SCORE!SUPERIOR DIGITAL EXPERIENCES WITH MLS

MAY 10, 2016

Brian Aznar(Director of Engineering, MLS Digital)

Steven Merrill (Director of DevOps, Phase2)

BACKGROUND MLS & DIGITAL PRODUCTS

MAJOR LEAGUE SOCCER STATE OF THE LEAGUE

‣ 20 current teams with plans to expand to 24 by 2020 ‣ Adding state of the art soccer-specific stadiums around the US -

average in stadium audience exceeds NHL and NBA ‣ Expanding our reach in the US and around the world with great new

TV distribution deals ‣ Young and multicultural fan base that is hungry for digital

MLS FANBASE

‣ Avids ‣ Regularly watches and

attends MLS matches ‣ Consume soccer related

content ‣ Regularly visits

MLSsoccer.com and other soccer specific sites

‣ Casuals ‣ Occasionally

watches and attends MLS Matches

MLS DIGITAL PRODUCT ECOSYSTEM

‣ Web: 20 club websites + 1 league web site using Drupal 7 ‣ Mobile Apps: iOS, Android ‣ Connected TV: Roku, Chomecast, Apple TV ‣ Services: Content API, Video & Stats ingest pipelines,

Content delivery, etc … ‣ Infrastructure: AWS

2015 CASE STUDY

GOAL

‣ Modernize and stabilize MLS’ web platform (backend and frontend) and successfully migrate 21 sites by the end of 2015 to AWS and D7.

THE TEAM

PHASE2

‣ Founded in 2001 ‣ Offices in DC, NYC, San Francisco, and Portland ‣ Committed to open source technology ‣ Focus on how content is delivered, how

teams collaborate, and how audiencesexperience information

PHASE2 & MLS

‣ Long-term partnership over 3 years ‣ Helped with D7 platform build ‣ March 2015 - we kicked off their migration to

MP7 Drupal platform and into AWS

PHASE2 & MLS TECHNOLOGY TEAM

‣ Drupal Developers (Phase 2) ‣ Fullstack Javascript Developers: React, Backbone, Node.js & Go (MLS) ‣ DevOps (Phase 2) ‣ Process = Scrumban

‣ Maximize development velocity, agility and responsiveness to business needs.

‣ Centralized Development Team ‣ Moving from segmented teams towards a single cross disciplinary team.

CHALLENGES & SOLUTIONS

CHALLENGES WE NEEDED TO ADDRESS

1. Outdated UX originally built for desktop viewing only

2. Inflexible Authoring - Needed an efficient and scalable way to give clubs more control to build out their own content

3. Wasted Server Capacity - High variability in traffic patterns means planning for the worst

CHALLENGES WE NEEDED TO ADDRESS

4. High Fixed infrastructure costs

5. Poor Performance

6. Technology Soup - Go, node, CoreOS/Fleet, Docker,PHP/Drupal (Multiple Versions) and Salt all managed with a lean team

7. Aggressive Migration - 21 websites and No downtime

a. Avoiding major tent pole events that varied across clubs

1. MODERN UI EXPERIENCE

‣ Building blocks that editors can put together ‣ All features were designed to be responsive ‣ Video highlights are pushed from MLS’s video providers ‣ Import of MLS entities (teams/players/seasons) from MLS’s pipeline

2. EDITORIAL FLEXIBILITY

3. SPIKEY TRAFFIC

‣ Traffic Spikes (avg 4x) centered around games and events ‣ League represented about half the traffic on average ‣ Great candidate for autoscaling

4.a HIGH INFRASTRUCTURE COST

‣ 2 Infrastructures (One foot in the cloud) ‣ Drupal running in Rackspace Servers ‣ Services Running in AWS

‣ Drupal running on fixed set of large servers configured to handle the average amount of traffic plus physical devices (firewalls, NetApp storage…)

‣ Paying for unnecessary resources needs during off peak and off season traffic

4.b MOVING INTO AWS

‣ Rackspace => AWS - everything in one cloud infrastructure ‣ Autoscaled Instances ‣ Additional operational benefits by using:

‣ Amazon RDS, ElastiCache, S3

AUTOSCALING‣ Reduces stress ‣ Automatic healing ‣ Performance benefits

‣ Manual adjustment up and down ‣ Automatic scaling based on metrics

‣ Cost benefits

‣ For autoscaling, you should build cattle, not pets ‣ Previous MLS autoscaling provisioned from scratch

‣ Phase2 built “fat” AMIs with all software + code ‣ Far faster deployment time ‣ An AMI can come up if everything else is down

‣ See Steven’s DevOps track session tomorrow for more info!

AUTOSCALING BACKGROUND

A USUAL MONTH

THE OVERALL RESULT

5. POOR PERFORMANCE

‣ Unsuccessful attempt at cloud multi region architecture ‣ Caching anomalies between Drupal in 2

datacenters ‣ Slow cache clearing bottlenecks

‣ Slow server response times > 1000 ms

‣ Right-sized AWS instances to their workloads ‣ Upgraded to PHP 5.5 for a performance boost ‣ Optimal instance size + PHP 5.5 = 40% increased throughput ‣ Added stability fixes

PERFORMANCE TUNING: QUICK WINS

‣ Load Testing & Application Monitoring to find and remove bottlenecks ‣ Disabled DB read/write split for a massive speed increase ‣ Doubled the size of memcache instance ‣ Killed legacy code that cleared all of Drupal’s caches ‣ Results:

‣ Instance sizes reduced by half ‣ Massive decrease in average page render time

PERFORMANCE TUNING

6. TECH COMPLEXITY

‣ Clubs and League Content Sites ‣ Drupal (6/7) ‣ PHP (5.1/5.3) ‣ Jenkins / Salt for Deployment

& Configuration Management

‣ Web Services / Stats & Video Ingest Pipeline / Misc Web Apps ‣ Node.js & Go ‣ CoreOS/Fleet/Docker & Teamcity for

Deployment & Configuration Management

‣ Server Configuration Drift

7. MIGRATION

‣ Seattle, Orlando, NYCFC were in place ‣ Large project management lift

‣ Coordination with stakeholders ‣ Training schedule ‣ Working with club schedules /

game schedules

‣ Central Digital Club Services team played a huge role by managing coordination w/ clubs and drupal editors

‣ Drupal multisite ‣ Saved server resources ‣ Made deploys smaller ‣ Worked for MLS Digital specifically because of

the integrated nature of their Drupal platform ‣ No bespoke development per-club ‣ Centralized feature development ‣ Far lower support burden (6 people)

MULTI-TENANT SETUP

PROJECT MILESTONES

‣ April 2015: Migrated first team site (NY Red Bulls)

‣ July: Turned on autoscaling ‣ April - November: Migrated MP7 ‣ October: League site launched! ‣ October: Decision Day

‣ December: Turned off Rackspace hosting ‣ January 2016: Helped hire in-house

DevOps Engineer ‣ March: Successful 2016 Season Kickoff

TOOLS

TOOLS

‣ New Relic for Application Performance Management / Instrumentation

‣ Blaze meter for Load testing ‣ Datadog for Infrastructure Metrics

NEW RELIC

‣ Table Stakes Tooling for the Modern Technology Team

‣ Real Time End to End Client & Server metrics and performance measurement ‣ Introspection of everything from end

page rendering on a user’s browser to PHP & DB bottlenecks on the server

‣ Tiers: ‣ Free - 1 day retention -

good enough for load tests ‣ Pro - 90 day retention

REALTIME FULL STACK METRICS: CLIENT TO SERVER TO REPO

CLUB ROLLUPS

REALTIME FULL STACK METRICS: CLIENT TO SERVER TO REPO

CLUB ROLLUPS

MODULE, HOOK & VIEW PERFORMANCE

GLOBAL ROLLUPS

MULTI SITE TOP LEVEL METRICS: AGGREGATE

CLUB ROLLUPS

MULTI SITE TOP LEVEL METRICS: INDIVIDUAL SITE

SUMMARY

OUTCOMES

‣ Migrated fully onto AWS ‣ Lowered hosting bill significantly by leveraging AWS features ‣ Helped reduce support load through policy changes

and new MLS hires ‣ Go see Steven’s session for more info

on immutable infra!

PROBLEMS ADDRESSED IN 2015

1. Outdated front-end UI experience originally built for desktop viewing only

2. Club editorial teams needed more control and flexibility to build content that met their needs but we needed it all to fit within the same brand ecosystem

3. High variability in traffic patterns resulting wasted capacity

PROBLEMS ADDRESSED IN 2015

4. High fixed infrastructure costs

5. Managing multiple technologies (CoreOS/Fleet,Salt, PHP, Go, node) and versions (PHP, Drupal) with a small team

6. Migrating 21 websites without any down time and avoiding major tent pole events that varied across clubs

THE FUTURE

THE FUTURE

‣ Improving performance ‣ Improving editorial usability ‣ Further consolidation & simplification

of tech stack ‣ #docker!

‣ Preparing for omnichannel

CHECK US OUT!

labs.mlssoccer.com

mlssoccer.com

phase2technology.com

SOCCER STADIUMSNEAR YOU!

QUESTIONS

So How Was It? - Tell Us What You ThinkEvaluate th is session - ht tps: / /events.drupal .org/neworleans2016/schedule

Thanks!