How SoundCloud scales...¥Alexa Ranking (233) ¥ Employees (190) - 80 engineers ¥ 4 office...

How SoundCloud scalesAlexander Grosse

@klangberater

Freitag, 5. April 13

“Youtube for Audio”

Freitag, 5. April 13

Fakten

Freitag, 5. April 13

• Alexa Ranking (233)

Fakten

Freitag, 5. April 13

• Alexa Ranking (233)• Employees (190) - 80 engineers

Fakten

Freitag, 5. April 13

• Alexa Ranking (233)• Employees (190) - 80 engineers• 4 office locations (Berlin, London, San Francisco,

Sofia)

Fakten

Freitag, 5. April 13

• Alexa Ranking (233)• Employees (190) - 80 engineers• 4 office locations (Berlin, London, San Francisco,

Sofia)• >35 million registered users

Fakten

Freitag, 5. April 13

• Alexa Ranking (233)• Employees (190) - 80 engineers• 4 office locations (Berlin, London, San Francisco,

Sofia)• >35 million registered users• partners (BBC, CNN)

Fakten

Freitag, 5. April 13

• Alexa Ranking (233)• Employees (190) - 80 engineers• 4 office locations (Berlin, London, San Francisco,

Sofia)• >35 million registered users• partners (BBC, CNN)• > 10.000 Apps

Fakten

Freitag, 5. April 13

Freitag, 5. April 13

How SoundCloud scales

Freitag, 5. April 13

?Freitag, 5. April 13

Organization

Technology

Freitag, 5. April 13

What is the foundation?

Freitag, 5. April 13

Freitag, 5. April 13

Principles

Freitag, 5. April 13

Top-Down vs. Bottom-Up

Principles

Freitag, 5. April 13

Top-Down vs. Bottom-Up

Trust

Principles

Freitag, 5. April 13

Top-Down vs. Bottom-Up

Trust

How to react to mistakes

Principles

Freitag, 5. April 13

Top-Down vs. Bottom-Up

Trust

How to react to mistakes

Continuous Improvement

Principles

Freitag, 5. April 13

Top-Down vs. Bottom-Up

Trust

How to react to mistakes

Continuous Improvement

Innovation

Principles

Freitag, 5. April 13

• Flat Hierarchies? Everybody has them...

• Engineering teams responsible for architecture (not architecture teams)

• Influence on the product

Top-Down vs. Bottom-Up

Freitag, 5. April 13

• Deployment

• Change Management

• Process

• Time for “technical debt”

Trust

Freitag, 5. April 13

• What will be changed if a mistake happens?

Reaction to mistakes

Freitag, 5. April 13

• R&D Department?

• Only Product?

Innovation

Freitag, 5. April 13

SoundCloud‘s basic principles

Freitag, 5. April 13

• No dedicated QA roles

SoundCloud‘s basic principles

Freitag, 5. April 13

• No dedicated QA roles

• “You build it - you run it”

SoundCloud‘s basic principles

Freitag, 5. April 13

• No dedicated QA roles

• “You build it - you run it”

• Team Building (Rotation, constant changes)

SoundCloud‘s basic principles

Freitag, 5. April 13

• No dedicated QA roles

• “You build it - you run it”

• Team Building (Rotation, constant changes)

• Hacker Time

SoundCloud‘s basic principles

Freitag, 5. April 13

• No dedicated QA roles

• “You build it - you run it”

• Team Building (Rotation, constant changes)

• Hacker Time

• Continuous Improvement

SoundCloud‘s basic principles

Freitag, 5. April 13

• No dedicated QA roles

• “You build it - you run it”

• Team Building (Rotation, constant changes)

• Hacker Time

• Continuous Improvement

• Move fast and break things

SoundCloud‘s basic principles

Freitag, 5. April 13

No QADoes that mean we don‘t test?

Freitag, 5. April 13

“You build it - you run it”

Freitag, 5. April 13

Flat Hierarchy

Freitag, 5. April 13

Innovation

Hacker Time

Freitag, 5. April 13

Freitag, 5. April 13

Continuous Improvement

RetrospectivesPost Mortems (Projekte)

Freitag, 5. April 13

Mistakes...

Freitag, 5. April 13

Scrum? Kanban?Process?

Freitag, 5. April 13

What is the most important thing you have to do so that this works?

Freitag, 5. April 13

Freitag, 5. April 13

???

Freitag, 5. April 13

???

Initial Culture

Freitag, 5. April 13

???

Initial Culture

Hiring!!!

Freitag, 5. April 13

Freitag, 5. April 13

SoundCloud Hiring Process

Freitag, 5. April 13

How does it look like in reality?

Freitag, 5. April 13

Organisation

Freitag, 5. April 13

HTML5 Mobile (native) other V1

System Engineering (Berlin / SF)

Search T&S Payment Partner Tools

Data Activ. Delivery Platform

API

Freitag, 5. April 13

How do we structure teams?

cross functional?

Freitag, 5. April 13

Do teams stay the same?

Change!

Freitag, 5. April 13

Technology

Freitag, 5. April 13

Ruby/Scala/Go/Clojure/Javascript

Freitag, 5. April 13

Scale Horizontally/Vertically?

Freitag, 5. April 13

Rails->Mysql

RabbitMQ->Cassandra

http://backstage.soundcloud.com/2012/08/evolution-of-soundclouds-architecture/

Freitag, 5. April 13

• 7 Master, 9 slaves, 900GB data

• Additional slaves in EC2

• Master Peak is 15K IOPS writes, 45k reads

• one slave lags intentionally 1 hour behind

• Je nach Konsistenzanforderung wird Slave oder Master gewählt (für reads - unterstützt read-only mode)

MySQL

Freitag, 5. April 13

• 3 Cluster

• 88 Nodes

• 15 TB of data

• up to 70K reads/sec

Cassandra

Freitag, 5. April 13

HadoopElastic Search

Proprietary Spam Detection

Freitag, 5. April 13

ElasticSearch

80 Mio Docs (gaining 1 Mio every 6 days)

175 QPS (gaining 15 QPS every month)

Freitag, 5. April 13

Hosted in S3/EC2and physical DCs

Freitag, 5. April 13

Outages

Freitag, 5. April 13

Freitag, 5. April 13

Post Mortems

Freitag, 5. April 13

• Move fast and break things

• Don‘t get slow because you fear to make mistakes

Summary

Freitag, 5. April 13

backstage.soundcloud.com

Freitag, 5. April 13

Alexander Grosse

alexandergrosse.com@klangberater

We are hiring!

www.soundcloud.com/jobs

Freitag, 5. April 13