Embed Size (px)
DESCRIPTION
Imagine that you have tens of millions of endpoints, each of which is sending you a constant stream of data - to the tunes of petabytes per second. You also have millions of uses who want to monitor and administer them. And don't forget all the historical reports that these users insist upon all the time. And imagine that this these users and endpoints are distributed all over the world. This pretty much describes our environment - one that we've implemented with not just Riak, but also ElasticSearch, *and* Cassandra (because why go with one, when you can have all three!) - with Erlang tying all the moving pieces together. Join me as I show you how we successfully drink from this firehose of data without spilling a drop.
Citation preview
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Drinking from the
FirehoseMahesh Paolini-Subramanya (@dieswaytoofast)V.P. R&D, Ubiquiti Networks
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Drinking from the
Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Drinking from the
FirehoseAka: “Scale Matters”
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
ARCHITECTURE
DEVICE
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Architecture
DEVICE
STATUS
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Scale-Fail
DEVICE
STATUS
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World
• Historical Data
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World
• Historical Data
• Configurations
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World
• Historical Data
• Configurations
• Billing
AGILITY
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Fault Tolerance
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
** exception error: no match of
right hand side value
{error,unhappy}
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
** exception error: no match of
right hand side value
{error,unhappy}
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Fault Tolerant System
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
LOOSE C
OUPLING
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Loose Coupling
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Loose Coupling
Breeds Trust
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Loose Coupling
Breeds Trust
Devotes Brainpower appropriately
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Loose Coupling
Breeds Trust
Devotes Brainpower appropriately
Minimizes bugs
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Fault Tolerance
60 - 90% of all SW projects fail
10 – 25% of all SW projects get abandoned
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six - Systems
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six - Systems
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
MONIT
ORING
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Monitoring?
Monitoring?• Dashboards
Monitoring?• Dashboards
• Out of band systems
Monitoring?• Dashboards
• Out of band systems
• Polyglot safety
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six - Systems
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six - Systems
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
Polyglo
t
Persis
tence
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Polyglot Persistence
Scale-Fail
DEVICE
STATUS
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World
• Historical Data
• Configurations
• Billing
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World ElasticSearch
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World ElasticSearch
• Historical Data
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World ElasticSearch
• Historical Data Cassandra
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World ElasticSearch
• Historical Data Cassandra
• Configurations
Source: Wholly unscientific research by me.
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World ElasticSearch
• Historical Data Cassandra
• Configurations Riak
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World ElasticSearch
• Historical Data Cassandra
• Configurations Riak
• Billing
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World ElasticSearch
• Historical Data Cassandra
• Configurations Riak
• Billing Postgres
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World ElasticSearch
• Historical Data Cassandra
• Configurations Riak
• Billing Postgres
Polyglo
t
Persis
tence
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World ElasticSearch
• Historical Data Cassandra
• Configurations Riak
• Billing Postgres
LOOSE C
OUPLING
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World ElasticSearch
• Historical Data Cassandra
• Configurations Riak
• Billing Postgres
MONIT
ORING
1
1
1
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Split Brain
• Its ubiquitous (thanks aphyr)
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Split Brain
• Its ubiquitous (thanks aphyr)
• Zookeeper doesn’t necessarily help
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Split Brain
• Its ubiquitous (thanks aphyr)
• Zookeeper doesn’t necessarily help
• Moving the problem around doesn’t necessarily help
Cheating classroom)
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Cheating
• Stale status messages are OK
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Cheating
• Stale status messages are OK
• Charts can miss data
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Cheating
• Stale status messages are OK
• Charts can miss data
• Billing is sharded
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Cheating
• Stale status messages are OK
• Charts can miss data
• Billing is sharded
• Availability zones (google.co.uk)
Scaling
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Mahesh Paolini-Subramanya
(@dieswaytoofast)
V.P. R&D, Ubiquiti Networks
![Erlang 1) Er [icsson] lang [uage] 2) [A.K.] Erlang](https://img.pdfslide.net/doc/110x75/56815162550346895dbf8b99/erlang-1-er-icsson-lang-uage-2-ak-erlang.jpg)
