Drinking From The Firehose - The Erlang Way

Tags:

Preview:

Click to see full reader

DESCRIPTION

Imagine that you have tens of millions of endpoints, each of which is sending you a constant stream of data - to the tunes of petabytes per second. You also have millions of uses who want to monitor and administer them. And don't forget all the historical reports that these users insist upon all the time. And imagine that this these users and endpoints are distributed all over the world. This pretty much describes our environment - one that we've implemented with not just Riak, but also ElasticSearch, *and* Cassandra (because why go with one, when you can have all three!) - with Erlang tying all the moving pieces together. Join me as I show you how we successfully drink from this firehose of data without spilling a drop.

Citation preview

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

Drinking from the

FirehoseMahesh Paolini-Subramanya (@dieswaytoofast)V.P. R&D, Ubiquiti Networks

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

Drinking from the

Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

Drinking from the

FirehoseAka: “Scale Matters”

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

ARCHITECTURE

DEVICE

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Architecture

DEVICE

STATUS

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Scale-Fail

DEVICE

STATUS

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Data Stores

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Data Stores

• State of the World

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Data Stores

• State of the World

• Historical Data

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Data Stores

• State of the World

• Historical Data

• Configurations

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Data Stores

• State of the World

• Historical Data

• Configurations

• Billing

AGILITY

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

Fault Tolerance

From

http://www.erlang.org/download/armstrong_thesis_2003.pdf

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

The Big Six

Concurrency

From

http://www.erlang.org/download/armstrong_thesis_2003.pdf

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

The Big Six

Concurrency

Fault detection

From

http://www.erlang.org/download/armstrong_thesis_2003.pdf

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

The Big Six

Concurrency

Fault detection

Fault Identification

From

http://www.erlang.org/download/armstrong_thesis_2003.pdf

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

The Big Six

Concurrency

Fault detection

Fault Identification

Error Encapsulation

From

http://www.erlang.org/download/armstrong_thesis_2003.pdf

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

The Big Six

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

From

http://www.erlang.org/download/armstrong_thesis_2003.pdf

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

The Big Six

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

From

http://www.erlang.org/download/armstrong_thesis_2003.pdf

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

The Big Six

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

From

http://www.erlang.org/download/armstrong_thesis_2003.pdf

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

The Big Six

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

From

http://www.erlang.org/download/armstrong_thesis_2003.pdf

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

** exception error: no match of

right hand side value

{error,unhappy}

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

The Big Six

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

From

http://www.erlang.org/download/armstrong_thesis_2003.pdf

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

** exception error: no match of

right hand side value

{error,unhappy}

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

The Big Six

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

From

http://www.erlang.org/download/armstrong_thesis_2003.pdf

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

The Big Six

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

From

http://www.erlang.org/download/armstrong_thesis_2003.pdf

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

The Big Six

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

From

http://www.erlang.org/download/armstrong_thesis_2003.pdf

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

The Fault Tolerant System

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

The Big Six

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

From

http://www.erlang.org/download/armstrong_thesis_2003.pdf

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

The Big Six

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

From

http://www.erlang.org/download/armstrong_thesis_2003.pdf

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

The Big Six

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

From

http://www.erlang.org/download/armstrong_thesis_2003.pdf

LOOSE C

OUPLING

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

Loose Coupling

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

Loose Coupling

Breeds Trust

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

Loose Coupling

Breeds Trust

Devotes Brainpower appropriately

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

Loose Coupling

Breeds Trust

Devotes Brainpower appropriately

Minimizes bugs

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

Fault Tolerance

60 - 90% of all SW projects fail

10 – 25% of all SW projects get abandoned

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

The Big Six - Systems

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

The Big Six - Systems

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

MONIT

ORING

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

Monitoring?

Monitoring?• Dashboards

Monitoring?• Dashboards

• Out of band systems

Monitoring?• Dashboards

• Out of band systems

• Polyglot safety

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

The Big Six - Systems

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

The Big Six - Systems

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

Polyglo

t

Persis

tence

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Polyglot Persistence

Scale-Fail

DEVICE

STATUS

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Data Stores

• State of the World

• Historical Data

• Configurations

• Billing

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Data Stores

• State of the World

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Data Stores

• State of the World ElasticSearch

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Data Stores

• State of the World ElasticSearch

• Historical Data

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Data Stores

• State of the World ElasticSearch

• Historical Data Cassandra

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Data Stores

• State of the World ElasticSearch

• Historical Data Cassandra

• Configurations

Source: Wholly unscientific research by me.

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Data Stores

• State of the World ElasticSearch

• Historical Data Cassandra

• Configurations Riak

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Data Stores

• State of the World ElasticSearch

• Historical Data Cassandra

• Configurations Riak

• Billing

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Data Stores

• State of the World ElasticSearch

• Historical Data Cassandra

• Configurations Riak

• Billing Postgres

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Data Stores

• State of the World ElasticSearch

• Historical Data Cassandra

• Configurations Riak

• Billing Postgres

Polyglo

t

Persis

tence

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Data Stores

• State of the World ElasticSearch

• Historical Data Cassandra

• Configurations Riak

• Billing Postgres

LOOSE C

OUPLING

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Data Stores

• State of the World ElasticSearch

• Historical Data Cassandra

• Configurations Riak

• Billing Postgres

MONIT

ORING

1

1

1

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Split Brain

• Its ubiquitous (thanks aphyr)

http://aphyr.com/

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Split Brain

• Its ubiquitous (thanks aphyr)

• Zookeeper doesn’t necessarily help

http://aphyr.com/

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Split Brain

• Its ubiquitous (thanks aphyr)

• Zookeeper doesn’t necessarily help

• Moving the problem around doesn’t necessarily help

http://aphyr.com/

Cheating classroom)

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Cheating

• Stale status messages are OK

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Cheating

• Stale status messages are OK

• Charts can miss data

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Cheating

• Stale status messages are OK

• Charts can miss data

• Billing is sharded

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Cheating

• Stale status messages are OK

• Charts can miss data

• Billing is sharded

• Availability zones (google.co.uk)

Scaling

Drinking from the Firehose

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

Mahesh Paolini-Subramanya

(@dieswaytoofast)

V.P. R&D, Ubiquiti Networks

Recommended

2016 11 03-GDC-Firehose-Rungdac.broadinstitute.org/runs/docs/2016_11_03-GDC-Firehose-Run.pdfGDAC Firehose Integration With The Genomic Data Commons mnoble@broadinstitute.org 2016_11_18
Documents
2013-07-11 1 Out-of-the-Box Computing Patents pending Google7/11 2013 Drinking from the Firehose The Belt machine model in the Mill CPU Architecture
Documents
MIT Torch or Firehose Spanish
Documents
Erlang Patterns Matching Business Needs€¦ · Erlang Patterns Matching Business Needs! Torben Hoffmann! CTO, Erlang Solutions! torben.hoffmann@erlang-solutions.com! @LeHoff. Erlang
Documents
The Torch or the Firehose - Arthur Mattuck
Documents
Evaluating the Erlang C and Erlang A Models for Call ...myweb.ecu.edu/ROBBINST/PDFs/Erlang Compare Working paper.pdf · 1 Evaluating the Erlang C and Erlang A Models for Call Center
Documents
Eonblast An Erlang Game Stack Erlang User Conference 2012 H. Diedrich Lua + Erlang + VoltDB
Documents
Drinking from the Firehose: Web Weaving and Acquisitions
Documents
Erlang Programming Erlang Training and Consulting Ltd for
Documents
Drinking From The Firehose
Technology
Scalable Distributed Erlang - dcs.gla.ac.ukdcs.gla.ac.uk/~natalia/chechina-sd-erlang-efl14.pdf · RELEASE Project Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics
Documents
Realtime Analytics on the Twitter Firehose with Cassandra
Technology
Erlang - lysator.liu.setab/erlang/upplysning.pdf · Experiment med Erlang. En interpretator skriven i Prolog k ˜ or Erlangkod. Man provar att skriva en MD110. 1991 JAM. 1993 Erlang
Documents
10 June 2015 1 Mill Computing, Inc.Patents pending One of a series… Drinking from the Firehose Compilation for a Belt Architecture
Documents
Amazon Kinesis Firehose API Reference - API Reference
Documents
Drinking from a firehose
Education
Cloud Scaling Erlang Erlang - erlang-factory.com · Cloud Scaling Erlang ... • Message bus with Direct and Topic based Exchanges connects all major components ... erlang-rfc4627,
Documents
Flex 4 components from the firehose
Technology
Firehose Engineering
Technology
Erlang - GOTO Bloggotocon.com/dl/...Hoffmann_2014-GotoNight-Erlang.pdf · Erlang! The Driver behind WhatsApp’s Success! Torben Hoffmann! CTO, Erlang Solutions! torben.hoffmann@erlang-solutions.com!
Documents