Scalable Event-Based Systemsfiles.meetup.com/3946322/Scalable-Evented-Systems.pdf · Given a stream of discrete data items, a chunk missing in the middle. Gaps happen when you don’t

@

Jeremy PierreYet Another Developer j14159

Lessons Learned, Traps Sprung, Stuff From The Trenches At HootSuite

Scalable Event-Based Systems

twitter.com/hootsuite facebook.com/hootsuite slideshare.com/hootsuite blog.hootsuite.com

Monday, 23 July, 12

Gaps happen when you don’t pull o!en enough.

REST breaks down at a certain volume.

Why?


Monday, 23 July, 12

Given a stream of discrete data items, a chunk missing

in the middle.

Gaps happen when you don’t pull o!en enough.

REST breaks down at a certain volume.

Why?

WTF is a gap?


Monday, 23 July, 12

Why Event Based at HootSuite?

Most public APIs have limitations:

• One user per API call


Monday, 23 July, 12



• Lots of requests in parallel

> 4 million users

millions of Twitter accounts



Monday, 23 July, 12



• Lots of requests in parallel

> 4 million users

millions of Twitter accounts

• HTTP/HTTPS is expensive



Monday, 23 July, 12

The First Steps

Use Twitter’s Streaming API...

• General data collection/categorization

• Push Notifications first

• Analytics


Monday, 23 July, 12

Halfway to Event Sourcing...

What is “Event Based”?

• Push vs pull

• Discrete domain object instances

• Time ordered


Monday, 23 July, 12

Halfway to Event Sourcing...

What is “Event Based”?

• Push vs pull

• Discrete domain object instances

• Time ordered


A system that relies on discrete time-ordered data elements pushed to components rather than

components pulling one or more elements from a common source.

Monday, 23 July, 12

Looks a little bit like OOP when you squint...

Evented System Characteristics

• Generalized standalone components


Monday, 23 July, 12




• Well-known domain objects (interfaces)


Monday, 23 July, 12




• Well-known domain objects (interfaces)

• Inputs and outputs are message queues


And no, it’s not RPC

Monday, 23 July, 12

The Players


• Scala: functional(cool) and objects(comfort)

• Akka: concurrency, fault tolerance

• RabbitMQ: client-agnostic messaging

• ZooKeeper: self-awareness

And this is growing fast.

Monday, 23 July, 12

The Streaming Base Layer


• Dispatch assigns work

• Harvesters collect data

• Processors perform loose categorization

A collection of loosely couple services

Monday, 23 July, 12

General message bus

How RabbitMQ is Used


• Harvesters → single durable queue• Processors consume from harvesters• Processors → topic exchange

Monday, 23 July, 12

site.twitter.user123.mentionsite.twitter.*.mention

Push service consumes from topic exchange:

General message bus

How RabbitMQ is Used


• Harvesters → single durable queue• Processors consume from harvesters• Processors → topic exchange

Monday, 23 July, 12

When RabbitMQ falls over...

The Development Apocalypse


Each device & user combo gets a queue

+

Monday, 23 July, 12





150,000 device & user combinations+

+

Monday, 23 July, 12





150,000 device & user combinations

Push server stops

Maximum Badness

+

+

=

Monday, 23 July, 12

The solutions...

• Do some routing/filtering in your app

• Federation

• TCP load balancers



No silver bullet

Monday, 23 July, 12

How ZooKeeper is Used

Self-Awareness:


• Lets dispatch watch for recovery/load balancing opportunities

• Less configuration

• Node health snapshots

Monday, 23 July, 12

How ZooKeeper is Used

Clustering and leader election:


• Dispatch nodes

• Push nodes

• Metrics master

Monday, 23 July, 12

The Production Apocalypse

When ZooKeeper meets The Cloud:


Default (sane) session timeout

+

Monday, 23 July, 12





Major reaction to disappearing nodes+

+

Monday, 23 July, 12





Major reaction to disappearing nodes

Unpredictable pseudo network

Maximum Badness

+

+

=

Monday, 23 July, 12

Events and LAMP


It’s not all bad news for REST...

Monday, 23 July, 12

Events and LAMP


...as long as it is only the source of events.

It’s not all bad news for REST...

• Push subscriptions from public API

• Subscriptions to base layer from Analytics

• New message posts

Monday, 23 July, 12

Push Servers → HootSuite API

Feedback is sometimes necessary:

• UrbanAirship reports invalid devices

• HootSuite API uses a different DB

• API has to poll a durable queue (cron)


Monday, 23 July, 12

Streaming Base → Analytics

Up to date and fast required...

• uses mention and retweet data

• high frequency polling, daemon process

• durable, non-autodelete, !auto-ack queue


Monday, 23 July, 12

The Speedbump

When RabbitMQ meets PHP

• cron and daemon are your friends

• non-autodelete, no auto-ack

• durable is optional

• consume as frequently as possible


Monday, 23 July, 12

Metrics for the Masses: Graphite

Graph Everything

Metrics updating every 5-15

minutes is insufficient


Monday, 23 July, 12

Metrics for the Masses: Graphite

Graph Everything

Metrics updating every 5-15

minutes is insufficient

• Instant insight• see external problems before they’re reported• easy custom graphs• UI’s not great but it works


Monday, 23 July, 12

Getting Rid of REST: ∅MQ

For internal APIs, timeliness > throughput


Monday, 23 July, 12

Getting Rid of REST: ∅MQ

For internal APIs, timeliness > throughput

• Enterprise clients have shifting team structures

• Access control changes must propagate quickly

• Uniform API


Decentralized Message Queues:

Monday, 23 July, 12

Documents

Scalable Event-Based Systemsfiles.meetup.com/3946322/Scalable-Evented-Systems.pdf · Given a stream of discrete data items, a chunk missing in the middle. Gaps happen when you don’t