Building Scalable Systems: an asynchronous approach

Building Scalable Systemsan Asynchronous Approach

pleasure and pain

Monday, June 20, 2011

Who am I? @postwait on twitter

Author of “Scalable Internet Architectures”Pearson, ISBN: 067232699X

Contributor to “Web Operations”O’Reilly, ISBN:

Founder of OmniTI, Message Systems, Fontdeck, & CirconusI like to tackle problems that are “always on” and “always growing.”

I am an EngineerA practitioner of academic computing.IEEE member and Senior ACM member.On the Editorial Board of ACM’s Queue magazine.

Some rants about systems.

cutting through the crap

BIG data

• BIG data: doesn’t exist

• but it sure is a good way to market things.

• If you measure things in petabytesas we have for the last several yearsyou have data, not BIG data.

Data stores

• “new” NoSQL systems can scale better than an RDBMS

• yes

• They scale better than sharded RDBMS

• no

• I need NoSQL systems for “BIG” data.

• no

Why NoSQL

• The choice to use a NoSQL system is driven by:

• a more suitable data model

• we desire to shift our CAP theorem constraints

The cloud

• The cloud is not magic.

• The cloud enables engineers to rapidly deploy an architecture to run an application.

• There is nothing wrong with this.

The cloud: gotcha 1

• Provisioning services “the old way” took time due to:

• installing systems

• install packages

• determining and enforcing data security

• determining and enforcing SLA monitoring

• determining and enforcing DR: RPO and RTO

• documenting escalation and remediation efforts

The cloud: gotcha 2

• Quality of service in multi-tenancy environments is a very, very hard problem.

• In fact, no one has solved it.

The cloud: gotcha 3

• Platform as a service provides:

• MySQL? PostgreSQL? Redis? Node.js?

• .. patched.

The cloud: gotcha 4

• Scaling isn’t magic pixie dust.

• “I can run more instances of my app.”

• horribly false sense of security.

• You must have a scalable architecture

Design & Implementation Techniques

some say architecture != implementation

Architecture vs. Implementation

Architecture is without specification of the vendor,make, and model of components.

Implementation is the adaptation of an architectureto embrace available technologies.

They are intrinsically tied.Insisting on separation is a metaphysical argument(with no winners)

Respect Engineering Math

Engineering math:

19 + 89 = 110

“Precise” Math:

19 + 89 = 10.8

Respect Engineering Math

Engineering math:

19 + 89 = 110

“Precise” Math:

19 + 89 = 10.8

Ok. Ok. I must have, I must have put a decimal point in the wrong place or something. Shit. I always do that. I always mess up some mundane detail.

- Michael Bolton in Office Space

Ensure the gods aren’t angry.

Bob: We need to grow our cluster of web servers.

Alice: How many requests per second do they do, how many do you have and what is their current resource utilization?

Bob: About 200 req/second, 8 servers and they have no headroom.

Alice: How many req/second do you need?

Bob: 800 req/second would be good.

Alice: Um, Bob, 200 x 8 = 1600... you have 50% headroom on your goal.

Bob: No... 200 / 8 = 25 req/second per server.

Alice: Bob... the gods are angry.

Why you’ve pissed off the gods.

Most web apps are CPU bound (as I/O happens on a different layer)

Typical box today: 8 cores are 2.8GHz or 22.4 BILLION instructions per second.

22x109 instr/s / 25 req/s = 880 MILLION instructions per request.

This same effort (per-request) provided me with approximately15 minutes enjoying “Might & Magic 2” on my Apple IIe- you’ve certainly pissed me off.

No wonder the gods are angry.

Develop a model

Queue theoretic models are for “other people.”

Develop a model

Sorta, not really.

Develop a model

Sorta, not really.

Problems:

Develop a model

Sorta, not really.

Problems:

very hard to develop a complete and accurate model

Develop a model

Sorta, not really.

Problems:

Benefits:

Develop a model

Sorta, not really.

Problems:

Benefits:

provides insight on architecture capacitance dependencies

Develop a model

Sorta, not really.

Problems:

Benefits:

relatively easy to understand

Develop a model

Sorta, not really.

Problems:

Benefits:

relatively easy to understand

illustrates opportunities to further isolate work

Rationalize your model

Draw your model out

Take measurements and walk through the model to rationalize iti.e. prove it to be empirically correct

Draw your model out

You should be able to map actions to consequences:

Draw your model out

A user signs up ➙ 4 synchronous DB inserts (1 synch IOPS + 4 asynch writes) 1 AMQP durable, persistent message 1 asynch DB read ➙ 1/10 IOPS writing new Lucene indexes

Draw your model out

In a dev environment, simulate traffic and rationalize your model

Draw your model out

In a dev environment, simulate traffic and rationalize your model

I call this a “data flow causality map”

Complexity will eat your lunch

there will always be empirical variance from your model

explaining the phantoms leads to enlightenment

service decoupling in complex systems gives:

simplified modeling and capacity planning

slight inefficiencies

promotes lower contention

requires design of systems with less coherency requirements

each isolated service is simpler and safer

SCALES.

Asynchronous Systems

it’s likely you have no idea what you’re doing

Asychronous

• of or requiring a form of computer control timing protocol in which a specific operation begins upon receipt of an indication (signal) that the preceding operation has been completed.

• ...or “I’ll act when you tell me you are done”

• ...or a protocol wherein the initiation of a task and the report of its completion are separate operations.

Protocols

• Standards:

• AMQP(impl: ActiveMQ, RabbitMQ, OpenAMQ, etc.)

• Others:

• ZeroMQ

• Gearman

Guarantees

• Queueing protocols can be misleading.

• Are you sure you did what you think you did?

• Let’s use a publish as an example.

Publication

• Imagine a Queue:

• You assume that by calling “publish” thatyour message is placed on the queue andwill eventually be consumed(assuming a consumer).

• Most systems are ‘more’ asynchronous than that.

Publication what you think happens

Kernel Network Stack Network Stack Queue

message framewrite

User Space

publish

message framewritepublish read

BOOMwritemessage framereaderror

return

writemessage framereaderror

return

Publication what really happens

Kernel Network Stack Network Stack Queue

message framewrite

User Space

publish

message framewritepublish

BOOMwrite

message framereaderror

return

• Why do queueing protocols use

“silence for success?”

• Simple: performance

• no need for a roundtrip before the next message

• success is common, failure rare

• AMQP is not alone in this...

• 0MQ as well.

Now what?

• In each component you must decide if you need:

• synchronous system w/ synchronous protocol

• asynchronous system w/ synchronous protocol

• asynchronous system w/ asynchronous protocol

Service decisions

• Knowing you can lose messages is... okay?

• it can be

• there are plenty of uses for unreliable communications

• however... generally,it is much easier to build services that have end-to-end guarantees.

Non-asynchronous: synchronous

Kernel Network Stack Network Stack Database

message framewrite

User Space

publish

message framewritepublish read

BOOMwritemessage framereaderror

return

writemessage framereaderror

return

Asynchronous to the purpose

• Why is a Queue “asynchronous”

• and a Database “synchronous”

• I lied... “asynchronous” is “to the purpose.”

• If the ultimate, final goal is: storage in a DB

• and you return the result only after a commit

• then you are synchronous

Simple example: image thumbnailing

• A user uploads an email to a web site

• you need to produce 7 different transformations

• (size, color, etc.)

• Asynchronous system:

• synchronous upload protocol:

• user upload -> thank you we have it

• asynchronous processing

• file -> 7 mutations

A (more) complete example.

foursquare-like, untappd.com-like service

Better example: rewards calculation

• A user performs an action on your site

• and you need to reward them based on:

• social network, history, value

• you want to show them their reward “immediately.”

• Step 1: engineer for failure.

Rewards calculation: step 1

• the inability to calculate the rewardshall not prevent the action.(think: beer checkin on untappd)

• I want the reward calculation immediately.

• I need the checkin to be recorded.

• Decouple the rewards calculation:

1. receive user request

2. store(C)

3. queue the checkin(C) on QC

4. wait up to 500ms (reading rewards R from QR)

5. return R witnessed.

• Decouple the rewards calculation:

1. dequeue checkin: C from QC

2. calculate rewards(C) -> R

3. store(R)

4. queue(R) on QR

Rewards calculation: win

• You win big.

• If the rewards calculation system is

• too slow, or

• goes offline

• checkins still proceed and

• responses are served within 500ms

• You have decoupled the service availability requirements of the checkin system from the rewards system: happier users.

Final random thoughts

think outside of the box

Things to look at: free your mind

• Node.js

• Javascript? Seriously?

• Yes.

• Forces you to think asynchronously

• Forces you to share nothing

• Forces you to build stateless systems

• These systems scale

unsafe: when to use

• “silence is success” messaging is almost always useful when new, more temporally relevant data is bound to arrive.

• game location data

• performance data

• status data

• the casual observer

be mindful

• Always monitor:

• message rates

• queue depths

• queue counts

• connection concurrency

Thank you.

• Thank you

• Merci beaucoup.

Building Scalable Systems: an asynchronous approach

Technology

A Hybrid Scheduling Approach for Scalable Heterogeneous ...datasys.cs.iit.edu/events/MTAGS12/s04.pdfA Hybrid Scheduling Approach for Scalable Heterogeneous Hadoop Systems Authors:

An Asynchronous Approach To Teaching Math And … asynchronous approach to teaching Math and Engineering software within the context of a course in ... MathSoft's Mathcad and MSC's

sAVSS: Scalable Asynchronous Veriﬁable Secret Sharing in

Virtual Literacy Instruction Guide · 2020. 8. 13. · 5 Synchronous vs Asynchronous Instruction Synchronous Real-Time Instructional Approach Asynchronous Learner-Centered Approach

A compositional approach to scalable Bayesian computation ...ndjw1/docs/djw-acmll.pdf · A compositional approach to scalable Bayesian computation and probabilistic programming Darren

an approach to building scalable platforms

A SCALABLE SEMIDEFINITE RELAXATION APPROACH ...rm3122/paper/ac_scheduling.pdfA SCALABLE SEMIDEFINITE RELAXATION APPROACH TO GRID SCHEDULING RAMTINMADANI,ALPERATAMTURK,ANDALIDAVOUDI¨

A Geodetic Approach to Scalable Visualization without ...hydrometronics.com/downloads/ECEF for Hydro full screen.pdf · A Geodetic Approach to Scalable Visualization without Distortion

A systemic approach toward scalable, reliable and safe

Scalable Asynchronous Gradient Descent Optimization for Out-of … · 2019-07-12 · Scalable Asynchronous Gradient Descent Optimization for Out-of-Core Models Chengjie Qinz Martin

Consent Management: A Scalable Nationwide Approach

Scalable Cloud Security via Asynchronous Virtual Machine … · 2019-12-18 · Scalable Cloud Security via Asynchronous Virtual Machine Introspection Sundaresan (sunny) Rajasekaran,

A stream computing approach towards scalable NLP

Scalable Asynchronous Contact Mechanics using Charm++ · 2015-07-22 · Scalable Asynchronous Contact Mechanics using Charm++ Xiang Ni, Laxmikant V. Kale and Rasmus Tamstorfy Department

Project Risk Management Handbook: A Scalable Approach

Towards Scalable Quantum Communication and Computation ... · approaches towards scalable quantum information processing. First, we consider a hybrid approach to scalable quantum

SCALABLE DESIGN APPROACH TO ANALYZE FLIGHT …

A Scalable Systems Approach for Critical Infrastructure

A Scalable Approach to Exact Model

Lightcuts: A Scalable Approach to Illumination - Cornell University