Damn Abstraction : Development vs Operational Efficiency

Patrick Debois

goto;conference

Copenhagen2011

Wednesday 11 May 2011

http://jedi.be/blog monitoring

http://devopsdays.orgWednesday 11 May 2011

Project TaskWe need performance!

1 million writes /5 seconds

“All problems in computer science can be solved by another level of abstraction”

“All problems in computer science can be solved by another level of indirection”

Butler Lampson

http://zedshaw.com/essays/indirection_is_not_abstraction.html

We love Ruby & Rails

Performance Testing

JmeterApache

BenchmarkTsung

Rely much on OS tuningLimitations (tcp sessions, max fd, sockets,...)

Rails Application

Ruby (threading)

OS (threading)

Active Record

NgnixApache, Passenger

Thin Unicorn,Rainbows

Event Machine

Threads, Fibers, PIL

FS, Network tuningWednesday 11 May 2011

Scaling

Web ServerApache/Nginx

MultiInstance

Loadbalancer

Database Pool

“... except for the problem of too many layers of indirection”

David Wheeler

Magic?

curl http://npmjs.org/

install.sh | sh

https://github.com/jordansissel/fpm/wiki/ConvertingNodeNPM

But Seriously??

bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)

https://github.com/jordansissel/rvm-packaging

Brew install

curl -LsSf https://github.com/mxcl/homebrew/tarball/master | sudo /usr/bin/tar xvz -C/usr/local --strip 1

https://github.com/sstephenson/gosh

gosh: Security theater for piped shell scripts

Select Database?

Logging

http://logging.apache.org/log4j/http://log4r.rubyforge.org/

https://github.com/indexzero/winston

https://github.com/cloudera/flume

https://github.com/facebook/scribe

http://www.loggly.com/

Dependency Mgmt

Capistrano

Warbler

BundlerMaven

Cargo Plugin

Fat Jar

“But that usually create another problem”

David Wheeler

http://www.dmst.aueb.gr/dds/pubs/inbook/beautiful_code/html/Spi07g.html

“Every abstraction layer adds (a little) overhead

to the CPU

but also to the poor human who has to read

that code

http://blog.billyang.me/2009/10/dark-side-of-abstraction.html

Dark Side of Abstraction

+++++++ +

Improvements

Runtime Abstraction

Database Abstraction

Do compilers, Languages, and frames

make dumber programmers?

http://stackoverflow.com/questions/428571/do-smarter-compilers-languages-and-frameworks-make-dumber-programmers

They don't make good programmers dumber, they

make them more productive;

But they make dumber people be able to program

and think they are great programmer

Sysadmin Land

Hardware Abstraction(OS/Drivers)

Disk Abstraction

Filesystem Abstraction

Iscsi NAS

reiserfs

VM/Storage Abstraction

http://libvirt.org/

OpenVZ

Usermode Linux

Virtualbox

Vmware

Disk Images

Cloud Abstraction

https://github.com/geemus/fog https://github.com/jclouds/jclouds

Abstraction Fusion

Config ManagementAbstraction

A model is an abstraction ... for a purpose

http://www.cs.princeton.edu/courses/archive/spr98/cs333/lectures/19/sld002.htm

http://docs.puppetlabs.com/guides/cloud_pack_getting_started.html

Puppet Cloudpack

https://github.com/opscode/knife-ec2

Knife ec2 Bootstrap

If all you have is a hammer

http://www.amusei.com/images/hammer-in-the-head1.jpg

Orchestration

http://docs.puppetlabs.com/mcollective/ http://rundeck.org/

http://www.iinteractive.com/yapc/slides.htmlThank you @jtimberman

Vagrant

http://vagrantup.com/

https://github.com/jedi4ever/veewee

https://github.com/jedi4ever/sahara

http://vagrantbox.es/

https://github.com/jedi4ever/mccloud

http://www.jedi.be/blog/2011/03/28/using-vagrant-as-a-team/

Packages Boundaries

tgzEffing Package Management

Source/Binary

wget ...tar -xzvf ...

cd ..../configure

makemake install

package install

External Download => Local RepoCompiling => Do it only once

Idempotent

“Abstraction is

selective ignorance”

Andrew König

http://en.wikipedia.org/wiki/Andrew_Koenig_(programmer)

Google App Engine

Cloud Stacks

Cloud API Standardisation

Result is suboptimization

http://www.michaelnygard.com/blog/2004/04/too_much_abstraction.html

Too much abstraction(1)

Do the Simplest Thing that could possibly work

http://c2.com/cgi/wiki?TooMuchAbstraction

Too much abstraction(2)

Abstraction should be based on Practical Experience

Premature Optimization

http://c2.com/cgi/wiki?TooMuchAbstraction

Abstraction is deciding which aspects of a

problem to consider and which ones to ignore

Sound a lot likeManagement

(without domain knowledge?)

Organization Abstractions

Something’s Fishy

http://weblog.raganwald.com/2007/12/somethings-fishy.html

Devops

http://mdzlog.alcor.net/2010/06/08/devops-and-cloud/

Devops

http://rationalautomation.wordpress.com/2010/02/04/dogs-and-cats-living-together/

Devops

http://www.rajiv.com/blog/2009/03/17/technology-department/

Generalist vs Specialist

Consider Specialization First

... And Generalization Second

http://www.noop.nl/2010/06/t-shaped-people.html

They have a principal skill that describes

the vertical leg of the T

T-shaped People

But they are so empathetic that they can branch out into other skills, such as anthropology, and do them as well.

http://images.icanhascheezburger.com/completestore/2008/9/11/128656414813278195.jpg

NOOPS = Human Abstraction?

http://rlv.zcache.com/i_love_noops_t_shirt-p235186382091096346trlf_400.jpg

abstraction is selective ignorance

http://blog.rightscale.com/2011/04/25/amazon-ec2-outage-summary-and-lessons-learned/

Amazon.com’s real problem isn’t the outage, it’s the communication

http://aws.amazon.com/message/65648/

The Chaos Monkey’s job is to randomly kill instances and services within our architecture

If we aren’t constantly tes0ng our ability to succeed despite failure, then it isn’t likely to work when it ma;ers most – in the event of an unexpected outage.

http://www.codinghorror.com/blog/2011/04/working-with-the-chaos-monkey.html

Chaos Monkey

https://github.com/simonmunro/ChaosMonkey

https://github.com/mattbasta/chaos-chimp

__ w c(..)o ( \__(-) __) /\ ( /(_)___) w /| | \ m m

All non-trivial abstractions, to some

degree, are leaky

http://www.joelonsoftware.com/articles/LeakyAbstractions.html

When you drive a car, thinking about howthe engine works is

a distraction

When you repair the car, thinking about

how the engine works is essential.

Questions?

Programming is a human activity;

forget that and all is lost

Meta-advicehttp://www.cs.princeton.edu/courses/archive/spr99/cs333/pdf/21.pdf

Damn Abstraction : Development vs Operational Efficiency

Technology

Lies, damn lies, estimates

DAMN Magazine

Improving Test Suites via Operational Abstraction

plan the damn

Damn Heels S/S 2013 Damn Bright & Beautiful

big damn hero

Visite à la DAMN

Damn Content Marketing

Damn Small Linux

Damn Yankees Libretto

Damn Yankees (Adler_Ross) (Revival)

Revista DAMN

So damn beautiful..x

Damn ! Data !

Damn Computer

DAMn Design Magazine

Damn Simics

03 damn you autocorrect

Abstraction How much Abstraction ? ABSTRACTIONREALISM

Damn Ph.D