Scaling Deployment at EtsyDaniel Schauenberg

dschauenberg@etsy.com@mrtazz

Thursday, April 18, 13

Thursday, April 18, 13

the marketplace we make together

Thursday, April 18, 13

February 2013

• 1.49 billion page views

• 4,215,169 items sold

• $94.7 million of goods sold

• 22+ million members

• 800,000+ active shopshttp://www.etsy.com/blog/news/2013/etsy-statistics-february-2013-weather-report/ | Items by RockerDollJewellery, ZulamimiLand, codice, 42Things

Thursday, April 18, 13

LAMMP

Item by TheBackPackShoppe

Thursday, April 18, 13

Item by FrankelPhotos

Monolithic App

Thursday, April 18, 13

No Branching

Item by NurseryWallArt

Thursday, April 18, 13

Deploy Frequency

Thursday, April 18, 13

First Day

Item by flowersandfleurons

Thursday, April 18, 13

Thursday, April 18, 13

IRC

Thursday, April 18, 13

Thursday, April 18, 13

Developer VMs

• KVM

•Dev version of full Etsy stack

•Chef

•DevTools

Thursday, April 18, 13

Thursday, April 18, 13

Item by codecards

Thursday, April 18, 13

% review -r dschauenberg

Thursday, April 18, 13

automatically assignedautomatically assigned

Thursday, April 18, 13

Try

Item by CSSDesign

Thursday, April 18, 13

Actually ...

Thursday, April 18, 13

CI - The Old Way

• ~10 buildtest servers, single SSD

• running multiple Jenkins executors

• tests pegging IO

• Jenkins is much better at running single executors

Thursday, April 18, 13

The Bobs

Item by Signz

Thursday, April 18, 13

The Bobs

• LXC containers on buildtests

•Multiple SSDs

• Labels for heavy/any execution

•One heavy executor per disk

Thursday, April 18, 13

CI/Try

• ~200 Bobs

•Mostly for try

•Constant monitoring for slow tests

Thursday, April 18, 13

push train

Item by decomodwalls

Thursday, April 18, 13

#push

• IRC channel to organize push trains

• Join a train if you want to deploy changes

• Schedule is planned via the channel topic

• First in the train is the driver (controls the deploy)

•Opening hours: 7am - 10pm NYC time

Thursday, April 18, 13

#push

<prod> kseever* + jameslee | jpaul | DanielConvissor (c)

Thursday, April 18, 13

#push

bateman* + krunal* + enorris* | tristan (c) + jameslee (c) + jlaster (c) | dawa + corey + sandosh + jklein + magera + seth_home + mpascual + nathan | bateman | russp (c)

Thursday, April 18, 13

pushbot

• .join

• .in

• .good

• .done

Thursday, April 18, 13

pushbot

Thursday, April 18, 13

https://github.com/etsy/PushBot

Thursday, April 18, 13

Item by EsalonPhotography

Thursday, April 18, 13

Deployinator

Thursday, April 18, 13

Thursday, April 18, 13

Thursday, April 18, 13

stale commits

Thursday, April 18, 13

version checks

buttons disabledbuttons disabled

Thursday, April 18, 13

version checks

Thursday, April 18, 13

lock down deploys

Thursday, April 18, 13

https://github.com/etsy/deployinator

Thursday, April 18, 13

Dashboards

deploy linesdeploy lines

Thursday, April 18, 13

https://github.com/etsy/dashboard

Thursday, April 18, 13

Supergrep

Thursday, April 18, 13

https://github.com/etsy/supergrep

Thursday, April 18, 13

Thursday, April 18, 13

Summary

•Current setup has scaled to ~150 people

•Constantly trying to improve the speed of deployment

• Find weak parts in the process and make them more robust/faster

• Bring Dev closer to Prod

•Not being able to deploy has the same status as the site being down

Thursday, April 18, 13

codeascraft.etsy.comwww.etsy.com/codeascraft/talks

etsy.github.comwww.etsy.com/careers

Thursday, April 18, 13

Questions?

Thursday, April 18, 13

Scaling Deployment at EtsyDaniel Schauenberg

dschauenberg@etsy.com@mrtazz

Thursday, April 18, 13