How build and deploy affects architecture at thetrainline.com - akqa anoraks

#akqaanoraks @matthewpskelton

How build and deployment shapes software architecture

at thetrainline.com

Matthew Skelton | thetrainline.com

AKQA Anoraks, London | #akqaanoraks

29 November 2012

#akqaanoraks @matthewpskelton

architecture

= f (build & deploy)

(for some systems)

#akqaanoraks @matthewpskelton

Systems engineering (robotics, control theory, sensors, neuroscience)

Software development

(finance, insurance, travel, pharma, media, medical imaging)

now

Build & Deployment at thetrainline.com

#akqaanoraks @matthewpskelton

#akqaanoraks @matthewpskelton

9m visits per month to thetrainline.com

2nd most visited UK travel booking website

(behind Easyjet)*

Over 100,000 daily visits via our mobile apps

*source: IMRG hotshops list Nov 2011

#akqaanoraks @matthewpskelton

Booking engines for many UK train operators

Ticketing systems for Corporates and TMCs

Several hundred individual websites

> £1bn ($1.5bn USD) per year in train ticket sales

#akqaanoraks @matthewpskelton

#akqaanoraks @matthewpskelton

• Early “distributed agile” success story

• http://www.thoughtworks.com/trainline

#akqaanoraks @matthewpskelton

10+ dev teams in 2 locations

#akqaanoraks @matthewpskelton

Server Farm (Build, Deployment, Replication)

100+ build agents 20 dev environments x 20 VMs per

environment = 400 dev VMs Git replication (Gitolite), …

MORE SERVERS THAN PRODUCTION

#akqaanoraks @matthewpskelton

#akqaanoraks @matthewpskelton

#akqaanoraks @matthewpskelton

#akqaanoraks @matthewpskelton

architecture

= f (build & deploy)

(for some systems)

#akqaanoraks @matthewpskelton

“HERESY!”

#webperfdays @matthewpskelton

#akqaanoraks @matthewpskelton

#akqaanoraks @matthewpskelton

#webperfdays @matthewpskelton

#akqaanoraks @matthewpskelton

#webperfdays @matthewpskelton

#akqaanoraks @matthewpskelton

#akqaanoraks @matthewpskelton

#webperfdays @matthewpskelton

#akqaanoraks @matthewpskelton

#akqaanoraks @matthewpskelton

Web-based

Frequently- changing

Public-facing

High-volume

#akqaanoraks @matthewpskelton

#akqaanoraks @matthewpskelton

RELIABLE

REPEATABLE

RAPID

RECURRING #webperfdays

@matthewpskelton #akqaanoraks

@matthewpskelton

#akqaanoraks @matthewpskelton

‘R-R-R-R’ BUILD AND DEPLOYMENT

Helps to avoid the Ball of Mud

Enables high-performance architectures

#webperfdays @matthewpskelton

#akqaanoraks @matthewpskelton

#akqaanoraks @matthewpskelton

#webperfdays @matthewpskelton

#akqaanoraks @matthewpskelton

#akqaanoraks @matthewpskelton

LOAD BALANCING HIGH AVAILABILITY

SCALING ELASTIC

RAPID RECOVERY

#webperfdays @matthewpskelton

#akqaanoraks @matthewpskelton

#akqaanoraks @matthewpskelton

BUILDABLE

Small pipelined builds on generic build machines

Seconds, not minutes or hours

Short feedback cycles (Dan Worthington-Bodart, @danielbodart - http://bit.ly/M85wsX)

#webperfdays @matthewpskelton

#akqaanoraks @matthewpskelton

#akqaanoraks @matthewpskelton

Build dependencies may mirror runtime deps

msbuild /m for multi-threaded builds (forces component isolation)

Gallio (MbUnit) – replacement engine for NUnit

NuGet + Squid for super-fast build artefacts

RAM disk for some working copy operations

#akqaanoraks @matthewpskelton

TESTABLE

Test (separation, harnesses, points)

IDENTIFIABLE

Meaningful versions, packages, defined dependencies, artefact

management

(think component boundaries)

#webperfdays @matthewpskelton

#akqaanoraks @matthewpskelton

#akqaanoraks @matthewpskelton

Enforced “onion” testing layers (test separation, build machine firewalls, build agent types, …)

Semantic Versioning (A.B.C.D + contract to ‘clients’)

30+ external services (test harnesses)

#akqaanoraks @matthewpskelton

DEPLOYABLE Rapid, scriptable, simple failure modes

MONITORABLE Logging, metrics, transaction tracing

CONFIGURABLE

Inject settings – no ‘black boxes’

LIGHTWEIGHT Keep things small and easily comprehendible

#webperfdays @matthewpskelton

#akqaanoraks @matthewpskelton

#akqaanoraks @matthewpskelton

GO, MSDeploy, hand-rolled, Octopus, PS remoting (“as simple as xcopy”)

Graphite + SqlToGraphite (https://github.com/perryofpeek/SqlToGraphite)

LogStash + log4net (tune logging levels with config, transaction tracing)

SCOM – operations manager (early monitoring in Dev)

#akqaanoraks @matthewpskelton

INSTANTIABLE

No snowflakes or singletons

RECOVERABLE

No nasty zombies after failures

MTTR more important than MTBF* * for most kinds of F

#webperfdays @matthewpskelton

#akqaanoraks @matthewpskelton

#akqaanoraks @matthewpskelton

Rebuild Build Agents with Chef

Chef for environment rebuilds

Call Start() and Stop() on each component

#akqaanoraks @matthewpskelton

#webperfdays @matthewpskelton

#akqaanoraks @matthewpskelton

#akqaanoraks @matthewpskelton

Well-known monolithic business rules engine

Costly to build, deploy, configure, test

Being replaced

#akqaanoraks @matthewpskelton

RELIABLE

REPEATABLE

RAPID

RECURRING #webperfdays

@matthewpskelton #akqaanoraks

@matthewpskelton

#akqaanoraks @matthewpskelton

LOAD BALANCING HIGH AVAILABILITY

SCALING ELASTIC

RAPID RECOVERY

#webperfdays @matthewpskelton

LIGHTWEIGHT TESTABLE MONITORABLE CONFIGURABLE RECOVERABLE IDENTIFIABLE

#akqaanoraks @matthewpskelton

#akqaanoraks @matthewpskelton

London Continuous Delivery

Next meetup: 17 January 2013, with

Opscode (Chef)

#londoncd

#akqaanoraks @matthewpskelton

architecture

= f (build & deploy)

(for some systems)

thank you

@AKQAAnoraks

Blog: http://engineering.thetrainline.com/

matthewskelton.net | @matthewpskelton

Thanks to: Attila S, Jack R and Owain P for feedback.

Picture credits:

Label: seanlabel.com; Sheep: gallery.hd.org; Train: worldontrains.blogspot.co.uk; Gurner: mog.com; Petra: Wikimedia/Berthold Werner; army engineers: US DoD; ball of mud: pwern.blogspot.co.uk; sports car: xarj.net; zombie: bjj.org; feather: Wikipedia; punch:

thelegalblitz.com; passport: coverpalace.com; dogs: reluctantmemsahib.wordpress.com; Meccano: dalefield.com