Lessons I Learned While Scaling to 5000 Puppet Agents

Confidential Slide 1

Learning Lessons Scaling to 5000 Agents

Russ Johnson rjohnson@stubhub.com

@professoruss

April 9, 2013 2

Started out in the mid 90’s

Recovering Windows Admin

Storage Guy

Datacenter Monkey

Once upon a time network guy

At StubHub since December 2006

Working on puppet adoption in a crazy infrastructure

Puppet certified

#whoami

April 9, 2013 3

4 Puppet Masters

DEV/QA PROD

DR CORP

April 9, 2013 4

The road to sanity

April 9, 2013 5

Apache/Passenger Tune Passenger

PassengerMaxPoolSize 32

PassengerMinInstances 4

PassengerMaxRequests 10000

PassengerStatThrottleRate 30

16 cores, 32GB load average: 4.03, 3.71, 3.45

4000+ agents

Set up your master properly

April 9, 2013 6

Thundering herds

April 9, 2013 7

Old way:

systemimager, vmware clones, manual installs

Results:

INCONSISTENCY!

Build your hosts the same way!

April 9, 2013 8

New way:

Cobbler < 5m bare metal to on the network

Results:

Same results every time! No drift between base

Build your hosts the same way!

April 9, 2013 9

Geppetto – eclipse based IDE

http://cloudsmith.github.com/geppetto/index.html

VIM

•  Pathogen – For autoloading vim plugins

•  Snipmate – Snippets

•  Tabular – Text filtering and alignment

•  Syntastic – Syntax checking

•  mv-vim-puppet – Make vim puppet friendly

•  puppet-lint – Syntax checker (gem)

Set up your working environment properly

April 9, 2013 10

Syntastic/puppet-lint

April 9, 2013 11

Set up your working environment properly

+

April 9, 2013 12

Ever do a 4 way diff across 60 modules to find most of them different?

Version Control is not enough

April 9, 2013 13

What to do?

Dynamic Environments?

Internal Forge?

Versioning?

How do I deal with 80+ Dev/QA Environments?

Branching?

Puppet Module Tool?

What does PuppetLabs do?

How do I stop manual edits?

Pulp?

Librarian?

April 9, 2013 14

PuppetLabs seems to know what to do

Let’s investigate puppet module tool

http://docs.puppetlabs.com/puppet/2.7/reference/modules_publishing.html

April 9, 2013 15

Generate a module

http://docs.puppetlabs.com/puppet/2.7/reference/modules_publishing.html

April 9, 2013 16

Edit Modulefile

http://docs.puppetlabs.com/puppet/2.7/reference/modules_publishing.html

April 9, 2013 17

Document the manifest

http://rdoc.sourceforge.net/

April 9, 2013 18

Write Documentation?

April 9, 2013 19

Free Docs!

puppet doc -a -o /var/www/html/puppetdocs --mode rdoc

April 9, 2013 20

What’s actually installed?

April 9, 2013 21

Catching live edits and preventing them

Splunk -> puppet module changes -> alerting

The NOC will hunt you down!

April 9, 2013 22

Build and install the module

tar –xzf /tmp/work/stubhub-puppetserver/pkg/stubhub-puppetserver-0.0.1.tar.gz –C /etc/puppet/environments/staging/modules/puppetserver

April 9, 2013 23

Releasing like that?

April 9, 2013 24

Internal Forge

mod_rewrite:

Simulate the api – redirect to json metadata files

$htmlroot/api/v1/releases.json?module=user/module

ruby script:

Generate metadata files for each module release and all modules.

Similar to createrepo (yum)

April 9, 2013 25

Internal Forge - Search

April 9, 2013 26

Internal Forge - install

April 9, 2013 27

Internal Forge - upgrade

April 9, 2013 28

Case statements? How bout Hiera?

April 9, 2013 29

Avoid case statement insanity

case $::system_role {

‘browse’, ‘search’: {

…do some stuff…

}

‘db’: {

…other stuff…

}

‘otherrole’: {

…please make it stop!!!!

}

}

April 9, 2013 30

hieradata

$hieradata/browse.yaml: ---

module::parameter: ‘foo’

$hieradata/search.yaml: ---

module::parameter: ‘bar’

$hieradata/defaults.yaml: ---

module::parameter: ‘I want this everywhere unless there are overrides’

April 9, 2013 31

Case -> variables -> hiera

§  9000 lines of case statements

§  1000 lines with case/variables

§  ~20 lines with defined type

Code compression FTW!

April 9, 2013 32

Dynamic Environments

§  puppet.conf:

modulepath = /etc/puppet/environments/$environment/modules

manifest = /etc/puppet/environments/$environment/manifests/site.pp

manifestdir = /etc/puppet/environments/$environment/manifests

§  hiera.yaml:

:datadir: '/etc/puppet/environments/%{environment}/hieradata'

April 9, 2013 33

Release process

§  Syntax check/validate

§  Test on VMs

§  Build module package

§  Release to internal forge

§  puppet module install to staging environment

§  Test again!

§  puppet module install to production environment

April 9, 2013 34

The road to yesop

§  Staging

§  Process

§  Repeatability

§  Consistency

§  Document everything

§  Breaking things where it’s cheap

§  Test everything!

April 9, 2013 35

Then VS now

§  Environment build time: –  Then: 3+ weeks

•  It was wrong

•  It didn’t work

•  Nobody knew what to expect

–  Now: < 1 day •  It’s the same every time

•  We know exactly what’s installed

•  Internal consumers get what they expect

•  Less outages from human error

April 9, 2013 36

Questions?