Upload
puppet-labs
View
11.646
Download
5
Tags:
Embed Size (px)
DESCRIPTION
Russ Johnson of StubHub talks about "Learning Lessons Scaling to 5000 Puppet Agents" at Puppet Camp San Francisco 2013. Find a Puppet Camp near you: puppetlabs.com/community/puppet-camp/
Citation preview
Confidential Slide 1
Learning Lessons Scaling to 5000 Agents
Russ Johnson [email protected]
@professoruss
April 9, 2013 2
Started out in the mid 90’s
Recovering Windows Admin
Storage Guy
Datacenter Monkey
Once upon a time network guy
At StubHub since December 2006
Working on puppet adoption in a crazy infrastructure
Puppet certified
#whoami
April 9, 2013 3
4 Puppet Masters
DEV/QA PROD
DR CORP
April 9, 2013 4
The road to sanity
April 9, 2013 5
Apache/Passenger Tune Passenger
PassengerMaxPoolSize 32
PassengerMinInstances 4
PassengerMaxRequests 10000
PassengerStatThrottleRate 30
16 cores, 32GB load average: 4.03, 3.71, 3.45
4000+ agents
Set up your master properly
April 9, 2013 6
Thundering herds
April 9, 2013 7
Old way:
systemimager, vmware clones, manual installs
Results:
INCONSISTENCY!
Build your hosts the same way!
April 9, 2013 8
New way:
Cobbler < 5m bare metal to on the network
Results:
Same results every time! No drift between base
Build your hosts the same way!
April 9, 2013 9
Geppetto – eclipse based IDE
http://cloudsmith.github.com/geppetto/index.html
VIM
• Pathogen – For autoloading vim plugins
• Snipmate – Snippets
• Tabular – Text filtering and alignment
• Syntastic – Syntax checking
• mv-vim-puppet – Make vim puppet friendly
• puppet-lint – Syntax checker (gem)
Set up your working environment properly
April 9, 2013 10
Syntastic/puppet-lint
April 9, 2013 11
Set up your working environment properly
+
April 9, 2013 12
Ever do a 4 way diff across 60 modules to find most of them different?
Version Control is not enough
April 9, 2013 13
What to do?
Dynamic Environments?
Internal Forge?
Versioning?
How do I deal with 80+ Dev/QA Environments?
Branching?
Puppet Module Tool?
What does PuppetLabs do?
How do I stop manual edits?
Pulp?
Librarian?
April 9, 2013 14
PuppetLabs seems to know what to do
Let’s investigate puppet module tool
http://docs.puppetlabs.com/puppet/2.7/reference/modules_publishing.html
April 9, 2013 15
Generate a module
http://docs.puppetlabs.com/puppet/2.7/reference/modules_publishing.html
April 9, 2013 16
Edit Modulefile
http://docs.puppetlabs.com/puppet/2.7/reference/modules_publishing.html
April 9, 2013 17
Document the manifest
http://rdoc.sourceforge.net/
April 9, 2013 18
Write Documentation?
April 9, 2013 19
Free Docs!
puppet doc -a -o /var/www/html/puppetdocs --mode rdoc
April 9, 2013 20
What’s actually installed?
April 9, 2013 21
Catching live edits and preventing them
Splunk -> puppet module changes -> alerting
The NOC will hunt you down!
April 9, 2013 22
Build and install the module
tar –xzf /tmp/work/stubhub-puppetserver/pkg/stubhub-puppetserver-0.0.1.tar.gz –C /etc/puppet/environments/staging/modules/puppetserver
April 9, 2013 23
Releasing like that?
April 9, 2013 24
Internal Forge
mod_rewrite:
Simulate the api – redirect to json metadata files
$htmlroot/api/v1/releases.json?module=user/module
ruby script:
Generate metadata files for each module release and all modules.
Similar to createrepo (yum)
April 9, 2013 25
Internal Forge - Search
April 9, 2013 26
Internal Forge - install
April 9, 2013 27
Internal Forge - upgrade
April 9, 2013 28
Case statements? How bout Hiera?
April 9, 2013 29
Avoid case statement insanity
case $::system_role {
‘browse’, ‘search’: {
…do some stuff…
}
‘db’: {
…other stuff…
}
‘otherrole’: {
…please make it stop!!!!
}
}
April 9, 2013 30
hieradata
$hieradata/browse.yaml: ---
module::parameter: ‘foo’
$hieradata/search.yaml: ---
module::parameter: ‘bar’
$hieradata/defaults.yaml: ---
module::parameter: ‘I want this everywhere unless there are overrides’
April 9, 2013 31
Case -> variables -> hiera
§ 9000 lines of case statements
§ 1000 lines with case/variables
§ ~20 lines with defined type
Code compression FTW!
April 9, 2013 32
Dynamic Environments
§ puppet.conf:
modulepath = /etc/puppet/environments/$environment/modules
manifest = /etc/puppet/environments/$environment/manifests/site.pp
manifestdir = /etc/puppet/environments/$environment/manifests
§ hiera.yaml:
:datadir: '/etc/puppet/environments/%{environment}/hieradata'
April 9, 2013 33
Release process
§ Syntax check/validate
§ Test on VMs
§ Build module package
§ Release to internal forge
§ puppet module install to staging environment
§ Test again!
§ puppet module install to production environment
April 9, 2013 34
The road to yesop
§ Staging
§ Process
§ Repeatability
§ Consistency
§ Document everything
§ Breaking things where it’s cheap
§ Test everything!
April 9, 2013 35
Then VS now
§ Environment build time: – Then: 3+ weeks
• It was wrong
• It didn’t work
• Nobody knew what to expect
– Now: < 1 day • It’s the same every time
• We know exactly what’s installed
• Internal consumers get what they expect
• Less outages from human error
April 9, 2013 36
Questions?