Embed Size (px)
Citation preview
It Sounded Good on PaperJeffery Smith
Manager, Site Reliability Engineering |GrubHub @DarkAndNerdy
Bio - Jeff Smith• Manager, Site Reliability
Engineering at GrubHub
• Yes, we are also hiring
• Yes, there is free food. Yes it’s totally awesome to work here
Email: [email protected]
Twitter: @DarkAndNerdy
Blog: www.allthingsdork.com
The Puppet Timeline
Reduce the Pain Through Sharing
10 11 12 Tips on What We Did Wrong
What Puppet Manages
Tip #1 - Minimize App Specific Config
• Drives up the number of roles to maintain • Can slow down developers if they don’t also write
Puppet Code • Pushes deployment responsibilities into Configuration
Management
Draw the line
• Deployment process should handle deployment type tasks
• Your roles can be more generic, like “Tomcat Server” or “Web Server” vs “Payroll_App_Server”
Tip #2 - Code Base Structure Matters
• Avoid doing too much in a module. It’s OK if your module makes assumptions, just document them
include apache::install
include apache::config
apache::vhost { ‘website’:
ensure => installed
}
apache::vhost { ‘website’:
ensure => installed,
}
Assumes nothing Assumes Apache is Installed
Tip #2 - Code Base Structure Matters
• Use the modules/profiles/roles pattern - Gary Larizza • A single code base regardless of environments
Tip #3 - Everyone is watching!
• Passwords in codebase prevents you from sharing
• Will an auditor throw-up in their mouth?
• That sense of shame you feel is probably a real issue
Tip #4 - A Repo Per Module
• In a single Repository, versioning becomes complicated
• Changes become inter-mingled • Automated testing requires some inspection of
changes to save on test time.
Speaking of Testing
• Not all testing is valuable. Basic Puppet operations aren’t worth testing
• Keep tests fast or you’ll get burned at 2am during an emergency patch
Tip #5 - Never Hand Touch a Server
Always build the server from scratch, using Puppet only. If you make a single change, destroy and rebuild.
Tip #6 - There are only 3 numbersWhen writing your manifests, you should think of your resources in these terms
• 0 - This resource can never exist • 1 - There can only ever be one of these resources • Many - There will be 1 or more of these resources
If you think there’s only 2 of a thing, someone is secretly planning the 3rd
Tip #7 - Exported Resources are EvilNot Entirely
• Attempt to solve the problem with PuppetDB Query • Move the exported resource to a runtime config store • If necessary, develop clear patterns for tagging
resources. It aids in collection
Tip #8 - Branching Strategies - Ugh
Gotchas with this approach
• Need to blow away develop branch and clone with master semi-regularly
• Bad change can break all of pre-prod (OOPS) • Merge conflicts in Develop that don’t exist in PROD • Do you do PRs for Develop? Slows things down
Alternatives• Separate Branch for each server environment?
• Need to keep in sync with master • Makes knowledge of changes local to the SRE that
made them • Always work off master
• Added risk • Tons of trash commits
Tip #9 - Be realistic about your secrets
• Use Hiera eyaml - https://github.com/TomPoulton/hiera-eyaml
• If you’re using JSON, get over it, convert to YAML and use Hiera eyaml. https://www.npmjs.com/package/yamljs
• Key distribution is still tricky
Your Rube Goldberg solution is useless if it ends in a plain text file being loaded
Key Distribution• It sucks everywhere. You’re not alone • Just because it sucks, doesn’t mean you can store
keys in Source Control • Puppet File Server has some options
• Allow by IP address or hostname • Distribution to Puppet Master varies :-(
Puppet File Server[ssl_keys]
path /secure/ssl_keys
allow 10.160.10.100
deny *.testdomain.com
file { ‘/etc/httpd/keys/ssl.key’:
source => ‘puppet:///ssl_keys/ssl.key'
}
Tip #10 - Build Monitoring in PuppetFind a Puppet module for Nagios or build one, but put your monitoring in Puppet
• Weird things happen in ALL environments • One Sys Admins test environment is a developer’s
production environment • Provisioning a new node and implementing monitoring
shouldn’t be two separate steps
Tip #11 - Stop Using ‘exec’
• Execs take the elegance and readability out of Puppet Code, especially when executing shell scripts
• You probably don’t do a good job of only triggering the exec when needed
• Congrats! Your script is idempotent……..but…
Tip #12 - Use Hiera
• CASE statements are often a code smell • Hiera allows you to use the same code for all
environments • Don’t be afraid of Hiera structures, but whatever you
do for the love of God don’t number them
Example Hiera Structure • Nodes
• node/certname
• Environment
• <environment names>
• DataCenter
• <datacenters>
• Performance Profile
• local, integration, staging, prod
• Globals
