View
8.350
Download
3
Category
Preview:
DESCRIPTION
"Functional Hostnames and Why they are Bad" by Andrew Fong and Gary Josack of Dropbox at Puppet Camp SF 2013. Find a Puppet Camp near you and learn more about configuration management: puppetlabs.com/community/puppet-camp/
Citation preview
Puppet Camp April 9th 2013
What’s in a name?
Andrew Fong and Gary Josack fong@dropbox.com gary@dropbox.com
About Dropbox
• Thousands of instances/servers
• Mostly Python Stack
• EC2 and Large Self Datacenters
• Over a billion file syncs per day
• Thousands of MySQL Shards
• 4 SREs and 1 DBA
A story of a startup...
Hostapuppet.com
Chapter One
• 1 or 2 teams
• Couple of hosts
• Webserver and a database
• Maybe one ops guy
What Ops People Like
• Simplicity
• Repeatability
• Assurances that things remain consistent
• Puppet / Configuration management
Config Management
node 'www1.example.com' { include common include apache include squid } node 'db1.example.com' { include common include mysql }
My First Puppet Config
Node ‘mickey.hostapuppet.com’ { include common include webserver include sudoers
} Node ‘donald.hostapuppet.com’{
include common include db include sudoers
}
Sudoers Module
… file { "/etc/sudoers": owner => root, group => root, mode => "440", source => "puppet:///modules/sudo/sudoers", } …
Sudoers File itself
Host_Alias DONALD=donald.hostapuppet.com
Host_Alias MICKEY=mickey.hostapuppet.com
db_guy DONALD=(all) NOPASSWD: ALL
ops_guy MICKEY=(all) NOPASSWD: ALL
Chapter Two: A growing service
• A few teams
• 2 or 3 services
• multiple types of hosts
– Web
– API
– DB
Hostnames
• sjc-web[1-N]
• sjc-db[1-N]
• sjc-api[1-N]
Host Regex
$hosttype = inline_template('<%= hostname.sub(/\w+-([a-z]+)\d*/){$1} %>’)
Hosttypes $hosttype = inline_template('<%= hostname.sub(/\w+-([a-z]+)\d*/){$1} %>’)
if $hosttype == ‘web’ { include sudoers include web
} If $hosttype == ‘db’ {
include sudoers include db
}
if $hosttype == ‘api’ { include sudoers include api
}
Back to sudoers
… file { "/etc/sudoers": owner => root, group => root, mode => "440", source => "puppet:///modules/sudo/sudoers", } …
Sudoers File itself
Host_Alias WEB=sjc-web*
Host_Alias DB=sjc-db*
Host_Alias API=sjc-api*
database_guy DB=(all) NOPASSWD: ALL
ops_guy WEB=(all) NOPASSWD: ALL
api_team API=(all) NOPASSWD: ALL
Hypergrowth
0
20
40
60
80
100
120
0 2 3 5
Users (millions)
Users (millions)
Chapter 3: An Expanding Infrastructure
• Lots of new hires!
• A bunch more developers
• Some PMs
• Some Designers
All Kinds Of Problems…
• Boxes of same hardware class running
different services
• Boxes serving more than one role
(remember sudoers?)
• Deploying or moving hosts quickly
Renaming a host
• Update dns
• Update dhcpd.conf
• Push both
• Update puppet configs
• Update code
OMG I JUST RENAME HOSTS!
Sudoers File From Chapter Two…
Host_Alias WEB= sjc-web* Host_Alias API=sjc-api* Host_Alias DB=sjc-db* database_guy DB=(all) NOPASSWD: ALL ops_guy WEB=(all) NOPASSWD: ALL api_team API(all) NOPASSWD: ALL
Sudoers File in Chapter 3
Host_Alias WEB= sjc-web* Host_Alias API=sjc-api*,sjc-web550,sjc-web551,sjc-web552,sjc-web553 Host_Alias DB=sjc-db* database_guy DB=(all) NOPASSWD: ALL ops_guy WEB=(all) NOPASSWD: ALL api_team API(all) NOPASSWD: ALL
Dropbox
• We did all that.
• We’re still paying the taxes for doing
that.
• But there is a light at the end of the
tunnel…
ABSTRACT THE SERVICE
FROM THE HOST!
So what does that mean?
• Make hosts role agnostic
• Do not require invasive changes
• Simple interfaces
Making hosts role agnostic
• Positional
• Serial Numbers
• Anything that doesn't change
The Dropbox Plan
• Positional names
• Custom Machine Database
• External Node Classifier
• Transitioning Puppet configs
• Naming service(s) for convenient names
Service/Machine Management Database
• Universal Source of Truth
• Manage roles / attributes
• Generated configs
- Gmond, Nagios, etc
What exactly is the ENC
• External Node Classifier
• Inject variables (and other) from external
process
• YAML Output
Part 2: External Node Encoders
Sudoers++
• Move from monolithic to modular
• Includes! (Weird caveats)
• Just use ALL for Host_Lists
Sudoers at Dropbox
Part 3: Helper Functions
Sudoers with tags
Sudoers with tags
Provisioning
• Preload MDB, DNS, DHCPD, etc.
- Set it and forget it
• Have spares ready for any roles
• Assigning a role is one command
• No more renames!
Dynamic Naming w/ PowerDNS
Dynamic Naming w/ PowerDNS
Zookeeper
• ZKNS included with the Vitess project
• ZK is in use at various different companies (YouTube, Twitter, AirBnB)
Q&A
λ FAQ #1: Are you hiring? - Yes! Come talk to us. :)
Recommended