41
Puppet Camp April 9 th 2013

Functional Hostnames and Why they are Bad

Embed Size (px)

DESCRIPTION

"Functional Hostnames and Why they are Bad" by Andrew Fong and Gary Josack of Dropbox at Puppet Camp SF 2013. Find a Puppet Camp near you and learn more about configuration management: puppetlabs.com/community/puppet-camp/

Citation preview

Page 1: Functional Hostnames and Why they are Bad

Puppet Camp April 9th 2013

Page 2: Functional Hostnames and Why they are Bad

What’s in a name?

Andrew Fong and Gary Josack [email protected] [email protected]

Page 3: Functional Hostnames and Why they are Bad

About Dropbox

•  Thousands of instances/servers

• Mostly Python Stack

•  EC2 and Large Self Datacenters

• Over a billion file syncs per day

•  Thousands of MySQL Shards

•  4 SREs and 1 DBA

Page 4: Functional Hostnames and Why they are Bad

A story of a startup...

Hostapuppet.com

Page 5: Functional Hostnames and Why they are Bad

Chapter One

•  1 or 2 teams

•  Couple of hosts

• Webserver and a database

• Maybe one ops guy

Page 6: Functional Hostnames and Why they are Bad

What Ops People Like

•  Simplicity

•  Repeatability

•  Assurances that things remain consistent

•  Puppet / Configuration management

Page 7: Functional Hostnames and Why they are Bad

Config Management

node 'www1.example.com' { include common include apache include squid } node 'db1.example.com' { include common include mysql }

Page 8: Functional Hostnames and Why they are Bad

My First Puppet Config

Node ‘mickey.hostapuppet.com’ { include common include webserver include sudoers

} Node ‘donald.hostapuppet.com’{

include common include db include sudoers

}

Page 9: Functional Hostnames and Why they are Bad

Sudoers Module

… file { "/etc/sudoers": owner => root, group => root, mode => "440", source => "puppet:///modules/sudo/sudoers", } …

Page 10: Functional Hostnames and Why they are Bad

Sudoers File itself

Host_Alias DONALD=donald.hostapuppet.com

Host_Alias MICKEY=mickey.hostapuppet.com

db_guy DONALD=(all) NOPASSWD: ALL

ops_guy MICKEY=(all) NOPASSWD: ALL

Page 11: Functional Hostnames and Why they are Bad

Chapter Two: A growing service

•  A few teams

•  2 or 3 services

• multiple types of hosts

–  Web

–  API

–  DB

Page 12: Functional Hostnames and Why they are Bad

Hostnames

•  sjc-web[1-N]

•  sjc-db[1-N]

•  sjc-api[1-N]

Page 13: Functional Hostnames and Why they are Bad

Host Regex

$hosttype = inline_template('<%= hostname.sub(/\w+-([a-z]+)\d*/){$1} %>’)

Page 14: Functional Hostnames and Why they are Bad

Hosttypes $hosttype = inline_template('<%= hostname.sub(/\w+-([a-z]+)\d*/){$1} %>’)

if $hosttype == ‘web’ { include sudoers include web

} If $hosttype == ‘db’ {

include sudoers include db

}

if $hosttype == ‘api’ { include sudoers include api

}

Page 15: Functional Hostnames and Why they are Bad

Back to sudoers

… file { "/etc/sudoers": owner => root, group => root, mode => "440", source => "puppet:///modules/sudo/sudoers", } …

Page 16: Functional Hostnames and Why they are Bad

Sudoers File itself

Host_Alias WEB=sjc-web*

Host_Alias DB=sjc-db*

Host_Alias API=sjc-api*

database_guy DB=(all) NOPASSWD: ALL

ops_guy WEB=(all) NOPASSWD: ALL

api_team API=(all) NOPASSWD: ALL

Page 17: Functional Hostnames and Why they are Bad

Hypergrowth

0

20

40

60

80

100

120

0 2 3 5

Users (millions)

Users (millions)

Page 18: Functional Hostnames and Why they are Bad

Chapter 3: An Expanding Infrastructure

•  Lots of new hires!

•  A bunch more developers

•  Some PMs

•  Some Designers

Page 19: Functional Hostnames and Why they are Bad

All Kinds Of Problems…

•  Boxes of same hardware class running

different services

•  Boxes serving more than one role

(remember sudoers?)

• Deploying or moving hosts quickly

Page 20: Functional Hostnames and Why they are Bad

Renaming a host

• Update dns

• Update dhcpd.conf

•  Push both

• Update puppet configs

• Update code

Page 21: Functional Hostnames and Why they are Bad

OMG I JUST RENAME HOSTS!

Page 22: Functional Hostnames and Why they are Bad

Sudoers File From Chapter Two…

Host_Alias WEB= sjc-web* Host_Alias API=sjc-api* Host_Alias DB=sjc-db* database_guy DB=(all) NOPASSWD: ALL ops_guy WEB=(all) NOPASSWD: ALL api_team API(all) NOPASSWD: ALL

Page 23: Functional Hostnames and Why they are Bad

Sudoers File in Chapter 3

Host_Alias WEB= sjc-web* Host_Alias API=sjc-api*,sjc-web550,sjc-web551,sjc-web552,sjc-web553 Host_Alias DB=sjc-db* database_guy DB=(all) NOPASSWD: ALL ops_guy WEB=(all) NOPASSWD: ALL api_team API(all) NOPASSWD: ALL

Page 24: Functional Hostnames and Why they are Bad

Dropbox

• We did all that.

• We’re still paying the taxes for doing

that.

•  But there is a light at the end of the

tunnel…

Page 25: Functional Hostnames and Why they are Bad

ABSTRACT THE SERVICE

FROM THE HOST!

Page 26: Functional Hostnames and Why they are Bad

So what does that mean?

• Make hosts role agnostic

• Do not require invasive changes

• Simple interfaces

Page 27: Functional Hostnames and Why they are Bad

Making hosts role agnostic

• Positional

• Serial Numbers

• Anything that doesn't change

Page 28: Functional Hostnames and Why they are Bad

The Dropbox Plan

• Positional names

• Custom Machine Database

• External Node Classifier

• Transitioning Puppet configs

• Naming service(s) for convenient names

Page 29: Functional Hostnames and Why they are Bad

Service/Machine Management Database

• Universal Source of Truth

• Manage roles / attributes

• Generated configs

- Gmond, Nagios, etc

Page 30: Functional Hostnames and Why they are Bad

What exactly is the ENC

• External Node Classifier

• Inject variables (and other) from external

process

• YAML Output

Page 31: Functional Hostnames and Why they are Bad

Part 2: External Node Encoders

Page 32: Functional Hostnames and Why they are Bad

Sudoers++

• Move from monolithic to modular

• Includes! (Weird caveats)

• Just use ALL for Host_Lists

Page 33: Functional Hostnames and Why they are Bad

Sudoers at Dropbox

Page 34: Functional Hostnames and Why they are Bad

Part 3: Helper Functions

Page 35: Functional Hostnames and Why they are Bad

Sudoers with tags

Page 36: Functional Hostnames and Why they are Bad

Sudoers with tags

Page 37: Functional Hostnames and Why they are Bad

Provisioning

• Preload MDB, DNS, DHCPD, etc.

- Set it and forget it

• Have spares ready for any roles

• Assigning a role is one command

• No more renames!

Page 38: Functional Hostnames and Why they are Bad

Dynamic Naming w/ PowerDNS

Page 39: Functional Hostnames and Why they are Bad

Dynamic Naming w/ PowerDNS

Page 40: Functional Hostnames and Why they are Bad

Zookeeper

• ZKNS included with the Vitess project

• ZK is in use at various different companies (YouTube, Twitter, AirBnB)

Page 41: Functional Hostnames and Why they are Bad

Q&A

λ FAQ #1: Are you hiring? - Yes! Come talk to us. :)