Michelin Starred Cooking with Chef

Michelin Starred Cooking with Chef

Jon Cowie, [email protected]

@jonlives

mailto:[email protected]

mailto:[email protected]

What?

What?

• Chef at Etsy

What?

• Chef at Etsy

• Familiarity and Understanding

What?

• Chef at Etsy


• Critical Approach and Experimentation

What?

• Chef at Etsy



• Use The Source

What?

• Chef at Etsy



• Use The Source

• A liberal sprinkling of screwups

What?

• Chef at Etsy



• Use The Source


• Open Sourced Goodness - We’re all here!

What?

• Chef at Etsy



• Use The Source


• Open Sourced Goodness - We’re all here!

• [x] = http://tiny.cc/velocity2012

http://tiny.cc/velocity2012

http://tiny.cc/velocity2012

Opscode is Orange,Velocity is Blue.In Soviet Russia,

Cookbook writes you.

Chef at Etsy

Our Setup

Our Setup• Open Source chef server 0.10.4


• Backup to Opscode Platform



• ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac)




• KVM & lxc virts, self hosted too.





• Never test in production!






• Many chefs don’t spoil our soup






• Many chefs don’t spoil our soup

Familiarity and Understanding

not

http://www.flickr.com/photos/photo_secessionist/5555167113/ (C) Alexander McQueen

http://www.flickr.com/photos/photo_secessionist/5555167113/




• Insight

• Insight

• Simplicity

• Insight

• Simplicity

• Standards

Insight

Insight

• You should never have to say “I don’t know.”

Insight


• What, where, when, why, how long?

Insight


• What, where, when, why, how long?

• Easy, and I’ll show you how!

Chef Dashboard

Chef Dashboard• Chef handler sends metrics to graphite [4]

Chef Dashboard• Chef handler sends metrics to graphite [4]• git clone git://github.com/etsy/chef-handlers.git


• Set your graphite server’s URL in graphite.rb



• Add the following to client.rb



• Add the following to client.rb• require "<clonedir>/graphite.rb"

graphite_handler = GraphiteReporting.newreport_handlers << graphite_handlerexception_handlers << graphite_handler



• Add the following to client.rb• require "<clonedir>/graphite.rb"

graphite_handler = GraphiteReporting.newreport_handlers << graphite_handlerexception_handlers << graphite_handler

• Etsy dashboards framework [5]

[12:54:01] <irccat> Chef run failed on gfernandez.vm.ny4dev.etsy.com[12:54:02] <irccat> https://github.etsycorp.com/gist/384228[12:54:02] <irccat>

[12:54:07] <irccat> Chef run failed on buildtest11.ny4dev.etsy.com[12:54:07] <irccat> https://github.etsycorp.com/gist/384227[12:54:07] <irccat>

https://github.etsycorp.com/gist/384228




Chef irccat Alerts

Chef irccat Alerts

• Chef handler send fails to irc[4] via irccat [6]

Chef irccat Alerts

• Chef handler send fails to irc[4] via irccat [6]• git clone git://github.com/etsy/chef-handlers.git

Chef irccat Alerts


• Set your irccat[6] server’s URL in logtoirc.rb

Chef irccat Alerts




Chef irccat Alerts



• Add the following to client.rb• require "<clonedir>/logtoirc.rb"

exception_handlers << Etsy::LogToIRC.new

~ > knife node lastrun buildtest11.ny4dev.etsy.comStatus failed Elapsed Time 4.628171438 Start Time 2012-06-18 10:06:28 +0000End Time 2012-06-18 10:06:32 +0000

Recipe Action Resource Type Resource

Backtrace<snip>

ExceptionChef::Exceptions::Package: package[php] (php::buildtest line 20) had an error: Version 5.3.10-1.el5 of php not found. Did you specify both version and release? (version-release, e.g. 1.84-10.fc6)

~ > knife search node 'lastrun_debug_formatted_exception:Chef\:\:Exceptions\:\:Package*' -a lastrun.debug.formatted_exception

5 items found

id: masterrestore.ny4.etsy.comlastrun.debug.formatted_exception: Chef::Exceptions::Package: package[postgresql-server] (postgresql::server-8.3 line 1) had an error: Installed package postgresql-server-8.3.16-1PGDG_id is newer than candidate package postgresql-server-8.3.11-1PGDG_id.rhel5

id: buildtest11.ny4dev.etsy.comlastrun.debug.formatted_exception: Chef::Exceptions::Package: package[php] (php::buildtest line 20) had an error: Version 5.3.10-1.el5 of php not found. Did you specify both version and release? (version-release, e.g. 1.84-10.fc6)

<snip>

Chef lastrun Info

Chef lastrun Info

• Chef handler and knife plugin [7]

Chef lastrun Info


• gem install knife-lastrun

Chef lastrun Info




Chef lastrun Info



• Add the following to client.rb• require "lastrun_update"

handler = LastRunUpdateHandler.newreport_handlers << handlerexception_handlers << handler

Chef lastrun Info



• Add the following to client.rb• require "lastrun_update"

handler = LastRunUpdateHandler.newreport_handlers << handlerexception_handlers << handler

• knife node lastrun <nodename>

Simplicity

Simplicity

• Think of yourself at 3AM!

Simplicity


• Please, won’t you think of the new guy?

Simplicity



• Minimize the logics!

Simplicity



• Minimize the logics!

• As few logical steps from start to finish as possible.

Simplicity - Not!Date: Mon Dec 05 2011 23:07:18 GMT+0000 (GMT)

Subject: so close to death

# Don't install v2 on search or Cent 5.6 nodes-if node[:fqdn] !~ /\b(^(preprod-)?search[0-9]{2}|ny4dev\.etsy\.com|^(preprod-)?giftsweb[0-9]{2}|^db(shard|spare|data)[0-9]{2}|^qa-web01|^devsearch[0-9]{2}|^nagios01|^webnest[0-9]{2}|^prodking[0-9]{2}|^sandboxweb[0-9]{2}|^virt((0[5-9])|(1[0-9]))|^msysmgr[0-9]{2}|^msysmta[0-9]{2}|^dbconvo[0-9]{2}|^dbshowcase01|atlasweb[0-9]{2}|devnagios[0-9]{2}|cimaster02|worker[0-9]{2}|^ganglia[0-9]{2}|îmgcache[0-9]{2}|imgconvert[0-9]{2}|îmgwriter[0-9]{2}|dev-img02|^datacache04|^graphite01|^graphite03|^webutil03|^webutil04|^statsd01|^maintweb[0-9]{2}|^(dev-|preprod-)?convosearch[0-9]{2}|deployinator[0-9]{2}|^wpadmin01|^(preprod-)?dbtasks[0-9]{2})\b/ and node.role?("Web56") == false and node.role?("Preprodweb56") == false and node.role?("Princess53") == false+if node[:fqdn] !~ /\b(^(preprod-)?search[0-9]{2}|ny4dev\.etsy\.com|^(preprod-)?giftsweb[0-9]{2}|^db(shard|spare|data)[0-9]{2}|^qa-web01|^devsearch[0-9]{2}|^nagios01|^webnest[0-9]{2}|^prodking[0-9]{2}|^sandboxweb[0-9]{2}|^virt((0[5-9])|(1[0-9]))|^msysmgr[0-9]{2}|^msysmta[0-9]{2}|^dbconvo[0-9]{2}|^dbshowcase01|atlasweb[0-9]{2}|devnagios[0-9]{2}|cimaster02|worker[0-9]{2}|^ganglia[0-9]{2}|îmgcache[0-9]{2}|imgconvert[0-9]{2}|îmgwriter[0-9]{2}|dev-img02|^datacache04|^graphite01|^graphite03|^webutil03|^webutil04|^statsd01|^maintweb[0-9]{2}|^(dev-|preprod-)?convosearch[0-9]{2}|deployinator[0-9]{2}|^wpadmin01|^(preprod-)?dbtasks[0-9]{2})\b/ and node.role?("Web56") == false and node.role?("Preprodweb56") == false and node.role?("Princess53") == false and node.role?("Auth") == false

Simplicity - Better!if node.chef_environment == "libmemcached_upgrade"

package "libmemcached" do

version "1.0.4-1"

action :install

end

<snip>

else

package "libmemcached" do

version "0.53-1.1"

action :install

end

<snip>

end

Simplicity - Complexity


• Sometimes you need complex behaviour



• Don’t fight it, try to abstract it.



• Don’t fight it, try to abstract it.

• Case in point: Syslog-ng refactor

Case Study: Syslog-ng


• 36 recipes


• 36 recipes

• 30 versions of syslog-ng.conf


• 36 recipes


• 27 manually configured files in /etc/syslog-ng.d on central server


• 36 recipes


• 27 manually configured files in /etc/syslog-ng.d on central server

• Edge cases and exceptions galore



• Down to:


• Down to:

• 2 recipes (one client, one server)


• Down to:


• 2 templates (one for syslog-ng.conf, one for stuff in /etc/syslog-ng.d)


• Down to:



• Attributes in roles


• Down to:



• Attributes in roles

• Not open sourced yet, sorry :(

Case Study: Syslog-ng"syslog": {

"group": "preprod_web",

"items": {

"web_apache_access_log": {

"source": "/var/log/httpd/access_log",

"source_program_override": "APACHEACCESS: ",

"destination": "<snip>/access.log",

"destination_filters": [

"host('^preprod-web')",

"match('APACHEACCESS')"

],

"destination_options": [

"template_escape(no)"

]

},

}

}

Remember, No Panacea!


• A new package hits the repo.



• Are you in control of when it goes out?




• Memcached Outage





• Do you know what services are going to restart and when?





• Do you know what services are going to restart and when?

• Image Service Outage

Standards

Standards

• Not going to talk about testing [8]!

Standards

• Not going to talk about testing [8]!

• But I don’t have time for standards!

Standards - No Time!


• I won’t say “Make Time”, but you should...



• For a quick win, try Foodcritic




• Good out of the box rules





• Jenkins integration in seconds






• Supports custom rules






• Supports custom rules

• Plays well with others

Foodcritic

Foodcritic•gem install foodcritic


•foodcritic <cookbook_repo>



•Simple Jenkins job:



•Simple Jenkins job:

#!/usr/bin/env rvm-shell 1.9.3foodcritic -f correctness .

Standards at Etsy

Standards at Etsy• “style” not “correctness”[9]


• ETSY001 - Package or yum_package resource used with :upgrade action



• ETSY002 - Execute resource used to run git commands




• ETSY003 - Execute resource used to run curl or wget commands





• ETSY004 - Execute resource defined without conditional or action :nothing






• ETSY005 - Action :restart sent to a core service







• ETSY006 - Execute resource used to run chef-provided command








• ETSY007 - Package or yum_package resource used to install core package without specific version number








• ETSY007 - Package or yum_package resource used to install core package without specific version number

Standards at Etsy

Standards at Etsy

• ETSY001 - Written after Memcached Outage

Standards at Etsy


• Package or yum_package resource used with :upgrade action

Standards at Etsy


• Package or yum_package resource used with :upgrade action

•package "memcached" do action :upgradeend

Standards at Etsy

Standards at Etsy

• ETSY005 - Written after a total Image service outage

Standards at Etsy


• Action :restart sent to a core service

Standards at Etsy


• Action :restart sent to a core service• cookbook_file "/etc/httpd/conf.d/myvhost.conf" do

source "myvhost.conf" notifies :restart, resources(:service => "httpd")end

Critical Approach and Experimentation

CA&E

CA&E

• Chef is by necessity generic

CA&E


• ...so don’t take Opscode’s word for it.

CA&E



• If it doesn’t work well for you, change it!

CA&E



• If it doesn’t work well for you, change it!

• Case Study - Etsy Environments rollout

Environments Rollout


• We knew we needed them



• Simple enough, right?



• Simple enough, right?

• Let’s look at the workflow...

Env: Standard Workflow


• knife cookbook show php



• Change version number in metadata.rb




• Change version constraint in foo.json





• Commit and push changes to git






• knife cookbook upload php --freeze






• knife cookbook upload php --freeze

• knife environment from file foo.json

Env: Our Issues

Env: Our Issues

• 41 people with Chef repo access

Env: Our Issues


• Most with knife keys

Env: Our Issues


• Most with knife keys

• All editing the same 2 files with every change...

Env: Solutions?

Env: Solutions?

• Go with it and hope for the best?

Env: Solutions?


• Don’t use environments?

Env: Solutions?



• Write a totally new workflow?

Env: Solutions?



• Write a totally new workflow?

• Tweak the existing one with some tooling?

Env: Solution!

Env: Solution!

• Tweak with some tooling!

Env: Solution!

• Tweak with some tooling!

• Presenting knife-spork [10]...

Spork: Workflow

Spork: Workflow

• Wrapper around standard environments workflow

Spork: Workflow


• check - cookbook versioning

Spork: Workflow



• bump - increment version component

Spork: Workflow




• upload - upload and freeze

Spork: Workflow




• upload - upload and freeze

• promote - set env constraints

Spork: Check$> knife spork check apache2

Checking versions for cookbook apache2...

Current local version: 1.0.6

Remote versions (Max. 5 most recent only):*1.0.6, frozen1.0.5, frozen<snip>

DANGER: Your local cookbook has same version number as the starred version above!

Please bump your local version or you won't be able to upload.

Spork: Bump

$> knife spork bump apache2 <major|minor|patch|manual>

Bumping patch level of the apache2 cookbook from 1.0.6 to 1.0.7

Spork: Upload

$> knife spork upload apache2

Uploading and freezing apache2 [1.0.7]upload complete

Spork: Promote$> knife spork promote foo php

Adding version constraint php = 1.0.6

Saving changes into foo.json

Promotion complete! Please remember to upload your changed Environment file to the Chef Server.

---

$> knife spork promote foo php --remoteAdding version constraint php = 0.1.0

Saving changes into foo.json

Uploading foo to server

Spork

Spork

• Worked well, avoided the issues it was designed for

Spork


• Subsequently evolved

Spork



• A lot of input & work came from Devs

Spork




• Organic evolution

Spork




• Organic evolution

• Open sourced, of course

Spork: Evolution

Spork: Evolution

• Safety Checks

Spork: Evolution

• Safety Checks

• Extra Features

Spork: Safety Checks


• Before promoting, check version is uploaded...


$> knife spork promote php --remote

<snip>

WARNING: It looks like you have multiple cookbook paths defined so I can't tell if you're running inside a git repo.

Checking that php version 0.1.93 exists on the server before promoting (any error means it hasn't been uploaded yet)...

ERROR: The object you are looking for could not be found

Response: Cannot find a cookbook named php with version 0.1.93


• Before promoting, check version is uploaded...

• Check if you’re promoting changes to more than you thought....

Spork: Safety ChecksWARNING: You're about to promote changes to several cookbooks:

WARNING: ganglia: = 0.1.26 changed to = 0.1.25installerz: = 0.1.66 changed to = 0.1.65php: = 0.1.92 changed to = 0.1.93

Are you sure you want to continue? (Y/N) N

You said no, so I'm done here.

Would you like to reset your local development.json to match the server?? (Y/N) Y

<snip>

development.json reset.

Spork: Features

Spork: Features• Default Environments


• Git support


• Git support

• Chat notifications + Gist (upload and promote)


• Git support


• IRCCat


• Git support


• IRCCat

• Hipchat (courtesy of Secondmarket)

Features: Chat Notifications

[19:43:46] <irccat> CHEF: pmcdonnell uploaded and froze cookbook immount version 0.0.24

[19:44:00] <irccat> CHEF: pmcdonnell uploaded environment production https://github.etsycorp.com/gist/385043

[19:44:01] <irccat> CHEF: pmcdonnell uploaded environment development https://github.etsycorp.com/gist/385044






• Git support


• IRCCat


• Graphite (on promote)

Features: Graphite


• Git support


• IRCCat


• Graphite (on promote)

• Foodcritic (on upload)

Features: Foodcritic

$> knife spork upload system

<snip>

Lint checking system...

ERROR: Lint check failed. Halting upload.

ERROR: Lint check output:

ERROR: ETSY003: Execute resource used to run curl or wget commands: /Users/jcowie/dev/etsy/chef/cookbooks/system/recipes/dev-ssl.rb:41

Use the Source!

tiny.cc/velocity2012

We’re hiring!

BoF on Tuesdayor just come and say Hi :)

Technology

Michelin Starred Cooking with Chef