Michelin Starred Cooking with Chef

Jon Cowie, Etsy.comjcowie@etsy.com

@jonlives

• Chef at Etsy

• Familiarity and Understanding

• Chef at Etsy

• Critical Approach and Experimentation

• Chef at Etsy

• Use The Source

• Chef at Etsy

• Use The Source

• A liberal sprinkling of screwups

• Chef at Etsy

• Use The Source

• Open Sourced Goodness - We’re all here!

• Chef at Etsy

• Use The Source

• Open Sourced Goodness - We’re all here!

• [x] = http://tiny.cc/velocity2012

Opscode is Orange,Velocity is Blue.In Soviet Russia,

Cookbook writes you.

Chef at Etsy

Our Setup

Our Setup• Open Source chef server 0.10.4

• Backup to Opscode Platform

• ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac)

• KVM & lxc virts, self hosted too.

• Never test in production!

• Many chefs don’t spoil our soup

Familiarity and Understanding

http://www.flickr.com/photos/photo_secessionist/5555167113/ (C) Alexander McQueen

• Insight

• Simplicity

• Insight

• Simplicity

• Standards

Insight

• You should never have to say “I don’t know.”

Insight

• What, where, when, why, how long?

Insight

• What, where, when, why, how long?

• Easy, and I’ll show you how!

Chef Dashboard

Chef Dashboard• Chef handler sends metrics to graphite [4]

Chef Dashboard• Chef handler sends metrics to graphite [4]• git clone git://github.com/etsy/chef-handlers.git

• Set your graphite server’s URL in graphite.rb

• Add the following to client.rb

• Add the following to client.rb• require "<clonedir>/graphite.rb"

graphite_handler = GraphiteReporting.newreport_handlers << graphite_handlerexception_handlers << graphite_handler

• Add the following to client.rb• require "<clonedir>/graphite.rb"

graphite_handler = GraphiteReporting.newreport_handlers << graphite_handlerexception_handlers << graphite_handler

• Etsy dashboards framework [5]

[12:54:01] <irccat> Chef run failed on gfernandez.vm.ny4dev.etsy.com[12:54:02] <irccat> https://github.etsycorp.com/gist/384228[12:54:02] <irccat>

[12:54:07] <irccat> Chef run failed on buildtest11.ny4dev.etsy.com[12:54:07] <irccat> https://github.etsycorp.com/gist/384227[12:54:07] <irccat>

Chef irccat Alerts

• Chef handler send fails to irc[4] via irccat [6]

Chef irccat Alerts

• Chef handler send fails to irc[4] via irccat [6]• git clone git://github.com/etsy/chef-handlers.git

Chef irccat Alerts

• Set your irccat[6] server’s URL in logtoirc.rb

Chef irccat Alerts

• Add the following to client.rb• require "<clonedir>/logtoirc.rb"

exception_handlers << Etsy::LogToIRC.new

~ > knife node lastrun buildtest11.ny4dev.etsy.comStatus failed Elapsed Time 4.628171438 Start Time 2012-06-18 10:06:28 +0000End Time 2012-06-18 10:06:32 +0000

Recipe Action Resource Type Resource

Backtrace<snip>

ExceptionChef::Exceptions::Package: package[php] (php::buildtest line 20) had an error: Version 5.3.10-1.el5 of php not found. Did you specify both version and release? (version-release, e.g. 1.84-10.fc6)

~ > knife search node 'lastrun_debug_formatted_exception:Chef\:\:Exceptions\:\:Package*' -a lastrun.debug.formatted_exception

5 items found

id: masterrestore.ny4.etsy.comlastrun.debug.formatted_exception: Chef::Exceptions::Package: package[postgresql-server] (postgresql::server-8.3 line 1) had an error: Installed package postgresql-server-8.3.16-1PGDG_id is newer than candidate package postgresql-server-8.3.11-1PGDG_id.rhel5

id: buildtest11.ny4dev.etsy.comlastrun.debug.formatted_exception: Chef::Exceptions::Package: package[php] (php::buildtest line 20) had an error: Version 5.3.10-1.el5 of php not found. Did you specify both version and release? (version-release, e.g. 1.84-10.fc6)

<snip>

Chef lastrun Info

• Chef handler and knife plugin [7]

Chef lastrun Info

• gem install knife-lastrun

Chef lastrun Info

• Add the following to client.rb• require "lastrun_update"

handler = LastRunUpdateHandler.newreport_handlers << handlerexception_handlers << handler

Chef lastrun Info

• Add the following to client.rb• require "lastrun_update"

handler = LastRunUpdateHandler.newreport_handlers << handlerexception_handlers << handler

• knife node lastrun <nodename>

Simplicity

• Think of yourself at 3AM!

Simplicity

• Please, won’t you think of the new guy?

Simplicity

• Minimize the logics!

Simplicity

• Minimize the logics!

• As few logical steps from start to finish as possible.

Simplicity - Not!Date: Mon Dec 05 2011 23:07:18 GMT+0000 (GMT)

Subject: so close to death

# Don't install v2 on search or Cent 5.6 nodes-if node[:fqdn] !~ /\b(^(preprod-)?search[0-9]{2}|ny4dev\.etsy\.com|^(preprod-)?giftsweb[0-9]{2}|^db(shard|spare|data)[0-9]{2}|^qa-web01|^devsearch[0-9]{2}|^nagios01|^webnest[0-9]{2}|^prodking[0-9]{2}|^sandboxweb[0-9]{2}|^virt((0[5-9])|(1[0-9]))|^msysmgr[0-9]{2}|^msysmta[0-9]{2}|^dbconvo[0-9]{2}|^dbshowcase01|atlasweb[0-9]{2}|devnagios[0-9]{2}|cimaster02|worker[0-9]{2}|^ganglia[0-9]{2}|îmgcache[0-9]{2}|imgconvert[0-9]{2}|îmgwriter[0-9]{2}|dev-img02|^datacache04|^graphite01|^graphite03|^webutil03|^webutil04|^statsd01|^maintweb[0-9]{2}|^(dev-|preprod-)?convosearch[0-9]{2}|deployinator[0-9]{2}|^wpadmin01|^(preprod-)?dbtasks[0-9]{2})\b/ and node.role?("Web56") == false and node.role?("Preprodweb56") == false and node.role?("Princess53") == false+if node[:fqdn] !~ /\b(^(preprod-)?search[0-9]{2}|ny4dev\.etsy\.com|^(preprod-)?giftsweb[0-9]{2}|^db(shard|spare|data)[0-9]{2}|^qa-web01|^devsearch[0-9]{2}|^nagios01|^webnest[0-9]{2}|^prodking[0-9]{2}|^sandboxweb[0-9]{2}|^virt((0[5-9])|(1[0-9]))|^msysmgr[0-9]{2}|^msysmta[0-9]{2}|^dbconvo[0-9]{2}|^dbshowcase01|atlasweb[0-9]{2}|devnagios[0-9]{2}|cimaster02|worker[0-9]{2}|^ganglia[0-9]{2}|îmgcache[0-9]{2}|imgconvert[0-9]{2}|îmgwriter[0-9]{2}|dev-img02|^datacache04|^graphite01|^graphite03|^webutil03|^webutil04|^statsd01|^maintweb[0-9]{2}|^(dev-|preprod-)?convosearch[0-9]{2}|deployinator[0-9]{2}|^wpadmin01|^(preprod-)?dbtasks[0-9]{2})\b/ and node.role?("Web56") == false and node.role?("Preprodweb56") == false and node.role?("Princess53") == false and node.role?("Auth") == false

Simplicity - Better!if node.chef_environment == "libmemcached_upgrade"

package "libmemcached" do

version "1.0.4-1"

action :install

<snip>

package "libmemcached" do

version "0.53-1.1"

action :install

<snip>

Simplicity - Complexity

• Sometimes you need complex behaviour

• Don’t fight it, try to abstract it.

• Case in point: Syslog-ng refactor

Case Study: Syslog-ng

• 36 recipes

• 30 versions of syslog-ng.conf

• 36 recipes

• 27 manually configured files in /etc/syslog-ng.d on central server

• 36 recipes

• 27 manually configured files in /etc/syslog-ng.d on central server

• Edge cases and exceptions galore

• Down to:

• 2 recipes (one client, one server)

• Down to:

• 2 templates (one for syslog-ng.conf, one for stuff in /etc/syslog-ng.d)

• Down to:

• Attributes in roles

• Down to:

• Attributes in roles

• Not open sourced yet, sorry :(

Case Study: Syslog-ng"syslog": {

"group": "preprod_web",

"items": {

"web_apache_access_log": {

"source": "/var/log/httpd/access_log",

"source_program_override": "APACHEACCESS: ",

"destination": "<snip>/access.log",

"destination_filters": [

"host('^preprod-web')",

"match('APACHEACCESS')"

"destination_options": [

"template_escape(no)"

Remember, No Panacea!

• A new package hits the repo.

• Are you in control of when it goes out?

• Memcached Outage

• Do you know what services are going to restart and when?

• Image Service Outage

Standards

• Not going to talk about testing [8]!

Standards

• Not going to talk about testing [8]!

• But I don’t have time for standards!

Standards - No Time!

• I won’t say “Make Time”, but you should...

• For a quick win, try Foodcritic

• Good out of the box rules

• Jenkins integration in seconds

• Supports custom rules

• Plays well with others

Foodcritic

Foodcritic•gem install foodcritic

•foodcritic <cookbook_repo>

•Simple Jenkins job:

#!/usr/bin/env rvm-shell 1.9.3foodcritic -f correctness .

Standards at Etsy

Standards at Etsy• “style” not “correctness”[9]

• ETSY001 - Package or yum_package resource used with :upgrade action

• ETSY002 - Execute resource used to run git commands

• ETSY003 - Execute resource used to run curl or wget commands

• ETSY004 - Execute resource defined without conditional or action :nothing

• ETSY005 - Action :restart sent to a core service

• ETSY006 - Execute resource used to run chef-provided command

• ETSY007 - Package or yum_package resource used to install core package without specific version number

Standards at Etsy

• ETSY001 - Written after Memcached Outage

Standards at Etsy

• Package or yum_package resource used with :upgrade action

Standards at Etsy

• Package or yum_package resource used with :upgrade action

•package "memcached" do action :upgradeend

Standards at Etsy

• ETSY005 - Written after a total Image service outage

Standards at Etsy

• Action :restart sent to a core service

Standards at Etsy

• Action :restart sent to a core service• cookbook_file "/etc/httpd/conf.d/myvhost.conf" do

source "myvhost.conf" notifies :restart, resources(:service => "httpd")end

Critical Approach and Experimentation

• Chef is by necessity generic

• ...so don’t take Opscode’s word for it.

• If it doesn’t work well for you, change it!

• Case Study - Etsy Environments rollout

Environments Rollout

• We knew we needed them

• Simple enough, right?

• Let’s look at the workflow...

Env: Standard Workflow

• knife cookbook show php

• Change version number in metadata.rb

• Change version constraint in foo.json

• Commit and push changes to git

• knife cookbook upload php --freeze

• knife environment from file foo.json

Env: Our Issues

• 41 people with Chef repo access

Env: Our Issues

• Most with knife keys

Env: Our Issues

• Most with knife keys

• All editing the same 2 files with every change...

Env: Solutions?

• Go with it and hope for the best?

Env: Solutions?

• Don’t use environments?

Env: Solutions?

• Write a totally new workflow?

Env: Solutions?

• Write a totally new workflow?

• Tweak the existing one with some tooling?

Env: Solution!

• Tweak with some tooling!

Env: Solution!

• Tweak with some tooling!

• Presenting knife-spork [10]...

Spork: Workflow

• Wrapper around standard environments workflow

Spork: Workflow

• check - cookbook versioning

Spork: Workflow

• bump - increment version component

Spork: Workflow

• upload - upload and freeze

Spork: Workflow

• upload - upload and freeze

• promote - set env constraints

Spork: Check$> knife spork check apache2

Checking versions for cookbook apache2...

Current local version: 1.0.6

Remote versions (Max. 5 most recent only):*1.0.6, frozen1.0.5, frozen<snip>

DANGER: Your local cookbook has same version number as the starred version above!

Please bump your local version or you won't be able to upload.

Spork: Bump

$> knife spork bump apache2 <major|minor|patch|manual>

Bumping patch level of the apache2 cookbook from 1.0.6 to 1.0.7

Spork: Upload

$> knife spork upload apache2

Uploading and freezing apache2 [1.0.7]upload complete

Spork: Promote$> knife spork promote foo php

Adding version constraint php = 1.0.6

Saving changes into foo.json

Promotion complete! Please remember to upload your changed Environment file to the Chef Server.

$> knife spork promote foo php --remoteAdding version constraint php = 0.1.0

Saving changes into foo.json

Uploading foo to server

• Worked well, avoided the issues it was designed for

• Subsequently evolved

• A lot of input & work came from Devs

• Organic evolution

• Open sourced, of course

Spork: Evolution

• Safety Checks

Spork: Evolution

• Safety Checks

• Extra Features

Spork: Safety Checks

• Before promoting, check version is uploaded...

$> knife spork promote php --remote

<snip>

WARNING: It looks like you have multiple cookbook paths defined so I can't tell if you're running inside a git repo.

Checking that php version 0.1.93 exists on the server before promoting (any error means it hasn't been uploaded yet)...

ERROR: The object you are looking for could not be found

Response: Cannot find a cookbook named php with version 0.1.93

• Before promoting, check version is uploaded...

• Check if you’re promoting changes to more than you thought....

Spork: Safety ChecksWARNING: You're about to promote changes to several cookbooks:

WARNING: ganglia: = 0.1.26 changed to = 0.1.25installerz: = 0.1.66 changed to = 0.1.65php: = 0.1.92 changed to = 0.1.93

Are you sure you want to continue? (Y/N) N

You said no, so I'm done here.

Would you like to reset your local development.json to match the server?? (Y/N) Y

<snip>

development.json reset.

Spork: Features

Spork: Features• Default Environments

• Git support

• Chat notifications + Gist (upload and promote)

• Git support

• IRCCat

• Git support

• IRCCat

• Hipchat (courtesy of Secondmarket)

Features: Chat Notifications

[19:43:46] <irccat> CHEF: pmcdonnell uploaded and froze cookbook immount version 0.0.24

[19:44:00] <irccat> CHEF: pmcdonnell uploaded environment production https://github.etsycorp.com/gist/385043

[19:44:01] <irccat> CHEF: pmcdonnell uploaded environment development https://github.etsycorp.com/gist/385044

• Git support

• IRCCat

• Graphite (on promote)

Features: Graphite

• Git support

• IRCCat

• Graphite (on promote)

• Foodcritic (on upload)

Features: Foodcritic

$> knife spork upload system

<snip>

Lint checking system...

ERROR: Lint check failed. Halting upload.

ERROR: Lint check output:

ERROR: ETSY003: Execute resource used to run curl or wget commands: /Users/jcowie/dev/etsy/chef/cookbooks/system/recipes/dev-ssl.rb:41

Use the Source!

tiny.cc/velocity2012

We’re hiring!

BoF on Tuesdayor just come and say Hi :)

Michelin Starred Cooking with Chef

Technology

Great Britain & Ireland MICHELIN guide 2015 Summary Auerbach/image/GM UK2015... · 2 Great Britain & Ireland MICHELIN guide 2015 Starred establishments o Exceptional cuisine, worth

Insider - FOCUS · Balmoral This luxurious hotel is a landmark in itself,centrally located on the main shopping street.The Michelin starred restaurant Number One is located within

Ecl Visions n7 2020 - eclisseworld.com...creations of world famous Michelin-starred chefs. The venue includes restaurants, conference halls, meeting rooms and other hospitality facilities

GASCON CONNECTION › wp-content › uploads › 2016 › 03 › GC... · 2016-05-09 · ABOUT US Gascon Connection Restaurants is a partnership between Michelin-starred Chef Pascal

Ahmed BALADI Lawyer Partner, Allen & Overyfrancechinafoundation.org/wp-content/uploads/2015/... · Michelin-starred chef, Restaurant Akrame & Ateliers Vivanda (Paris, Hong Kong) Akrame

Celebrity Edge · EAT. DRINK. MEET. • Globally inspired meals with locally sourced ingredients • Menus by a Michelin-starred chef; meals prepared on board • Perfect pairings,

2015 – The Shortlist - veitingageirinn...Michelin-starred establishments to the fleeting dynamism of pop-ups. A unique event, strongly positioned on ‘innovation’, the 2015 Restaurant

Guest Chef Dinner with Michelin-starred Chef Christophe Émé

Michelin starred cooking with chef presentation

Swiss precision for UK’s finest furniture makers Starred ... · Starred chefs amongst joiners To say you were trained in cooking by Jamie Oliver would be excellent reference. This

Supporting your buSineSS - TheAA.com · compleaT angler Michelin-starred chef Atul Kochhar has been confirmed as taking over the new Riverside restaurant at the Macdonald Compleat

Home | Esquire Developments - Experience modern living in a · 2019-01-28 · style dining and British pub favourites. Choose from Michelin-starred restaurants, cosy country inns

The Venetian macao - assets.sandsresortsmacao.cn · more venues – from our Michelin-starred Indian cuisine, The Golden Peacock and ﬁne Chinese dining, to intimate bars and cafes

THE CITYWIDE CELEBRATION OF EATING OUT - …...– great value menus in over 250 restaurants, from the Michelin starred to the neighbourhood favourites really making October the month

Win a Michelin-starred My hols break in Scotland

thebikinibandit.comthebikinibandit.com/pdfs/FDNovThanksgiving.pdf · ADOUR ALAIN DUCASSE AT THE ST. REGIS St./ FIJt11 Ave., 212, 710.2277) At this two-Michelin-starred restaurant,

MARIAGE À - q.bstatic.comcelebrating your wedding day in the ‘City of Love’ and the magical surroundings of L’Hotel. From Michelin starred menus and alluring interiors to the

WELCOME 2021 Michelin starred chef Atul Kochhar’s KANISHKA ...€¦ · share joy and togetherness! Welcome to Atul Kochhar’s KANISHKA located in the heart of Mayfair’s Maddox

IT’S ABOUT LIVING LIFE TO THE FULLEST...gadgets, fashion and more • Feature of Singapore’s listed Michelin-starred restaurants/chefs ... • Top 5 most popular leisure pursuits

French Heritage Society · at the Michelin-starred restaurant Le Coquillage. Housed in the Château de Richeux by the Famille Roellinger, Le Coquillage features refined selections