chef@recordedfutureUlf Månsson, infra coder@ulfmansson

DevOps - CAMS

● Break the silos

● Culture

● Automation

● Measurement

● Sharing

Why automation and configuration management

● Quality

● Continuous delivery, deploy many times per day

● Deployment by developers

● Everything is documented as it has been created by code

● No single point of knowledge, human redundancy

● Working in a global team

Process driven

● Don't document

● Don't train people in processes

Automate and build the processes into our tools

Lower uptime

● We are aiming for lowering uptime to 1-2 weeks per server

● Use and throw

● Never patch

● Deploy often - every day

● Green-blue deployment

● Running on latest version of OS and components

How?

● Automated infrastructure

● Shared version control - one system!

● One step build and deploy

● Green - Blue deployment

● Feature flags

● Metrics and metrics

What to automate - everything!What to automate - everything!

● ProvisioningProvisioning

● Deployment and configurationDeployment and configuration

● MetricsMetrics

● MonitoringMonitoring

● Scaling & up and downScaling & up and down

● Build process - continuous deliveryBuild process - continuous delivery

● Log handlingLog handling

● Obscure host namesObscure host names

● Never login to a serverNever login to a server

Standardize on Ruby as lingua franca

One language for everything ops are doing!

● Scripting in Ruby

● Configuration management in Ruby

● Operation applications in Ruby

● System tools in Ruby

● Add-on applications in Ruby

● Application integration in Ruby

Why Ruby?

● We like Ruby

● Used by Chef

● Easy to script

● Object oriented and functional

● Lot of good libs - gems

● Great community - “Matz is nice so we are nice”

● Reuse of the same gem both in Chef, system tools and applications

What do we do with Chef?

● Provision EC2 instances

● Deploy packages

● Deploy our own code

● Deploy configurations

● Deploy metrics

● Update metrics

● Collect data

● Deploy monitoring configurations

● Deploy users

● Create databases

● Provision vagrant instances

● Local installations for customers

Chef recipe

app_name = 'xignite'

dir_inst = File.join(node['rf']['inst_dir'], app_name)

dir_conf = File.join(dir_inst, 'conf')

dir_log = File.join(node['rf']['log_dir'], app_name)

[dir_inst, dir_conf, dir_log].each do |dir|

directory dir do

recursive true

action :create

end

end

deploy_repodist app_name do

dir_inst_root dir_inst

action :deploy

end

Code

All code is code

Keep the code in the same repository

Treat all code in the same way, it doesn’t matter if it’s application code, infra code, test code or whatever

Wrapper cookbooks

Use community cookbooks!

Wrap community cookbooks!

Cookbook elasticsearch_rf → elasticsearch

In elasticserch_rf/recipes/default.rb:

include_recipe "elasticsearch::default"

Chef rewind to override resources in original cookbooks

Use LWRP - Light Weight Resource Providers

● Create your own providers

● Abstraction

● Cleaner code

● Easier to use

● Better error handling

deploy_build “ha_document_analyzer” do      action :deployend

Search & Node data

● Use search to populate with data during Chef run, find servers with specific roles

● Use node data collected by ohai in recipes, for example about memory, aws instance, ip addresses etc

● Gives a good overview of your infrastructure

Lint tools – to check the code

● Use foodcritic

● By default all rules are enabled

● Consider to use Ruby lint tools

– Rubocop

– Laser

– ruby-lint

$ foodcritic  app_sinatra_rfFC019: Access node attributes in a consistent manner: ./providers/web_app.rb:87FC048: Prefer Mixlib::ShellOut: ./recipes/nginx.rb:38

Manage cookbooks

● To manage cookbooks and dependency use tool like librarian-chef or berkshelf

● Like bundler and maven

● Separate your own cookbooks from community cookbooks

chef/community_cookbooksdatabagsforked_cookbooksrf_cookbooksroles

Orchestration

● We use mcollective

● RabbitMQ for messaging

● Stable

● Not for puppet only, fits well to Chef

mco service cluster_reindexer restart ­C role.Cluster_aggregator

Testing of Chef

● Test kitchen!

– kitchen test lxc

– kitchen converge lxc

– kitchen login lxc

● Server spec isolated tests via Test kitchen

● Chef spec, quick unit tests

● Create test cookbooks with the test data and preparation recipes

Use vagrant (lxc) for testing

. |-attributes |-files |-libraries |-providers |-recipes |-resources |-templates |-test |---cookbooks |-----app_java_rf_test |-------attributes |-------libraries |-------recipes |-------templates |---integration |-----data_bags |-------rfapps |-------rfconf |-----default |-------serverspec |---------localhost

Roles

● Keep roles simple, just include a recipe

● Don't put a lot of attributes in roles

● When using search, search for roles

● Just roles to show what running on the server

Treat your servers as cattle not as cows

● AutoscaleAutoscale

● Make logs availableMake logs available

● Make it hard to loginMake it hard to login

● Use obscure server namesUse obscure server names

● Make it easy to launch an instanceMake it easy to launch an instance

● Make process manipulation easyMake process manipulation easy

● Run Chef every 30 minute or soRun Chef every 30 minute or so

● Use Chef server to collect info and queryUse Chef server to collect info and query

bygge shows the jenkins build pipeline

Read more

● Continuous delivery by Jez Humble

● Test-Driven Infrastructure with Chef

● Chef Infrastructure Automation Cookbook

● #opschef