Islands: Puppet at Bulletproof Networks

Islands

Who are these guys?

Mick Pollard@aussielunix

&

Lindsay Holmwood@auxesis

Puppet users for > 5 years

BULLETPROOF

IaaS &Managed Services

Movember Australian Museum Rebel Group

Blackmores Angus & Robertson Telstra

Perisher BlueScope Steel Woolworths

DMG Radio Clive Peters Deloitte

Clemenger budget.gov.au Nissan

AOC Nova Sydney Airports

Whirlpool Smooth Theiss

Borders Fosters Country Road

Midas Australian Geographic Sensis

Tourism Victoria ABS AusPost

DET FWA Vodafone

Using Puppet since 2008

Unique challenges

Strong isolation

Network segregationwith VLANs

Central Puppetmasterisnʼt an option

Thorough change control

Rapid growth

How do we use Puppet?

Standalone systems

(puppetmaster-less)

Internal infrastructure

Full customer environments

Standalone systems

(puppetmaster-less)

Campaign driven

business

budget.gov.au

movember.com

mamamia.com.au

Reverseproxies

Nginxwith customisation

Rump

More detail in

John Ferlitoʼstalk at 14.00

Internal infrastructure

Vanilla

One Puppetmaster

Ubuntu(Lucid or Precise)

Full customer environments

Every customer has their own puppetmaster

“Islands of Puppet”

Copypasta

Configuration drift

apache

apache

apache

customer-a

apache

customer-a

apache

customer-a

customer-b

apache

customer-a

customer-b

apache

customer-a

customer-b

customer-c

apache

customer-a

customer-b

customer-c

apache

customer-a

customer-b

customer-c

customer-d

Poor code share

What if customers

edit the code?

How dowe maintain

common code?

Commonalities

Mix of Puppet

versions

0.25 (as provided by

Ubuntu)

2.7 (as provided by Puppet Labs)

Mix of Operating Systems

lucid precise

2.7 internal infrastructure some customers

0.25 most customers

Passenger > webrick

--no-daemonize

Default behaviour is orthogonal to

change control

We don't want systems to change

without control

All changes initiated byan engineer

nodes + roles

node 'stlyqy-lvs02.cust.bulletproof.net' { server { $fqdn: }

include snmp::server::lvs include sysctl::lvs

include keepalive::lvs

include network::conntrack::modules include network::conntrack::hashsize include network::bonding::activebackup include network::type::bonded_vlan

include ript}

node 'stlyqy-lvs02.cust.bulletproof.net' { server { $fqdn: }

include snmp::server::lvs include sysctl::lvs

include keepalive::lvs

include network::conntrack::modules include network::conntrack::hashsize include network::bonding::activebackup include network::type::bonded_vlan

include ript}

define server($collectd_client_report_to='collectd.bulletproof.net') { include motd include augeas include apt include utils include puppet::client

include ssh::server include ssh::authorized_keys include ntp::client include postfix::satellite

include ruby::dev include ruby::rubygems

include bzr::client include git::common include git::github

include snmp::server include vmware::tools

include apparmor::disable collectd::client { "${fqdn}": report_to => $collectd_client_report_to }}

app_serverdatabase_serverfile_servermanagement_servermemcache_servermonitor_serverproxy_serverpuppetmaster_serverredis_serversingle_serversphinx_serverstatic_server

Heira is the future

We useCapistrano

ssh in-a-parallel-for-loop

Why cap and not mcollective?

We deploy everything with cap

Monitoring configurationFirewall configuration

Web applicationsInternal tools

Consistent deployment tool across all projects

Principle of least surprise

Engineers learn1 tool

Puppet is no different to the

rest of our stack

How do we usecap + Puppet?

Puppet changes

cap puppet:go ROLES=lvs options="--noop"cap puppet:go ROLES=lvs

Smoke tests

$ cap puppet:go options="--noop"

# ...

infmon hosts serves a Nagios page over https

Finished in 1.67 seconds1 example, 0 failures

Works out all roles that hosts within a

run belong to

Runs tests tagged with those roles

Fast feedback on

change success

Bootstrapping

cap node:bootstrap HOSTFILTER=lvs-08.bp.net

Takes VM in unknown state

Brings into known state for Puppet 2.7 run

Limitations

Singling out hosts is tricky

Re-using data across commands

requires... creativity

servers = []

run "mysql -e \"SHOW MASTER STATUS;\" | tail -n 1" do |channel, type, data| hostname = channel[:host] filename = data.split(/\s+/).first position = data.split(/\s+/).last servers << { :hostname => hostname, :filename => filename, :position => position }end

ROLESand

HOSTFILTER

TLDR;

There are edge cases

It does the job

How have we tried to solve

them?

First iteration

Modules

modules/apache/

| manifests/init.pp

| files/

| templates/

| lib/

| README.markdown

Stored on GitHub

Drink from the firehose

puppet-module-tool

github.compuppetlabs/puppet-module-tool

gem install puppet-module

Modulefile

puppet-module build

Turns

modules/apache/

| manifests/init.pp

| files/

| templates/

| lib/

| README.markdown

into

bulletproofnetworks-apache-1.3.0.tar.gz

Puppet forge

Public by default

Not great if you don't want to open source all your secret sauce

But puppet-module-tool is interesting...

...can we fake the forge?

Pain points

Arduous release

workflow

Better suited for infrequent

changes

High barrier of entry for customers to submit patches

Sharing bugfixes & improvements

requires significant refactoring

Limited reporting on customer lag

Second iteration

Bundler

Gemfile

#!/usr/bin/env ruby

source :rubygems

gem 'capistrano', '2.9.0'gem 'capistrano-ext', '1.2.1'gem 'colorize'gem 'puppet', '2.7.13'gem 'puppet-module'

group :test do gem 'rspec', '2.8.0' gem 'mechanize' gem 'puppet-lint'end

Can we reuse it?

Tim Sharpe@ GitHub

(@rodjek)

Messy working prototype

pre-alpha quality

More research required

Librarian

“A framework for bundlers”

librarian-chefmanages Chef repositories

by Jay Feldblumgithub.com

applicationsonline/librarian

librarian-puppet

“You can all stop using git submodules now”

gem install librarian-puppet --pre

github.comrodjek/librarian-puppet

Bundler-like behavior

for Puppet

Puppetfile

#!/usr/bin/env ruby

forge "http://forge.puppetlabs.com"

mod "puppetlabs/razor"mod "puppetlabs/ntp", "0.0.3"

#!/usr/bin/env ruby

forge "http://forge.puppetlabs.com"

mod "puppetlabs/razor"mod "puppetlabs/ntp", "0.0.3"

mod "stdlib", :git => "git://github.com/puppetlabs/puppetlabs-stdlib.git"

#!/usr/bin/env ruby

forge "http://forge.puppetlabs.com"

mod "puppetlabs/razor"mod "puppetlabs/ntp", "0.0.3"

mod "apt", :git => "git://github.com/puppetlabs/puppetlabs-apt.git" :ref => 'feature/master/dans_refactor'

mod "stdlib", :git => "git://github.com/puppetlabs/puppetlabs-stdlib.git"

One canonical module

modules/ in

.gitignore

librarian-puppet outdated

tells you what needs to be updated

librarian-puppet update

updates your modules

Demo!http://aussielunix.github.com/jenkins-appliance/

Policy

Always use a module'smaster

Make changes to modules as

usual

git commit &&

git push the module

Make module changes generic

by default

Only branch a module when:

1. something is superclient specific

2. there is an unmissable deadline

3. testing ideas

Set a reminder to merge changes into

master

Use pull requests on GitHub for

dangerous changes

Take aways:

Puppet modulesare pretty neat

Keep feedback loops short

Code shareis king

Do you have similar

problems?

How do you solve them?

Thank you!