Puppet.From 0 to 100in 30 minutes

(intermediary and next steps are on you)

Alessandro Franceschi @alvagante

From 0Puppet essentials

What is Puppet?

• Puppet is a Configuration Management Tool

• It manages applications, systems' components, network and cloud resources.

• It provides a language that describes the managed IT resources abstracting the implementation details.

• It implements paradigms like Infrastructure as Code and Infrastructure as Data

Puppet components• On managed nodes are executed Puppet agent and Facter.

• On the central Master runs the Puppet server, here public modules and our Puppet code and data are deployed with r10k.

• Puppet generated data is stored to PuppetDB (it uses PostgreSQL for persistence ;)

• Our data to configure classes and manage Infrastructure can be defined using Hiera.

• It's also possible to define what classes to use in which nodes, with External Node Classifiers (ENC) like Puppet Enterprise Console or The Foreman.

A Puppet run in essence• Puppet agent use facter to collect facts about the

system and sends them to the Puppet Master

• Puppet Master, using nodes' facts and our Puppet code and data builds a catalog of resources to manage

• The catalog is sent back to the client. The included resources are applied locally: packages are installed, files changed, services started, databases initialised...

• Whatever the number of runs, at the end the system should be at the desired configuration state (idempotency)

• A report of what happens on the client is sent back to the Puppet Master and typically stored on PuppetDB

Puppet language essentials• Puppet represents managed resources on the systems

via a declarative Domain Specific Language (DSL)

• Puppet language is written in files called manifests, which have a .pp extension

• The single unit of configurations are resource types

• Puppet resources are grouped in classes or defines

• Classes, user defined types (defines), along with relevant configuration files, are shipped in modules

• Public modules are released on Puppet Forge or GitHub

(Resource) types• Puppet types manage resources abstracting the

underlying OS, elements of the system. Examples:package { 'postgresql': ensure => present, }

service { 'postgresql': ensure => running, enable => true, }

file { '/var/lib/pgsql/data/pg_hba.conf': ensure => present, source => puppet:///modules/postgres/pg_hba.conf',}

• Common native resources: package, service, file, user, group, host, cron, exec, host, mount...

Extra resources from modules• Additional resources can be provided by dedicated

modules. Examples from puppetlabs-postgresql module:postgresql::server::db { 'mydatabasename': user => 'mydatabaseuser', password => postgresql_password('mydatabaseuser', 'mypassword'), }

postgresql::server::role { 'marmot': password_hash => postgresql_password('marmot', 'mypasswd'), }

postgresql::server::database_grant { 'test1': privilege => 'ALL', db => 'test1', role => 'marmot', }

postgresql::server::table_grant { 'my_table of test2': privilege => 'ALL', table => 'my_table', db => 'test2', role => 'marmot', }

postgresql::server::pg_hba_rule { 'allow application network to access app database': description => "Open up PostgreSQL for access from 200.1.2.0/24", type => 'host', database => 'app', user => 'app', address => '200.1.2.0/24', auth_method => 'md5', }

Extra resources examples• There are modules to manage almost everything:

- Specific applications (nginx, postgresql...) - Systems' features (iptables, sysctl, network...) - Network equipment (cisco, arista, cumulus... ) - Storage equipment (netapp, emc...)- Cloud resources (aws, azure, digitalocean...) - Cloud infrastructures (openstack, opennebula...) - Containers (docker, mesos, rkt...)

• Each module may provide specific classes and resources to manage with Puppet DSL specific components

Classes• Puppet classes expose parameters which define

what and how are the managed resourcesclass postgresql::server ( $postgres_password = undef, $package_ensure = $postgresql::params::package_ensure, $service_ensure = $postgresql::params::service_ensure, $service_enable = $postgresql::params::service_enable, ) inherits postgresql::params { [ ... ] package { 'postgresql': ensure => $package_ensure, # [...] } service { 'postgresql': ensure => $service_ensure, # [...] enable => $service_enable, # [...] } }

• Class parameters can be set via Hiera, for example using the yaml backendpostgresql::server::postgres_password: 'my_cleartext_password'postgresql::server::package_ensure: absent# Secrets can be encrypted using the hiera-eyaml backend. They look like: postgresql::server::postgres_password: 'ENC[PKCS7,MIIISA ...]

Modules• Modules are distributable directories containing manifests,

extensions, templates and files to manage.

• They have a standard structure:mysql/ # Main module directorymysql/manifests/ # Manifests directory. Puppet code stays here.mysql/lib/ # Plugins directory. Ruby code that extends # Puppet (types, providers, facts...) is here.mysql/templates/ # ERB and EPP Templates directorymysql/files/ # Static files directory

include mysql# Main mysql class is placed in: $modulepath/mysql/manifests/init.pp

include mysql::server# This class is defined in: $modulepath/mysql/manifests/server.pp

mysql::db { ...}# This define is defined in: $modulepath/mysql/manifests/db.ppfile { '/etc/motd': content => template('motd/motd.conf.erb'), } # Template is in: $modulepath/motd/templates/motd.conf.erb

• This allows some conventions:

Hiera• Hiera is a key-value lookup tool based on a

configurable hierarchy

• It can have different backends to store data in different places (yaml, eyaml, json, redis, mysql...)

• Hiera is configured in hiera.yaml--- :backends: - yaml

:hierarchy: - "hostname/%{::trusted.certname}" - "role/%{::role}-%{::env}" - "role/%{::role}" - common

:yaml: :datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata"

Roles and profiles• A popular pattern to manage what resources

should be manages on which nodes

• A role is assigned to nodes (using different methods: custom facts, ENC...)

• A role represents what a node does and can include one or more profiles

• In profiles resources from modules are used to configure local systems in the desired way

Puppet environments• A Puppet environment is directory under /etc/puppetlabs/code/environments/ where are is placed our Puppet manifests, modules and Hiera data

• Puppet environments can can be distributed as git control-repos. They contain: manifest directory for common manifests modules directory for moduleshieradata directory for Hiera data, if used Puppetfile to configure r10k for public modules

to 100Introducing example42 Puppet control repo

example42 control-repo• The result of several years of Puppet experience.

A git repository with the content of a Puppet 4 environment featuring:

• Data and code for a state of the art modern Puppet setup

• Fabric integration for common operations

• Puppet 4 optimised code following updated design principles

• Customisable Vagrant environments to locally test code

• Docker integrations to test code or build images

• Classification based on Roles and Profiles pattern

• In green fields: whatever is needed to start a new Puppet project from scratch

• In brown fields: source of inspiration

• ... and, yes, a lot more.

Setup• Download the repo from GitHub

• Setup environment via a shell script ...

• Or via Fabric:

• Install git hooks for pre-commit syntax checks

git clone https://github.com/example42/control-repocd control-repo

bin/setup.sh

fab puppet.setup

fab git.install_hooks

Explore• Give a look to the following files and directories: # The first manifest parsed by Puppet server manifests/site.pp

# r10k Puppetfile and directory for public modules Puppetfilemodules/

# Directory containing local site modulessite/

# Sample Hiera configuration file and data directoryhiera.yamlhieradata/

# Directories for Vagrant and Docker operationsvagrant/docker/

# Blueprint directory for new Puppet 4 modules skeleton/

Test local code with VagrantDifferent Vagrant environments available: fab vagrant.all_status

Single roles can be tested in relevant VMs: fab vagrant.up:vm=dev-local-log-01fab vagrant.provision:vm=dev-local-log-01

# All Linux servers use this class of common resources:# site/profile/manifests/base/linux.pp# Common settings are in: # hieradata/common.yaml# For role "log" specific Hiera data is in# hieradata/role/log.yaml

Vagrant environments can be customised: # vi vagrant/environments/$vagrant_environment/config.yamlvi vagrant/environments/puppetinfra/config.yaml

Test local code with DockerBuild multiOS Docker images: fab docker.build_multios

# Images are buid based on the data in:# hieradata/role/docker_multios_build.yaml

Test a role on a given OS: fab docker.test_role:log,ubuntu-14.04

# Available images: ubuntu-12.04, ubuntu-14.04, ubuntu-14.06 # centos-7, debian-7, debian-8, alpine-3.3

Build an image based on a role (WIP): fab docker.build_role:log,ubuntu-14.04

Build multiple OS images based on a role (WIP): fab docker.build_role_multios:log

next stepslearn Puppet and customise your control-repo

Learning PuppetCheck the Official docs http://docs.puppet.com Ask and get involved in the Community https://puppet.com/community

Look for modules of Puppet Forge http://forge.puppet.com

Give a look to Tiny Puppet (used in example42/control-repo) http://www.tiny-puppet.com

Customise• The control-repo is just the starting point for a

greenfield modern Puppet setup

• Define a way to set your nodes' roles

• Select the public modules to use and add them to Puppetfile

• Write local profiles in site/profile/manifests

• Review hiera.yaml logic and customise data in hieradata/

• Customise your Vagrant environment

• Customise the skeleton to use for custom modules

Contribute• example42's control repo has a lot of room for

enhancements

• Provide profiles to manage applications

• Send bug and features reports

• Use and improve our code

• Spread the word!

#example42

From 0 to 100in 30 minutes?

Thank You(feel free to ask me anything)

Alessandro Franceschi @alvagante