A Presentation about Puppet that I've made at the OSSPAC conference

Infrastructure Automation

Ohad Levy – Infineon Technologiesohad.levy@infineon.com

Page 2

Automated

”…a process which may once have been performed manually but has been altered in some way which allows a machine or computer to either wholly or partially manipulate the process to save time”

Page 3

Infrastructure

“Infrastructure is generally a set of interconnected structural elements that provide the framework supporting an entire structure”

Page 4

Automated Infrastructure

“Having the basic services necessary for your infrastructure to operate largely without the aid of a keeper.”

Page 5

Grow – from This:

http://i68.photobucket.com/albums/i2/RealConnections/googlecomputers.jpg

Page 6

Large Scale Environment

Page 7

Where does it fit in?

Page 8

Typical System Lifecycle

Pre/ Installation

DNS/DHCP…

Initial Configuration

KickstartJumpstart

Custom Scripts

FixesUpdatesAudits

Page 9

System configuration possibilities Manual System Configuration

„log in and „just do it!““¬ + OK for small number of servers¬ + Quick and easy

¬ - Very difficult to ensure similar configurations¬ - No auditing capability¬ - No change history¬ - No release documentation¬ - No way to deploy multiple servers quickly.¬ - No way to rebuild critical servers quickly.

Page 10

System configuration possibilities Install Time Auto-Configuration

„Deploy using Kickstart, Jumpstart etc“¬ + Ensures that new systems are brought into proper state as part

of installation process ¬ + Possible to deploy multiple servers quickly

¬ - No Validation that settings remain correct¬ - No easy way to deploy changes¬ - Auto-Configuration process is different between the sites leads

to different configurations ¬ - No Auditing¬ - No change history¬ - No release documentation

Page 11

Typical System Lifecycle

Pre/ Installation

DNS/DHCP…

Initial Configuration

KickstartJumpstart

Custom Scripts

FixesUpdatesAudits

Page 12

Puppet What is Puppet ?

A product designed to deploy system configurations. It is – ¬ Open source based on Ruby¬ Policy based¬ Runs every 30 minutes¬ An abstraction layer between the system administrator and the

system¬ Capable to run on any UNIX operating system

Page 13

Our Challenges

Keep our systems "harmonized“ Audit / Know what's going on each system Replace a server if it dies or to be able to add another server

that is exactly like it Similar Applications, different OS's Push out changes to all the servers that need a particular change Stop duplicating effort Go home early

How does Puppet works?

Page 15

Puppet Types

A Type is a particular element that Puppet knows how to configure

Files (content, permissions, ownership) Packages (installations, updates..) Services (enabled/disabled, running/stopped) Exec (run commands) Full List: cron, exec, file, filebucket, group, host, interface,

k5login, mailalias, maillist, mount, nagios*, package, service, sshkey, tidy, user, yumrepo, zone

Page 16

Example: Managing sudoers file

file { /etc/sudoers: owner => root, group => root, mode => 600, source => puppet://server/files/sudoer

}

Page 17

Dependencies

"require" and "before" / "after" settings ensures that types are applied in the correct orderfile { "/etc/sudoers": ... require => Package[sudo]}package { "sudo": ensure => present, before => File["/etc/sudoers"]}

Page 18

Dependencies - continued "notify" and "subscribe" settings can trigger cascaded updates

Particularly useful in services, exec

file { "/etc/ssh/sshd_conf": ... notify => Service["sshd"]

}service { "sshd": subscribe => File["/etc/ssh/sshd_conf“]}

Page 19

What is Facter?

Facter gathers information about the client, which can be used as variables within puppet.

You can add custom facts as needed.package {"sshd":

ensure => installed,name => $operatingsystem ? {

solaris => "IFKLssh",default => “openssh-server”

}}

Page 20

Example Facts$ sudo facterarchitecture => amd64domain => sin.infineon.comfacterversion => 1.3.8fqdn => sinn1636.sin.infineon.comhardwaremodel => x86_64hostname => sinn1636ipaddress => 172.20.89.113kernel => Linuxkernelrelease => 2.6.24-21-genericlsbdistcodename => hardylsbdistdescription => Ubuntu 8.04.1lsbdistid => Ubuntulsbdistrelease => 8.04macaddress => 00:1c:25:14:26:abmanufacturer => LENOVOmax speed => 2000 MHzmaximum memory module size => 4096 MB

maximum total memory size => 8192 MBmemoryfree => 988.30 MBmemorysize => 1.94 GBoperatingsystem => Debianoperatingsystemrelease => 2.6.24-21-genericprocessor0 => Intel(R) Core(TM)2 Duo CPU T7300 @ 2.00GHzprocessor1 => Intel(R) Core(TM)2 Duo CPU T7300 @ 2.00GHzprocessorcount => 2productname => 6458B43ps => ps -efpuppetversion => 0.24.5rubysitedir => /usr/local/lib/site_ruby/1.8rubyversion => 1.8.6serialnumber => L3A8632swapfree => 912.23 MBswapsize => 956.99 MBtype => Notebookuniqueid => 007f0101vendor => LENOVO

Page 21

What is a Class? A named collection of type objects Can include or inherit from other classesclass sudo_class { include foo_class file { "/etc/sudoers": ... } package{ "sudo": ... }}

Page 22

Class inheritance

class afile { file { "/tmp/foo": ensure => file source => "/src/versionA" }}class another_file inherits afile { File["/tmp/foo"] { source => "/src/versionB" }}

Page 23

What is a Node ? A configuration block matching a client Can contain types, classes "default" node matches any client without a node block

node "ohad.myself" { include sudo_class include other_class}

Page 24

External Nodes Node definitions can be defined outside of puppet – e.g. LDAP,

external script… Ideal when you have too many nodes..

Page 25

Classes and definitions Classes are groups of resources. Definitions are similar to classes, but they can be instantiated multiple

times with different arguments on the same node.

class apache2 {define simple-vhost ( $admin = "webmaster", $aliases, $docroot) {

file { "/etc/apache2/sites-available/$name":mode => "644",require => [ Package["apache2"], Service["apache2"] ],content => template("apache/vhost.conf"),

} }

}

Page 26

Classes and definitions - Continued

node mywebserver {include apache2

apache2::simple-vhost { "test.example.com": docroot => "/var/www/test“

}

apache2::simple-vhost { "debian.example.com": docroot =>"/var/www/debian“,alias => [“debiantest.example.com”,”debian”],admin => “system@example.com”

}}

Page 27

vhost.conf template Puppet uses Ruby's ERB template system:<VirtualHost *> ServerAdmin <%= admin %> DocumentRoot <%= docroot %> ServerName <%= name %><% aliases.each do |al| -%> ServerAlias <%= al %><% end -%> ErrorLog "|/usr/bin/cronolog /var/log/apache/<%= name %>/%Y-%m/error-%d" CustomLog "|/usr/bin/cronolog /var/log/apache/<%= name %>/%Y-%m/access %d" sane</VirtualHost>

Page 28

Template output

more /etc/apache2/sites-available/debian.example.com<VirtualHost *> ServerAdmin system@example.com DocumentRoot /var/www/debian ServerName debian.example.com ServerAlias debiantest.example.com ServerAlias debian ErrorLog |/usr/bin/cronolog /var/log/apache/debian.example.com/%Y-%m/error-%d“ CustomLog "|/usr/bin/cronolog /var/log/apache/debian.example.com/%Y-%m/access-%d“ sane

</VirtualHost>

Page 29

OS API - It also works theother way around:

$ ralsh user levyouser { 'levyo': groups => ['tidc'], ensure => 'present', password => 'x', shell => '/bin/bash', uid => '49960', comment => 'Ohad Levy, +65 xxxx xxxx', gid => '11100', home => '/home/levyo'}

Page 30

Puppet Modules Modules allow you to group both the logic and the files for an

application together. Puppet automatically searches its module path to find modules. Modules can contain four types of files, each of which must be

stored in a separate subdirectory: Manifests – must be stored in manifests/, and if you create

manifests/init.pp then that file will be loaded if you import that moudle name directly (e.g. import “mymodule”)

Templates – must be stored under templates/, put in your manifest some where template(“mymodule/mytemplate”)

Files – stored in files/, can be accessed as: source => puppet://”mymodule”/myfile

Plugins – Additional types, provides or facts

Page 31

File server and File Bucket Puppet also includes a file server which you can use for

transferring files from the server to the client. If you configure it, puppet can also save a backup of each file

that is changed on the client to the server. The backups go in a filebucket and can be retrieved later.

Page 32

Getting Started Install puppet (yum,pkg-get,…) or manually install ruby, facter

and puppet Setup a puppet server (puppetmaster / puppet-server). Write a manifest for your node (classes etc) Start puppet master on the server Start puppetd on the client

Page 33

Some more info Puppet uses SSL for all communications, therefor it includes a

CA, you can automatically sign CSR or use puppetca tool to mange them.

Storeconfig, option to save machine states(facts, configuration runs) and special facts (e.g. SSH keys) in a database.

Reporting, puppet has a few reporting options, most common are emails with changes, RRD files, yaml files and puppetshow web interface.

Puppet HA, load balancing etc

Page 34

What are the benefits ? Costs

Based on open source tools – No license cost Reduced Labor Reduced Cost

Productivity Reproducibility, Accuracy and homogenous environment (Reduce needed knowledge to operate our environment) Allow easy ramp up of new sites and services

Change Management Testing Processes, Auditing Global control over servers Harmonization Consolidation

…

Page 35

Conclusions We're all stuck on the hamster wheel Makes easy stuff easier, hard stuff possible Harmonize! (not just today, also in the future) Enable the business for consolidation Similar projects

cfengine bcfg2

Additional Resources http://reductivelabs.com/trac/puppet http://reductivelabs.com/trac/puppet/wiki/LanguageTutorial http://reductivelabs.com/trac/puppet/wiki/CompleteConfiguration

#puppet on irc.freenode.org

Backup SlidesAdditional information

Page 37

Configuration DB - Terminology Module – A bundle of settings related to a single applicationAt Infineon we use 3 types of modules:

Host Type - e.g. Login Servers, Compute nodes… Service - module which is needed by other modules, e.g. ssh, x11 etc. Sites Modules - sites customizations

Versioning – The ability to refer (tag) to a certain state/PiT Host Type and Service modules are associated with a version, e.g.

Login Server v1.0 Site Modules are not tagged in order to reduce complexity

* Every change in a module, requires a check in action, which is identified by a user, and a comment (change log)

* A user access restriction should be applied to each module and to the tagging mechanism

Page 38

Configuration DB - Terminology Environments – The ability to select a group of modules in a

certain versionEach site is associated with at least two environments – Production

and Testing: Testing – an environment which refer to versioned modules

under a test tag. Production – A selection of modules in a certain version that

has already been approved for Production usage. Development environment – The state where the modules are in

the latest check-in version, but are not tagged (e.g. not a testing candidate yet).

Development Testing Production (Per Host Type)

Page 39

Configuration DB - Terminology Global modules / owners

Each owner is responsible for his module (or modules) A special attentions is required for Service module as many

other modules might depend on them

Site modules As Site modules are not tagged (versioned) any change in the

site specific settings, will be rolled out immediately.

Page 40

Risks & Challenges If the right processes are not in place, it would be possible to

implement the wrong changes globally Learning Curve

How our platform works How to use version control How to work in the new processes/workflows

Fear (Am I obsolete?, will I continue to learn?)

Page 41

Why Did we choose Puppet Open source More flexible than other open source alternatives Really works (Declarative language, Abstracts configuration as

resources, Allows relationships, Transactional, Modular) Provides an API to the OS, making it very simple to support other

OS’s in the future Puppet is now being adopted in many projects For more info see

http://reductivelabs.com/trac/puppet/wiki/CfengineVsPuppethttp://reductivelabs.com/trac/puppet/wiki/WhosUsingPuppet

Page 42

Datacenters Challenges Harmonization Simplify Servers management Implement and maintain Global standards Accurate Configuration Database Rapid recovery and deployment of services globally Eliminates duplicate / unnecessary effort Same look and feel everywhere (enables consolidation / global

operation) Audits / Compliance

Page 43

Infineon Puppet Infrastructure

Configuration Database(Puppeteer, SVN, Netapp)

configuration Server

Site A

deployment &monitoring

Managed Servers

Failover

Site B

deployment &monitoring

Managed Servers

configuration Server

• Puppeteer (Puppet Master of site PM’s) • Puppet Master per site (or based on size/load)• SQL Database for Web portal, Inventory• Configuration database (Version control & Ticketing system)

Page 44

Gini’s Features AD/LDAP Authentication DHCP/DNS Connectors (API, + MS(yacks) DHCP, MS(yacks) DNS and infoblox) Puppet Host type management Disk Layout Management Host Hardware management (needed for Solaris DHCP jumpstart, and general host info) Multi OS management (e.g. RH, CentOs, Solaris.... in their various versions) User role management (who is allowed to install, in which site, which host type etc) Subnet and IP Management (auto assignment of IP address, DHCP reservation DNS records etc) Multi Domain / site management Hands-free auto OS installation (currently Kickstart and Jumpstart (sparc) using DHCP) Management of Puppet Environments, choose an env per host (e.g. we have different production states for

different sites, or for different host types). Management of puppet certificates (also support chained certificates). Import of Machine Facts (we cant use store config due to the large setup of our env) Nice Graphs (with Gruff) Puppet modules selection per host(for those who needs more than the default host type) SP (Service Processor, iLOM etc) management TFTP management Status indications for machine state (pre installation, during puppet first run, etc) ... and probably a lot of other small stuff that I cant think of now

Page 45

Our Solution

Web Portal

Puppet

Configuration DBInventory

DHCP/DNS..

Page 46

Where does GINI live?

GINI

DBDNS

TFTP

CERTIFICATES

CMDB

DHCP

LOGS LSF

Future work