Deploying Foreman in Enterprise Environments

best practices

and lessons learned…

Nils Domrose

Gent, February, 3 2014

▸ senior linux systems engineer at

inovex GmbH

▸ worked as a network engineer,

software developer and systems

engineer

▸ using foreman for about 1 year

▸ using bare-metal deployment for

ages

▸ life is short – let’s focus on

interesting stuff!

2

About me

__endy__

@endyman

https://plus.google.com/+NilsDomrose

#irc

▸ http://www.inovex.de

▸ offices in [‘Pforzheim’, ‘Karlsruhe’, ‘Cologne’, ‘Munich’]

▸ we have open positions...

3

About inovex

We use technology to make our customers happy. And ourselves.

1

SHAPE

Consulting

2

BUILD

Application

Development

3

RUN

IT Engineering &

Operations

5

TEACH

Academy

4

TRACK

Business

Intelligence

4

▸ the chicken & egg problem

▸ foreman components

▸ we are the borg

▸ fail! foreman HA

▸ best practices

Agenda

▸ some shell scripts to push

installer and answers.yaml to

new system (installer is subject

to rpm/deb packaging)

▸ run a custom kafo installer

▸ setup repos

▸ configure nics (bonding)

▸ setup pacemaker &

corosync

▸ setup drbd

▸ setup dhcp server

▸ setup postgreSQL

▸ setup puppetDB

▸ run foreman kafo installer

The Chicken & Egg Problem

...deploying your deployment infrastructure

▸ kafo?

▸ kafo! - imagine master-less

puppet plus config data in yaml

(like hiera)

▸ can’t tell wether it’s the chicken

or the egg or both but it’s cool

▸ stores parameters for your

modules in answers.yaml

▸ simply copy your modules into

the installer’s modules directory

▸ or even use puppet-librarian to

manage your modules (poor

man’s git-submodules)

The Chicken & Egg Problem

build your first kafo installer

https://github.com/rodjek/librarian-puppet

https://github.com/theforeman/kafo

The Chicken & Egg Problem

kafo - getting started

i=my-installer; mkdir $i && cd $i && kafofy $i && mkdir /etc/$i && cat

> /etc/$i/$i.yaml << EOF

> ---

> :log_dir: /var/log/$i

> :log_name: configuration.log

> :log_level: info

> :no_prefix: false

> :mapping: {}

> :answer_file: /etc/$i/answers.yaml

> :installer_dir: $(pwd)

> :modules_dir: $(pwd)/modules

> :default_values_dir: /tmp

> :colors: true

> :kafo_modules_dir: /usr/lib/ruby/vendor_ruby/kafo/modules

> EOF

root@foreman:~# cd /var/tmp/

root@foreman:/var/tmp# i=my-installer; mkdir $i; cd $i; kafofy $i

▸ Never give up, never surrender!:

depending on your kafo gem it

might not (yet) be as easy as:

▸ It’s more like:

▸ delete the modules directory

The Chicken & Egg Problem

get the modules

▸ next: package it, install it

root@foreman:/var/tmp/my-installer# librarian-puppet install --clean

root@foreman:/var/tmp/my-installer# ll modules

total 16

drwxr-xr-x 4 root root 4096 Jan 29 12:25 ./

drwxr-xr-x 7 root root 4096 Jan 29 12:17 ../

drwxr-xr-x 7 root root 4096 Jan 29 12:25 ntp/

drwxr-xr-x 6 root root 4096 Jan 29 12:24 stdlib/

root@foreman:/var/tmp/my-installer# cat Puppetfile

forge "http://forge.puppetlabs.com"

mod 'puppetlabs/stdlib'

mod 'ntp',

:git => 'git://github.com/puppetlabs/puppetlabs-ntp.git'

root@foreman:/var/tmp/my-installer# rm -Rf modules/

▸ run the librarian

▸ create a Puppetfile

9

Foreman Components

foreman

foreman

frontendREST API

unattended

resources

apache mod_passenger

Port 80 HTTP Port 443 HTTPS

▸ rails application

▸ apache mod_passenger

▸ Webinterface

▸ REST API

▸ unattended resources –

rendered templates

database

▸ WEBrick based REST

server to manage

supporting infrastructure

10

Foreman Components

smart-proxy

DNSDHCP TFTP

apache

mod_passenger

Port 8140 HTTPS

Port 8443 HTTPS

WEBrick server

BMCpuppetpuppet

CA

Port 69

UDP

tftpdISC

dhcpd

Port 67

UDP

nsupdateipmitool

▸ flows at each side to local puppet

and smart proxy

▸ all VLANs directly access puppet ca

▸ all VLANs directly access foreman

unattended resources

▸ we need DNS only once (so let’s

limit TSIG key distribution)

11

Foreman Components

placement of foreman components

Foreman

Smartproxy, puppet, CA,

DNS, DHCP

Smartproxy, puppet,

DHCP

VLAN A Site1

VLAN B Site1

VLAN C Site2

VLAN D Site2

fw

fwmgmt VLAN2

mgmt VLAN1

12

Foreman

smartproxy, puppet, CA,

DNS, DHCP

smartproxy, puppet,

DHCP, Proxy

VLAN A Site1

VLAN B Site1

VLAN C Site2

VLAN D Site2

fw

fwmgmt VLAN2

mgmt VLAN1

▸ limit access to the smartproxy and

foreman to local clients only

▸ open dedicate flow from proxy to

foreman and puppet

Foreman Components

optimizing comms flows

▸ install apache mod_rewrite on

remote smartproxy

▸ rewrite/proxy unattended resources

▸ rewrite/proxy puppet CA resources

13

Foreman Components

optimizing comms flows

...

rewrite_rule => [

‘^/unattended/(.*)$ http://foreman.mysite.com/unattended/$1 [P]’,

]

rewrite_rule => [

‘^/([^/]+/certificate.*)$ https://puppetca.mysite.com:8140/$1’,

]

...

▸ we must identify proxied servers

▸ foreman url must be customized

▸ templates must be adapted

14

Foreman Components

foreman customization

▸ create parameter in hostgroups for each

segment (unattended_url in 1.4 does not

seam to help here…)

▸ adapt templates

▸ enable tokens (default in 1.4)

Integrating foreman into corporate

infrastructure is:

▸ not a technical issue

▸ mostly even not a security issue

▸ It’s about convincing people to do

things differently

▸ It’s about responsibilities

▸ It’s about fear

▸ It’s about laziness

15

Resistance is futile

Integrating foreman into corporate infrastructure

16

Resistance is futile

The one with the wrong <enter vendor name> DHCP product

The corporate DHCP does not provide a proper API

and changes are distributed across the DHCP

infrastructure in a slow, asynchronous manner…

Deploy your own DHCP server:

- You must be able to create and delete leases

instantly

- You only need it to bootstrap your servers. Later a

static NIC configuration can be applied using your

favorite cfgmgmt. No, it’s not that static – we can

use ENC variables to update IP addresses.

- Repeat: I won’t create my leases manually and I

won’t wait for changes to be replicated until I

power-on my server

17

Resistance is futile

The one with the DHCP caused Zombiecalypse

You are about to deploy your own DHCP but people

complain that your DHCP will nuke the corporate

datacenter and will cause an epic loss of revenue…

Ask for dedicated VLANs and for ip-helper configs

on routers and firewalls

- Your DHCP is limited to your VLANs only

- You only need one DHCP server

- It’s not an evil plan to make the guy running the

corporate DHCP redundant

18

Resistance is futile

The one with static DNS

You kindly ask for TSIG keys to create your A and

PTR records automatically but the DNS is managed

manually and there is no TSIG key although dynamic

updates are enabled…

Make a deal: you won’t talk about the missing TSIG

keys and get your key or keytab and permissions

right after the guy fixed the security issue.

Ask for dedicated zones and for credentials which

allow you to manage your zones using nsupdate

- Alternatively ask for delegation of your zones and

setup a dedicated DNS server (protected by keys)

19

FAIL!

Foreman HA

19

pacemaker pacemaker

DRBD

postgreSQL

Apache

mod_passenger

corosync

puppetDB

HA IP(s)

DRBD

postgreSQL

Apache

mod_passenger

puppetDB

HA IP(s)

node1 node2

▸ use pacemaker/corosync to provide

basic resilience for postgreSQL and

puppetDB, puppetCA

▸ If you scale out, you might setup a

nice pg-pool2 cluster

▸ puppetmasters could be ran active

active as well as the foreman rails

application (keep memcached in

mind)

▸ on pure smartproxies no pacemaker

cluster is required. A simple failover

using VRRP (via keepalived) should

be sufficient – if you replicate the

tftp and dhcp data (for puppet you’ll

have to do the exercise anyhow.)

▸ use separate IP addresses for each service so you

can split up stuff later on

▸ If you can’t use DHCP the bootdisk plugin might be

an option (currently .deb package 1.2.3 is broken

for 1.4 but version plugin source is ready for 1.4)

▸ LDAP or kerberos5 (1.4) for user authentication

▸ use Locations for filtering

▸ classes are not tight to locations so you better use

the same set of classes and import only once from

you master puppet master

▸ use the column plugin for better overview

▸ ENC and smart variables are nice – ever configured

dhcp pool as yaml hash? Use smart variables as

switches, and for simple datatypes only - do the

complex stuff in service modules - ideally by using

data in modules

20

Best practices

▸ integrate the reverse proxy feature into

smart-proxy (there is a ticket for that

somewhere)

▸ implement iso image provisioning based

vm installation in conjunction with the

bootdisk plugin

▸ support more configuration management

stacks

▸ better support for discovery, ipmi and

hardware provisioning (like HW-Raid)

21

Wishlist

...secret guide to world domination

22

Thank You!

Contact

Nils Domrose

Senior Systems Engineer

inovex GmbH

Office cologne

Schanzenstr. 6-20

51063 Köln

nils.domrose@inovex.de