A one stop solution for Puppet and Openstack · Openstack Keystone (Identity) RBAC Integrated with...

A one stop solution

for Puppet and Openstack

Daniel Lobato Garcia

daniel.lobato.garcia@cern,ch

@eLobatoss

What is CERN

Between Geneva and the Jura mountains, straddling the Swiss-French border

Mission: learn what is the universe made of and how does it work?

3

Fundamental

questions in

physics

Why do particles have mass?

What is 96% of the universe made of?

Why isn’t there anti-matter in the universe?

What was the state of matter after the Big Bang?

4

8/12/2013 Document reference 5

8/12/2013 Document reference 6

8/12/2013 Document reference 7

8/12/2013 Document reference 8

Current status

• 270 Openstack hypervisors

• 2900 virtual machines

• 300 users

• 14 Puppet masters

• 6 Foreman backend nodes

• Some production services migrating to our

cloud – early birds

9

Goals

• Ramp up to 15K hypervisors – 150-200K

vms in 2015

• Multi-site (Hungary)

10

8/12/2013 Document reference 11

8/12/2013 Document reference 12

Why?

• Unnecessary homebrew stack of tools

• Shift to cloud standards with minimal

customizations

• High turnover – can’t teach new tools

13

Why?

• Symbiotic relationship with the community

14

Openstack?

• Modular IaaS free open source project

• APIs ~compatible with those of Amazon

15

Openstack Nova

(compute)

Cloud fabric controller

16

Openstack

Keystone (Identity)

RBAC

Integrated with LDAP

Multiple auth* methods

17

Openstack Glance

(Images)

Discovery, registration,

delivery of images

18

Openstack Horizon

(Dashboard)

19

Modules

• Puppet definitions for every use case you

can imagine.

• Dynamic environments

• Hadoop node

• Openstack hypervisor

• … you name it

20

21

Workflow..?

Modules and Git

• Manifests and hieradata are version

controlled

22

23

Git workflow

Puppet masters

24

Easy cherry pick

25

Git workflow

26

Git workflow

Jens

‘Puppetfiles’

Separate repositories

Makes environments from

YAML config files

Puppet master

rsync

27

upstream.yaml --- default: production notifications: ai@cern.ch overrides: modules: foreman: upstream puppet: upstream

‘Puppetfiles’

Foreman

• Lifecycle management tool for VMs and

physical servers

• External Node Classifier – tells the puppet

master what a node should look like

28

29

30

Power operations & Foreman

8/12/2013 Document reference 31

Foreman Proxy

Physical

box IPMI

Physical

box IPMI

Physical

box IPMI

VM VM VM

Openstack

Nova API

Openstack VM creation

8/12/2013 Document reference 32

Openstack VM creation

8/12/2013 Document reference 33

Openstack VM creation

8/12/2013 Document reference 34

Scalability experiences

• Split up services

• Puppet – critical vs non critical

35

12 backend nodes

Batch

4 backend nodes

Interactive

Scalability experiences

• Foreman – split into different services

36

ENC Reports

processing UI/API

Load balancer

9443 – UI/API

9444 – Reports

9445 – ENC

…

Scalability experiences

• Autoscale via alarms (Heat)

• Define situations (i.e: load threshold..)

• Spin up VMs as needed

37

Scalability guidelines

38

github.com

/

cernops

39

40