Paul Angus (ShapeBlue) - Push infrastructure with Ansible #DOXLON

Push Infrastructure with Ansible Paul Angus

Cloud Architect & Chief Technology Strategist ShapeBlue [email protected]

Twitter: @CloudyAngus

@CloudyAngus

v  How we use Ansible to push out entire cloud infrastructures

v  A bit of background v  How we use Ansible

v  Work flow v  Code snippets

Push Infrastructure with Ansible

@CloudyAngus

v  Cloud Architect & Chief Technology Strategist for ShapeBlue

v  Apache CloudStack Committer v  Specialise in….

v  Designing and deploying enterprise and public clouds v  Helping organisations use their cloud

v  Involved with CloudStack before donation to Apache Foundation

v  Designed clouds for Orange, TomTom, PaddyPower, Ascenty, BSkyB

About Me

Unofficial Ansible Evangelist/Cheerleader

@CloudyAngus

“ShapeBlue are expert builders of public & private clouds. They are the leading global

Apache CloudStack integrator & consultancy”

About ShapeBlue

@CloudyAngus


What is CloudStack?

@CloudyAngus

CloudStack is a open source IaaS platform. Hypervisor agnostic -‐ KVM, vSphere, XenServer, LXC, OVM, Baremetal

CloudStack orchestrates hypervisors and network appliances to give simple control to complex tasks through API or web GUI.

(Yes, it’s like OpenStack)

What is CloudStack

@CloudyAngus

Think ‘your own Amazon Web Services’

Public clouds (SPs/MSPs) General public can create or log into instances themselves

Private Clouds (Enterprises) Anyone who wants to be able to orchestrate their environment

Hybrid Clouds (Enterprises) Balance/share load between their own DC and a Public Cloud

Who uses CloudStack

@CloudyAngus

This is a UI (for Linux guys

who’ve forgotten).

What is CloudStack

@CloudyAngus

It’s OK it has an API cloudmonkey create network name=DemoNet displaytext=DemoNet zoneid=$ZONEID networkofferingid=$NEWORKOFFERINGID account=$ACCOUNTNAME domainid=$DOMAINID

cloudmonkey list networks domainid="$DOMAINID" filter=id,name | jq '.network[] | select(.name=="DemoNet") | .id‘

cloudmonkey deploy virtualmachine networkids="$NETWORKID" serviceofferingid="$SERVICEOFFERINGID" templateid="$TEMPLATEID" zoneid="$ZONEID" account="$ACCOUNTNAME" domainid="$DOMAINID" startvm=true displayname=HTTPServer1 displayvm=true name=HTTPServer1

What is CloudStack

@CloudyAngus


Why? (Duuurrr)

@CloudyAngus

CEO

Why

“Building CloudStack environments using Ansible?

Are you just having fun with Ansible?”

@CloudyAngus

Why

Talented Cloud Architect

“Noooo, if we can automate the building of environments using a powerful, simple and agentless technology we can make building at scale easy

while ensuring that our results are consistent and repeatable.”

“Building CloudStack environments using Ansible? Are you just having fun with

Ansible?”

@CloudyAngus

Why

CEO “That would be excellent. Go ahead. Oh, and here’s a pay rise”

“Noooo, if we can automate the building of environments using a powerful, simple and agentless technology we can make building at scale easy

while ensuring that our results are consistent and repeatable.”

@CloudyAngus

Some of that might actually have happened.

Disclaimer

Why

@CloudyAngus

CSForge™ v  CSForge delivers the rapid deployment of a standardised CloudStack powered IaaS

cloud for small production deployments, or medium scale POCs or pilots. The framework can be used as a basis for public cloud or enterprise private cloud deployments

Production v  Cloud-‐scale environments initial deployment typically 24 – 100s of hosts v  Often multi-‐hypervisor

Why

@CloudyAngus

Test/Dev

Need to be able to create full environments to test: v  CloudStack release candidates v  CloudStack features v  ShapeBlue patches

Why

@CloudyAngus

Why Ansible v  Technical:

v  Client/Server architecture not required v  Only SSH connectivity required (password or public/private keys) v  …making it easier to build virgin environments v  Modules can be in any language capable of returning JSON or key=value text pairs v  Has an API

v  User:

v  Much shallower learning curve v  Don’t need to learn a programming language (i.e. Ruby)

v  Not as many pre-‐existing playbooks (recipes/manifests) about, but improving with Ansible Galaxy

@CloudyAngus

Typical Logical Production Topology

Management (1Gb)

CIMC (1GB)

CIMC (1GB)

Management (1GB LACP Bond)

WWW

Compute Hosts

Storage Nodes

GuestPublicStorage

(10GB LACP Bond)

Management Hosts

Management1GB Active/Passive Bond

Load Balancers

DNS/NTP Servers

MySQL Master & Slave

ACS Management Servers

DeploymentServer

Management(1GB Active/Passive Bond)

Storage

(10GB LACP Bond)

iDRAC (1GB)

Storage (10GB LACP Bond)

Public

Cloud Compute Hosts

Storage Link to ACS Managers

@CloudyAngus

v  3 zones v  2 geographic

locations v  Upgrade done,

then tests run for a week. Then VRs restarted

Client Test Environment

CCP3.0.7B

MySQL

CPBM 2.2

MySQL

ESXi 1a

ESXi 1b

ESXi 1cvCenter Appliance

ESXi 2a

ESXi 2b

ESXi 2cvCenter ApplianceZone 1 (local)

Zone 2 (local)

Zone 3 (remote)

NFS

NFS

VPN VPN

ESXi 2a

ESXi 2b

ESXi 2cvCenter Appliance

NFS

@CloudyAngus

Using Ansible to Deploy CloudStack Environments

How? (variables, I Love ‘em)

@CloudyAngus

# Copyright (C) ShapeBlue Ltd -‐ All Rights Reserved # Unauthorized copying of this file, via any medium is strictly prohibited # Proprietary and confidential # Released by ShapeBlue <[email protected]>, April 2014 -‐-‐-‐ -‐ name: Ensure selinux python bindings are installed yum: name=libselinux-‐python state=present -‐ name: Ensure the Apache Cloudstack Repo file is configured template: src=cloudstack.repo.j2 dest=/etc/yum.repos.d/cloudstack.repo -‐ name: Ensure selinux is to permissive command: setenforce permissive changed_when: false -‐ name: Ensure selinux is set permanently selinux: policy=targeted state=permissive -‐ name: Ensure CloudStack packages are installed yum: name=cloudstack-‐management state=present

-‐ name: Ensure MySQL Client is present yum: name=mysql state=present -‐ name: Ensure vhd-‐util is present get_url: url="{{ vhdutil_url }}" dest=/usr/share/cloudstack-‐common/scripts/vm/hypervisor/xenserver/vhd-‐util mode=0755 -‐ name: Ensure CloudStack Usage Service is installed yum: name=cloudstack-‐usage state=present -‐ name: Ensure CloudStack Usage Service is started service: name=cloudstack-‐usage state=started -‐ include: ./../galera-‐cluster/tasks/main.yml when: "{{ db_type }} == 'galera'" -‐ include: ./setupdb.yml

CloudStack Management Server Role

@CloudyAngus

# Copyright (C) ShapeBlue Ltd -‐ All Rights Reserved # Unauthorized copying of this file, via any medium is strictly prohibited # Proprietary and confidential # Released by ShapeBlue <[email protected]>, April 2014 [cloudstack] name=cloudstack baseurl=http://{{ acs_build_repo }}/{{ acs_build_path }} enabled=1 gpgcheck=1

cloudstack.repo.j2

@CloudyAngus

# ansible-‐playbook management_vms_build.yml -‐i acs_demo_hosts -‐-‐extra-‐vars "acs_build_path=tescng/centos/master"

Playbook Arguments

@CloudyAngus

Create bare VM +

yum install git

git clone mega repo with roles

etc.

Install & configure Ansible (included in repo)

Update hosts and group_vars

Create Deployment

server (from role) locally

PXE boot hosts/mgmt VMs to

bare OS

Push application configuration to VMs and Hosts

Building Environments

@CloudyAngus

Distributed Architecture

TEST STAGE PROD

Deployment Srvrwith cloned repo



@CloudyAngus

v  PXE VM against Ansible created KickStart file

v  DNS Servers (Bind 9) v  Galera Cluster (Maria DB10 – any size) v  MySQL Master/Slave v  Clustered CloudStack Management Servers v  HA Proxy (HA pair with keepalived)

Management Infrastructure Roles

@CloudyAngus

v  PXE host against Ansible created KickStart or Answer file v  KVM

v  Straight configuration of networking and packages

v  XenServer v  XE commands via shell v  Use ‘facts’ to determine installed patches

v  vSphere v  99% configured through kickstart file v  SSH enabled (through KickStart)

Hypervisor Host Roles

@CloudyAngus


Snippets

@CloudyAngus

[management_hosts:children] cloudstack_manager_hosts loadbalancer_hosts deployment_hosts db_hosts [kvm_hosts:children] pod1_cluster1 pod1_cluster2 [pod1_cluster1] 10.101.1.13 pxemac=00:1c:c0:ec:b5:bc hostname=kvm-‐p1-‐c1-‐1 mgmt_ip=10.101.1.13 mgmt_netmask=255.255.255.0

Host File

@CloudyAngus

Host File [mysql_master_host]

10.101.1.6 pxemac=00:50:56:8E:19:20 hostname=mysqlmasterhost mgmt_ip=10.101.1.6 mgmt_netmask=255.255.255.0 serverid=1

[pod1_cluster1]

10.101.1.13 pxemac=00:1c:c0:ec:b5:bc hostname=kvm-‐p1-‐c1-‐1 mgmt_ip=10.101.1.13 10.101.1.14 pxemac=00:1c:c0:ec:b5:43 hostname=kvm-‐p1-‐c1-‐2 mgmt_ip=10.101.1.14

Creating PXE files

@CloudyAngus

File format: AA:BB:CC:DD:EE:FF must be 01-‐aa-‐bb-‐cc-‐dd-‐ee-‐ff

-‐ name: create centos pxeconfig files

template: src=pxelinuxcfg-‐centos6.j2 dest=/ptpboot/pxelinux.cfg/01-‐{{ hostvars[item]['pxemac'] | lower | replace(":","-‐") }} with_items:

-‐ "{{ groups.management_hosts }}“

-‐ name: create KVM pxeconfig files

template: src=pxelinuxcfg-‐kvm.j2 dest=/ptpboot/pxelinux.cfg/01-‐{{ hostvars[item]['pxemac'] | lower | replace(":","-‐") }} with_items:

-‐ "{{ groups.kvm_hosts }}"

Creating PXE files

@CloudyAngus

# Ansible built pxeconfig script DEFAULT menu.c32 menu ctle ShapeBlue CSForge Deployment cmeout 10 LABEL {{ centos_iso_version }}-‐KVM kernel htp://{{ hostvars[inventory_hostname]['mgmt_ip'] }}/{{ centos_iso_version }}/isolinux/vmlinuz append initrd=htp://{{ hostvars[inventory_hostname]['mgmt_ip'] }}/{{ centos_iso_version }}/isolinux/initrd.img ramdisk_size=15491 ip=dhcp ksdevice={{ ks_device }} ks=htp://{{ hostvars[inventory_hostname]['mgmt_ip'] }}/{{ pxe_files_dir }}/{{ kvm_ks_file }} MENU LABEL {{ centos_iso_version }} for KVM

pxelinuxcfg-‐kvm.j2

@CloudyAngus

# Specifies the keyboard layout keyboard {{ keyboard_lang }} # Used with an HTTP install to specify where the install files are located url -‐-‐url htp://{{ hostvars[inventory_hostname]['mgmt_ip'] }}/{{ centos_iso_version }} # Assign a stacc IP address upon first boot & set the hostname network -‐-‐onboot yes -‐-‐device {{ ks_device }} -‐-‐bootproto dhcp -‐-‐noipv6 # Set the root password rootpw {{ mgmt_root_password }} mkdir /root/.ssh curl htp://{{ hostvars[inventory_hostname]['mgmt_ip'] }}/{{ publickey_file_name }} >> /root/.ssh/authorized_keys

kickstart.j2

@CloudyAngus

opcon subnet-‐mask {{ dhcp_netmask }}; opcon routers {{ dhcp_router }}; opcon broadcast-‐address {{ dhcp_broadcast }};

opcon domain-‐name-‐servers {{ int_dns_servers | join(", ") }}; opcon domain-‐name "{{ dns_domain }}"; subnet {{ dhcp_subnet }} netmask {{ dhcp_netmask }} { range {{ dhcp_range_starcp }} {{ dhcp_range_endip }}; # KVM Hosts

{% for host in groups['kvm_hosts'] %} host {{ hostvars[host]['hostname'] }} { hardware ethernet {{ hostvars[host]['pxemac'] | lower }}; fixed-‐address {{ hostvars[host]['mgmt_ip'] }}; } {% endfor %}

Creating dhcpd.conf

@CloudyAngus

v  Reverse Zone – Format: 2 IN PTR blah1.domain.com.

; Address Records {% for host in groups['management_hosts'] %}

{{ hostvars[host]['mgmt_ip']|split('.')[3] }} IN PTR {{ hostvars[host]['hostname'] }}. {% endfor %} {% for host in groups['xenserver_hosts'] %} {{ hostvars[host]['mgmt_ip']|split('.')[3] }} IN PTR {{ hostvars[host]['hostname'] }}. {% endfor %}

DNS Entries into Zone File – Jinja2

@CloudyAngus

v  Custom Filter to Split string import re

def split_string(string, seperator=' '): return string.split(seperator)

def split_regex(string, seperator_patern): return re.split(seperator_patern, string) class FilterModule(object):

''' A filter to split a string into a list. ''' def filters(self): return { 'split' : split_string,

'split_regex' : split_regex, }

Custom Filter

@CloudyAngus

v  ansible.cfg

# set plugin path directories here, separate with colons

accon_plugins = /usr/share/ansible_plugins/accon_plugins

callback_plugins = /usr/share/ansible_plugins/callback_plugins

conneccon_plugins = /usr/share/ansible_plugins/conneccon_plugins

lookup_plugins = /usr/share/ansible_plugins/lookup_plugins

vars_plugins = /usr/share/ansible_plugins/vars_plugins

filter_plugins = /usr/share/ansible_plugins/filter_plugins:/CSForge/custom_plugins/filter_plugins

Add path if required

@CloudyAngus

v  ACS_env_group_vars.yml

db_type: galera # opcons: mysql, mysql-‐masterslave, galera

v  build_db_servers.yml -‐-‐-‐

-‐ hosts: db_hosts

roles:

-‐ { role: galera-‐cluster, when: "'{{ db_type }}' == 'galera'" }

-‐ { role: mysql, when: "'{{ db_type }}' != 'galera'" }

Conditional Roles


Paul Angus Cloud Architect ShapeBlue [email protected]

Twitter: @CloudyAngus