Deploying OpenStack with Ansible

Created by: Kevin Carter & Curtis Collicutt

OS-Ansible-Deployment

Deploying OpenStack with Ansible

presentation > osad <<EOP

1

www.rackspace.com 2

Who am I?

Kevin Carter● Developer at the Rackspace Private Cloud● Open source activist● Cloud operations junkie● Python aficionado● Recovering rubyist● Beer lover● Soccer fanatic

www.rackspace.com 3

Who am I?

Curtis Collicutt● Lead OpenStack Engineer @ AURO● Information Security● Storage systems● How do computers even?● Films

4www.rackspace.com

@robertcathey

Cloud infra is hard. OpenStack is cloud infra. OpenStack is hard. This is news, apparently.

“”

www.rackspace.com 5

OSAD and what we’re about

● Deployer Experience● Vanilla OpenStack● Scalability● Stability

www.rackspace.com 6

Why are we here?

In late 2013, the Rackspace Private Cloud team set out to solve our common deployment, maintenance, scalability, and stability problems.

www.rackspace.com 7

Distribution packaging of OpenStack

● Out of date packages ● Out of band configuration● Packages include proprietary patches● Time to bug resolution is longer than it

should● Broken dependencies

www.rackspace.com 8

Available deployment tooling

● Maybe, sometimes, sorta, eventually “consistent”, kinda?● Upgrades difficult or impossible● Steep learning curve

www.rackspace.com 9

Legacy architecture does not scale

● Almost all deployment systems reference an architecture that suffers from the “controller 1 controller 2” model

● VIP failover for OpenStack supporting services bound to break and when it does it’ll break spectacularly!

www.rackspace.com 10

What we devised

A source-based installation of OpenStack, built within LXC containers, using a multi-master architecture orchestrated and deployed via Ansible.


Why Ansible?

● Community engagement

● Orchestration

● Almost no code

● Low barrier to entry

● Crazy powerful, stupid simple


Why containers?

● LXC ≈ More bare metal

● Compatible with many networking architectures

● Supports an LVM backend

● Stable


What is OSAD?

OSAD == OpenStack Ansible Deployment

● Uses LXC containers to isolate components and services

● Deploys OpenStack from upstream sources

● Runs on Ubuntu 14.04

● Built for production

● No proprietary secrete sauce

○ But you could bolt on as much as you want

● Created following the KISS principle


● All Ansible tasks and roles target multiple nodes, even if that number is

a multiple of one (1)

○ EVERYTHING is tagged!

● Process separation on infrastructure components (controller nodes)

○ Microservice-like, where it makes sense

OSAD architecture


● Galera multi-master cluster

● RabbitMQ with mirrored queues and deterministic sorting of the master

queues

● “Cheese shop” index build for your environment stored within your

environment

OSAD infrastructure components


● OSAD does not know about the “all in one” deployment

○ LXC enables the base system to deploy a multi-node cloud even

with only one physical node

○ An AIO in our gate job emulates a 32 node cloud

● Neutron with the Linux Bridge agent offer stability and supportability

○ Open vSwitch is feature-full but Linux Bridge “just works”™

OSAD scale


Community project

● We support Juno and Icehouse but the code contains

Rackspace-isms

● Kilo is our first “community” release of OSAD

● 41 contributors presently in the project

○ Not all Rackers


Community project

We take our role within the community seriously!

# Lines of change between Juno and Kilogit diff --stat juno kilo 1158 files changed, 39061 insertions(+), 81368 deletions(-)


● Deployer experience: Ansible● Vanilla OpenStack: Source-based installation● Scalability: Built within LXC containers● Stability: Obviously!

OSAD and what we’re about


OSAD configuration

● OSAD configuration is your window into inventory

○ lives in /etc/openstack_deploy

● Dynamic inventory generated via config

● Compatible with Ansible static inventory

● Execution made simple using the openstack-ansible wrapper.


OSAD deployment

# Change to the playbooks directory

cd /opt/os-ansible-deployment/playbooks

# Open your favorite terminal multiplexer

tmux new -s osad-deployment

# Do all the things!

openstack-ansible setup-everything.yml

Go get coffee|food|beer, this will take a minute.


What an OpenStack deployment looks like with OSAD

Diagram not built to scale.Derived from an All in One Installation.


OSAD adding a compute node

# Execute run limited to the nova_compute group

openstack-ansible setup-everything.yml \

--limit nova_compute

compute_hosts: compute1: ip: 172.29.236.201 compute2: ip: 172.29.236.202 compute3: ip: 172.29.236.203 compute4: ip: 172.29.236.204 compute5: ip: 172.29.236.205

EDIT: /etc/openstack_deploy/openstack_user_config.yml


OSAD adding an infrastructure node

# Execute the setup with a limit on the infra groups we’re adding


--limit os-infra_all,\

shared-infra_all,\

identity_all

shared-infra_hosts: infra1: ip: 172.29.236.101

os-infra_hosts: infra1: ip: 172.29.236.101

identity_hosts: infra1: ip: 172.29.236.101

EDIT: /etc/openstack_deploy/openstack_user_config.yml


OSAD reconfiguring all of neutron

# Execute a run limited to neutron_all


--limit neutron_all

global_overrides: provider_networks:

- network:

container_bridge: "br-

vxlan"

container_type: "veth"

container_interface:

"eth10"

ip_from_q: "tunnel"

type: "vxlan"

range: "1:1000"

net_name: "vxlan"

group_binds:

-

neutron_linuxbridge_agent

EDIT: /etc/openstack_deploy/conf.d/neutron_networks.yml


● AURO - Public OpenStack Cloud● Compute, Volume, Swift, Heat, Neutron● Canadian data residency, ownership● Vancouver region, Toronto up next

AURO - OpenStack


● Not using as much as we’d like● Mostly the infrastructure components

○ Rabbit, Galera, Memcached, etc● Absolutely invaluable as an example● Will continue to bring in more OSAD components as we operate over

time● Team somewhat new to config mgmt

AURO & OSAD - What we are using


● Great example of:o Using Ansibleo Deploying OpenStacko Testing - All in one, use of OpenStack infra

● Already supports Kilo● Packaging and deploying OpenStack (ie. not using OS packages -

Python Wheels very cool)● Segregation of services

AURO & OSAD - What we like


● Public cloud● Midonet● Different HA Model● Billing● Support Model

○ Multiple tiers of internal support

AURO - Differences from OSAD


● Not to restart services in same run as changeso Need to control restarts in HA manner, rolling

● Every task tagged● Continuously run (from Ansible Tower and/or Jenkins)● Installing once is easy, operating forever is hard● Ansible to help manage many small changes faster● People don’t ssh into servers, only Ansible

AURO - Ansible Guiding Principles


● Easy to use mostly idempotent modules then run a command or shell

task and make a mess of it● changed_when: False is too easy to stumble with● Multiple environments● Being able to run one-time commands across all systems is as powerful

as it is dangerous

$ ansible -a reboot all

AURO - Ansible Struggles


● Deploy OpenStack from source

● Segregation of services● More monitoring● Ansible callback plugins are useful● Learn more from OpenStack testing infra● Need a couple modules

o Midoneto Swift

AURO - Near term improvements


● Be “Pluggable?” (What does that even mean?)o Neutron network - eg. Midoneto HA model - eg. ECMP/BGP load balancing

● Balancing community roles and playbooks with custom

requirements● Learn how to consume OSAD properly

AURO - OSAD Comments/Ideas/Questions

www.rackspace.com

● Secrets (eg. Hashicorp Vault, KeyWhiz)● Continuous integration...err integration● Caching (Ansible has Redis, other ideas?)● What is the “future” of config mgmt? Must be more than just

pkg/config/start/bootstrap● Change request workflow

34

AURO - Configuration Management Future


● Increase community participation in OSAD

○ Community members wanted!

○ Pull requests welcome :)

● Build out the operational modules found within the upstream

● Modular Dynamic inventory

● etc . . .

Where does Ansible and OpenStack go from here?


Q & A

Twitter: @cloudnullIRC: cloudnull

Email: [email protected]

Twitter: @ccollicuttIRC: serverascode

Email: [email protected]

mailto:[email protected]

mailto:[email protected]

37www.rackspace.com

OSAD makes OpenStack™ NOSAD“”

www.rackspace.com

Thank You!

EOP

38

Technology

Deploying OpenStack with Ansible