Rackspace Private Cloud presentation for ChefConf 2013

1

April 26, 2013

Deploying OpenStack with Chef and Operational Tooling

#chefconf 2013

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

Who Are We?

Justin Shepherd

Principal Architect

Rackspace Private Cloud

github.com/galstrom21

Joseph Breu

Deployment Engineer

Rackspace Private Cloud

github.com/rackerjoe

@rackerjoe

3RACKSPACE® HOSTING | WWW.RACKSPACE.COM

• Only consume Open Source projects

• All changes go upstream

• CHEF-3889 - tunables for postgres

• CHEF-3849 - redirect login bug

• Everything must be automated through the deployment and management stacks.

• Everything we build is Open Souce

• https:/github.com/rcbops

• github.com/rcbops-cookbooks

Rackspace Private Cloud Philosophy


• Jenkins is used for testing and gating of our cookbooks

• Syntax Verification (pep8, foodcritic)

• Unit Tests for OpenCenter

• Full deployment of OpenStack utilizing the cookbooks with the proposed changeset applied

• Functional Tests of OpenStack

• OpenStack API Testing

• OpenStack CLI Testing

• Upstream opscode cookbooks are tested before inclusion

Testing


How Did We Get Here?

5


• Installing Bexar and Cactus by hand

• Problems

• Deployed from trunk

• No packages available for distributions

• Not easily repeatable

• Not easily supported

In The BeginningEarly 2011


Defining Our Mission

7


• We need to make it easy to install

• We need to make the installation repeatable

• We need to model “Real World Deployments”

The MissionEarly 2011


Automating OpenStack Deployment

9

10


• Deploying Cactus and Diablo

• Puppet

• Inherited from OpenStack Team

• djeep based

• Very simplistic, not flexible

• Represented the developer case - not real world deployments

10

Automate!Late 2011

11


Mission Status

We need to make it easy to install FAIL

We need to make the installation repeatable PASS

We need to model “Real World Deployments” FAIL

12


• Crowbar

• Dell platform for deployment

• Chef based

• Very heavyweight

• ISO installation

• Boot shim for system discovery

• Used for a few deployments and contributed code upstream

• Upgrades were painful or not possible at all

12

Automate!Late 2011 - Diablo

13


Mission Status

We need to make it easy to install Almost Passed


We need to model “Real World Deployments” Almost Passed

14


Feature Creep

14

15




• We need to mode “Real World Deployments”

• We need to handle updates

• We need to quickly consume new OpenStack services

• We need to plug into existing infrastructure (PXE, DNS, LDAP, etc)

15

We now need to…March 2012

16


How did Puppet do?

We need to make it easy to install FAIL


We need to model “Real World Deployments” FAIL

We need to handle updates FAIL

We need to quickly consume new OpenStack Services FAIL

We need to plug into existing infrastructure FAIL

17


How did Crowbar do?

We need to make it easy to install Almost Passed


We need to model “Real World Deployments” Almost Passed

We need to handle updates FAIL

We need to quickly consume new OpenStack Services FAIL

We need to plug into existing infrastructure FAIL

18


Next Steps in Automation

18

19


• Deploying Essex

• Started to build our own cookbooks

• Heavy utilization of Chef search

• Modeled real world deployments

• Support for multiple Operating Systems

Next Steps in AutomationMarch 2012

20


• Deploying Folsom

• Creating LWRPs for REST based Services

• Rewrites of many cookbooks to support multiple Operating Systems

• Debian and RedHat based

• Search wrapper functions for ease of use

Next Steps in AutomationLate 2012

21


How did we do?

We need to make it easy to install PASSED

We need to make the installation repeatable PASSED

We need to model “Real World Deployments” PASSED

We need to handle updates PASSED

We need to quickly consume new OpenStack Services PASSED

We need to plug into existing infrastructure Almost PASSED

22


More Feature Creep

22

23




• We need to mode “Real World Deployments”

• We need to handle updates

• We need to quickly consume new OpenStack services

• We need to plug into existing infrastructure (PXE, DNS, LDAP, etc)

• We need provide Highly Available OpenStack Services

• We need to provide support for Vendor Integration for Block Storage

• We need to support Software Defined Networks

23

We now need to…December 2012

24


• keepalived for failover IP

• haproxy for load balancing REST based services

• Multi-master MySQL replication and keepalived

• Active/Passive RabbitmQ and keepalived

• This is sketchy in Folsom - better in Grizzly

Creating Highly Available ServicesEarly 2013

25


• Support for multiple Cinder Backends

• Currently NetApp and EMC

• Support for SDN

• Currently only OVS

• LDAP Support

• Greenfield LDAP deployments only

• No AD support... yet...

Other ServicesEarly 2013

26


How did we do?

We need to make it easy to install FAILED!





We need to plug into existing infrastructure PASSED

We need to provide Highly Available OpenStack services

PASSED

We need to provide support for Vendor Integration for Block Storage

PASSED

We need to Support Software Defined Networks PASSED

27


The Tipping Point

27

28


• We had a system that can do real interesting things

• You now need to have staff with strong Chef knowledge to use it

• And strong OpenStack knowledge...

• OpenSource Chef is great for some things – but not everything

• We need to orchestrate changes across a cluster

• We need to do it in a defined order

The Tipping PointEarly 2013

29


OpenCenter

29

30


• Simplifies, speeds up deployment

• Lowers Operator overhead

• Minimizes the internal OpenStack knowledge

• Provides resiliency in the Cloud

• API for command line access

What is OpenCenter?

31


What is OpenCenter?

32


What is OpenCenter

33


Lowers Operator Overhead

34


Resiliency for Private Cloud

35


Solver – Automates Operator Tasks

36


Extensible

Write your own Python/Bash scripts to create workflows that extend OpenCenter

37


How did we do?

We need to make it easy to install PASSED





We need to plug into existing infrastructure PASSED

We need to provide Highly Available OpenStack services

PASSED

We need to provide support for Vendor Integration for Block Storage

PASSED

We need to Support Software Defined Networks PASSED

38


Errata

38

39


• Partial Searches and Templates

• Remove dependencies on monitoring and logging

• Make our cookbooks more easily consumable

• Upstream our cookbooks to the OpsCode OpenStack repos

• Utilize OpenStack CI testing and gating

Looking forward

4040

RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218

US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM

RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM

Technology

Rackspace Private Cloud presentation for ChefConf 2013