SUSEcon Crowbar Presentation

Crowbar: The Cloud Operations Framework

Greg Althaus Dell Principal Engineer and Crowbar Co-Founder

Revolutionary Cloud & Big Data Solutions

Agenda

• What Led to Crowbar?

• What is Crowbar?

• Crowbar Architecture

• SUSE + Crowbar = A Path to Clouds

• Crowbar 2.0 and Beyond

• Q&A

2


What Led to Crowbar?

3


Crowbar <3 OpenStack “Boxes to working OpenStack in <4 hours” • Crowbar’s birthright was “an OpenStack installer”

– But we started it knowing there was a much bigger problem

• Like many open source projects, OpenStack is complex and changes

constantly – Want the innovation and afraid of change

• To create an OpenStack project, needed:

– To get customers running quickly – Have a way to cope with frequent updates – Connect a lot of parts together – Be part of the open source community

• Got & get substantial uplift from

the OpenStack community

4 Confidential


Problem: Clouds do not operate themselves!

HW

HW

SW

SW

Ops Ops

Operations

Expertise

Servers Networks

Storage Etc.

5 Confidential


Ops automation (“devops”) jumpstarts learning, reduces complexity and manages scale

HW

HW

Ops

Ops

SW

SW

Operations

Expertise

Operations Automation

(DevOps)

Servers Networks

Storage Etc.

6 Confidential


What is DevOps? Why is it important?

• DevOps extended to infrastructure is “CloudOps”

• Operational Model for Software – “Operations as Code” – Highly automated – Layered deployments (no image based) – Continuous deployment

• Shaped by Amazon Cloud

– API only access to infrastructure – Elastic resources that are easier to create and destroy than change

• Configuration Management Tooling – Chef & Puppet – Builds applications from open source modules/recipes – Not quite coding, but far from manual

7 Confidential


OpenStack has a lot of interlocking parts!

8 Confidential


…and it matters how they are deployed!

Cookbooks

9 Confidential


Images vs. Layers: Lifecycle

Images: Replacement Layers: Upgrade

I+A+U+O/S

Config

OS

U Bar v1 Foo

I

Co

nfi

g

I+A+U+O/S

Config

I+A+U+O/S

Config

I+A+U+O/S

Config

OS

U Bar v2 Foo

I

Co

nfi

g

Bar v2

10 Confidential


CloudOps for OpenStack

There are lot of pieces, but they share many elements!

OS

U

Nova Node

I

Co

nfi

g

OS

U

Nova API

I

Co

nfi

g

OS

U

Nova Node

I

Co

nfi

g

OS

U

Nova Node

I

Co

nfi

g

OS

U

Nova Node

I

Co

nfi

g

OS

U

Nova Node

I

Co

nfi

g

OS

U

Nova Node

I

Co

nfi

g

OS

U

Nova DB

I

Co

nfi

g

OS

U

Nova Cntrl

I

Co

nfi

g

OS

U

Dash Board

I

Co

nfi

g

OS

U

Nagios

I C

on

fig

OS

U

Crowbar Chef

I

Co

nfi

g

11 Confidential


CloudOps for OpenStack

Plus interconnections are highly complex and dynamic

OS

U

Nova Node

I

Co

nfi

g

OS

U

Nova API

I

Co

nfi

g

OS

U

Nova Node

I

Co

nfi

g

OS

U

Nova Node

I

Co

nfi

g

OS

U

Nova Node

I

Co

nfi

g

OS

U

Nova Node

I

Co

nfi

g

OS

U

Nova Node

I

Co

nfi

g

OS

U

Nova DB

I

Co

nfi

g

OS

U

Nova Cntrl

I

Co

nfi

g

OS

U

Dash Board

I

Co

nfi

g

OS

U

Nagios

I C

on

fig

OS

U

Crowbar Chef

I

Co

nfi

g

12 Confidential


What is Crowbar?

13


Crowbar, the Cloud Deployer++

• What is Crowbar – Not only reduces the effort for deploying clouds – Also provides an operational model for managing them

• Field proven technologies – Build on locally deployed Chef Server – Raw servers to full cloud in <2 hours, multi-node from the ground up – Hardened with over a year of deployments

• Apache 2 open source (on github/dellcloudedge)

– Multi-apps (OpenStack & Hadoop), multi-OS (Ubuntu, RHEL, CentOS, SUSE), Development “in the open” w/ community engagement

• Hadoop & OpenStack releases (dell.com/OpenStack)

– Cactus, Diablo and Essex!

• Open version is not Dell hardware Limited – This is been an important factor in adoption

14 Confidential


Crowbar in the Open (Community!)

• Community – Focused on OpenStack, expanding in other areas – Engineering benefits from community test and dev – Dell engineering active participants

• Developed in the Open – Most code development is current & live on github – Needed for collaboration – Allows faster response to customers and partners

• Active Community – 300+ email list users & nearly 500 Github followers – 500+ downloads of the Crowbar image – Partners: Intel, Opscode, SUSE, etc – Customers: telecom, cloud, financial, etc

15 Confidential


Cloud Taxonomy

Admin Software Web Services & APIs

Customer Management

Entitlement, rights Billing

Self Service Portal

Metering

Infrastructure Software

Abstraction Software

Physical

Software as a Service IT as a Service Platform as a Service Infrastructure as a Service Everything as a Service

HVAC Power Facility

Network

Environmentals

Compute Switch Storage

OS

Hardware Virtualization

Operating System Virtualization

Application Run-Time Virtualization

Data Store

Monitoring

Intelligent Resource Manager

Workload Lifecycle Management

Platform Provisioning

Ser Gov/Workflow Automation

Orchestration

Info

rma

tio

n S

erv

ice

Ma

na

ge

me

nt

Le

ga

cy

Ma

na

ge

me

nt

Fir

ew

all

Re

po

rtin

g

An

aly

tic

s

IPS

S

ec

uri

ty

LD

AP

/AD

S

SO

Overarching Systems


Crowbar Builds the Cloud Taxonomy

Admin Software OpenStack APIs

Customer Management Customer Management

Entitlement, rights Entitlement, rights Billing Billing

OpenStack Dashboard

Metering Metering

Infrastructure Software

Abstraction Software

Physical

Software as a Service IT as a Service Platform as a Service Infrastructure as a Service Everything as a Service

BIOS BIOS RAID RAID NETWORK NETWORK

Quantum

Environmentals Environmentals

Nova Switch Switch Swift

OS OS

Hardware Virtualization Hardware Virtualization





Data Store Data Store

Monitoring Monitoring

Intelligent Resource Manager Intelligent Resource Manager



Ser Gov/Workflow Automation Ser Gov/Workflow Automation

Orchestration Orchestration

Info

rma

tio

n S

erv

ice

Ma

na

ge

me

nt

Info

rma

tio

n S

erv

ice

Ma

na

ge

me

nt

Le

ga

cy

Ma

na

ge

me

nt

Le

ga

cy

Ma

na

ge

me

nt

Fir

ew

all

F

ire

wa

ll

Re

po

rtin

g

Re

po

rtin

g

An

aly

tic

s A

na

lyti

cs

IPS

IP

S

Ke

yst

on

e

LD

AP

/AD

L

DA

P/A

D

SS

O

SS

O

Overarching Systems

Overarching Systems

Nova Glance Swift

OS

KVM

Crowbar


Crowbar Builds Up Clouds in layers

De

ll “

Cro

wb

ar”

O

ps

Ma

na

ge

me

nt

Core Components & Operating Systems


Cloud Infrastructure Cloud Infrastructure

Physical Resources Physical Resources

APIs, User Access, & Ecosystem Partners APIs, User Access, & Ecosystem Partners

18 Confidential


1) Hardware Configuration

De

ll “

Cro

wb

ar”

O

ps

Ma

na

ge

me

nt






19 Confidential


2) OS and Ops Infrastructure

De

ll “

Cro

wb

ar”

O

ps

Ma

na

ge

me

nt






20 Confidential


3) Cloud Platform Integration

De

ll “

Cro

wb

ar”

O

ps

Ma

na

ge

me

nt






21 Confidential


4) Cloud App Management

De

ll “

Cro

wb

ar”

O

ps

Ma

na

ge

me

nt






22 Confidential


Late Binding Matters! It’s “Just-in-time” Deployment Late binding is a critical component of Crowbar flexiblity

Discover

Inventory Provision

Deploy Layer N

Allocate with System View Allocate with System View

Network Config

• Integrated configuration of RAID, BIOS, provision & networking, so no longer have to do that work in advance.

• Do NOT need to make these decisions early in the process

• Can adapt and change over time.

User App & Ops decisions drive the entire config & deploy process!

23 Confidential


Crowbar Architecture + Demo

24


OpenStack Orchestration: Chef & Crowbar

Server(s)

De

ll “

Cro

wb

ar”

O

ps

Ma

na

ge

me

nt

Operating System Operating System

Chef Server Chef

Server

Network Configuration Network Configuration

DHCP DHCP

Cro

wb

ar

Ma

gic

Ha

pp

en

s!

Cro

wb

ar

Ma

gic

Ha

pp

en

s!

Orchestration State Machine Orchestration State Machine

Run List

Config.

Hardware Config (uses “Sledgehammer”)

Hardware Config (uses “Sledgehammer”)

OpenStack Components OpenStack Components

25 Confidential


Dell Cloud Installer: Crowbar Barclamps D

ell

“C

row

ba

r”

Op

s M

an

ag

em

en

t

Core Components & Operating Systems Core Components & Operating Systems

Cloud Infrastructure & Dell IP Extensions Cloud Infrastructure & Dell IP Extensions



* Crowbar has potential to service other programs beyond OpenStack

Crowbar Crowbar

Deployer Deployer

Provisioner

Network RAID

BIOS IPMI

NTP NTP

DNS DNS Logging Logging

Nova Swift Glance

Nagios Nagios Ganglia Ganglia

enStratus Cloudera

Dashboard

Block

Horizon

Keystone Hadoop

26 Confidential


De

ll “

Cro

wb

ar”

O

ps

Ma

na

ge

me

nt

What is a Barclamp?

Barclamp Barclamp

Chef Recipes Chef Recipes O/S App

Packages O/S App

Packages Crowbar API & Partial UI

Crowbar API & Partial UI

Components & Scripts


• Crowbar API & Partial UI • Adds states and transitions to orchestration • Customizes UI specific to barclamp • Visual extensions to nodes/networks/utils

• Chef Recipes • DevOps description of application • Enforced repeatable configuration

• O/A App Packages • Operating system dependencies (Debs & RPMs)

• Components & Scripts • Ad hoc configuration not covered by Chef

27 Confidential


De

ll “

Cro

wb

ar”

O

ps

Ma

na

ge

me

nt

What is a Barclamp?

Barclamp Barclamp

Chef Recipes Chef Recipes O/S App

Packages O/S App

Packages Crowbar API & Partial UI

Crowbar API & Partial UI



Barclamp Barclamp

Barclamp Barclamp

• Interact as service layers • Have dependencies and pre-requisites • Can operate on multiple operating systems • Have independent version streams • Leverage/reuse Chef cookbooks

28 Confidential


Crowbar’s Proposal Workflow

• Proposals manage deployments • Proposals are a core design objective to ensure customer flexibility

and system level management • Workflow:

• Templates are default configurations without resources • Proposals all users to choose config and nodes (resources) • Applying a proposal tells Crowbar to “make it so”

Deployed Services

Barc

lam

p Active Proposal

Proposal Proposal

Nodes Apply

Active Active Nodes Nodes Apply Apply

Discovered Nodes

29 Confidential


Crowbar Orchestration = State Machine

30 Confidential


Networking in Crowbar

Server Server

IMPI

eth0

eth1

eth#

VLAN

VLAN

VLAN

VLAN

Teamed Teamed

IP Allocations

DHCP

Network Barclamp Network Barclamp

Chef Client Chef Client

Crowbar Does NOT Configure The Switch! O/S Only* not

mgm’d not

mgm’d

Crowbar Discovers Switches and Ports

* For now

31 Confidential


Networking in Crowbar

Server Server

IMPI

eth0

eth1

eth#

VLAN

VLAN

VLAN

VLAN

Teamed Teamed

IP Allocations

DHCP

Network Barclamp Network Barclamp

Chef Client Chef Client

not mgm’d

not mgm’d

1g1

1g2

1g3

Conduit

Conduit

Conduit

Other Barclamps Request “Conduits”

Interface & Conduits Normalize Networks Between Nodes

32 Confidential


SUSE + Crowbar = A Path to Clouds

33


SUSE Cloud with Crowbar

• Version of Crowbar that runs on SUSE

• Packaged to take advantage of SUSE repositories and methods

• Provides OpenStack Essex

• Extends and builds SUSE Studio • Released and available as of the beginning

of September

34 Confidential


Team Work

35 Confidential

• A huge key to the success has been team work

• Joint development

• Design and planning discussion

• Not just limited to SUSE, but definite leader


Crowbar 2.0 and Beyond

36 Confidential


Crowbar 2.0 (currently in progress) http://bit.ly/crowbar2 • Simplify use of Chef, eliminate Crowbar requirements in Chef recipes

– Reduce the initial effort required to leverage Crowbar – Opens Crowbar to a broader audience (see Upstreaming)

• Provide heterogeneous/multiple operating system deployments to

enable: – Multiple versions of the same OS running for upgrades – Different OS operating simultaneously (and deal with heterogeneous

packaging issues) – Accommodation of no-agent systems like locked systems (e.g.: virtualization

hosts) and switches (aka external entities) – UEFI booting in Sledgehammer

• Strengthen networking abstractions

– Allow networking configurations to be created dynamically (so that users are not locked into choices made before Crowbar deployment)

– Better manage connected operations – Enable pull-from-source deployments that are ahead of (or forked from) available packages

37 Confidential

http://bit.ly/crowbar2


Crowbar 2.0 (currently in progress) http://bit.ly/crowbar2 • Improve Crowbar's core database and state machine to enable

– Larger scale concerns – Controlled production migrations and upgrades

• Packaging rework

– Lead by some of the SUSE work – Create distribution-based packages

• Upgrade components

– Start the process of enabling security updates – Staged Updates from test to production environments – Start considering application

• Other important items

– Make documentation more coupled to current features and easier to maintain

– Upgrade to Rails 3 to simplify code base, security and performance – Deepen automated test coverage and capabilities

38 Confidential

http://bit.ly/crowbar2


Crowbar 2.0 (currently in progress) http://crowbar.sync.in/crowbar2

• Network models (http://youtu.be/eTSTLecNlGM ) -

http://crowbar.sync.in/crowbar2-network

• Online mode (http://youtu.be/U05apbOqq0k ) - http://crowbar.sync.in/crowbar2-online

• Pull from source (http://youtu.be/Bo4V6Dax3LU ) - http://crowbar.sync.in/crowbar2-pullsource

• Heterogeneous OS (http://youtu.be/xL-2eefqH8s) - http://crowbar.sync.in/crowbar2-multi-os

• Attribute driven recipes (http://youtu.be/wtUNJfrHU1Q) - http://crowbar.sync.in/crowbar2-attributeconfig

39 Confidential

http://crowbar.sync.in/crowbar2

http://youtu.be/eTSTLecNlGM

http://youtu.be/eTSTLecNlGM





http://youtu.be/U05apbOqq0k

http://youtu.be/U05apbOqq0k

http://crowbar.sync.in/crowbar2-online




http://youtu.be/Bo4V6Dax3LU

http://youtu.be/Bo4V6Dax3LU

http://crowbar.sync.in/crowbar2-pullsource




http://youtu.be/xL-2eefqH8s



http://crowbar.sync.in/crowbar2-multi-os






http://youtu.be/wtUNJfrHU1Q

http://crowbar.sync.in/crowbar2-attributeconfig






Q&A

40 Confidential


Thanks for Listening!

• Greg Althaus [email protected] Twitter: @galthaus

41 Confidential

mailto:[email protected]

Technology

SUSEcon Crowbar Presentation