1

Introduction & Overview of OpenStack for IaaS Clouds

Keith Basil Principal Product Manager, OpenStackRed HatJune 12, 2013

2

Your presenter..

personalVirginia hare scrambler, plays chess..

professionalRed HatCloudscaling, Time Warner Cable,FederalCloud.com, Cisco and a couple of startups

blendedskype/twitter/github/irc, life: noslzzp

3

Agenda

✦ Introduction to OpenStack

✦ OpenStack Architecture

✦ Understanding the Elastic Cloud

✦ OpenStack in the Real World

4

What Is OpenStack?

OpenStack is both Software and a Community

5

What Problem Does OpenStack Solve?

OpenStack provides a framework for buildingelastic cloud infrastructure at massive scale.

✦Facilitates management of

✦COMPUTE,

✦NETWORK, and

✦STORAGE resources

✦Provides natural infrastructure for elastic applications

We will cover the “elastic cloud” later in more detail..

6

OpenStack Components Provide..

✦Compute

✦Virtual machine management

✦Comprehensive hypervisor support

✦Networks

✦IP address management

✦Security services

✦Storage

✦Volumes (block storage)

✦Object storage for VM images and files

7

OpenStack as Community

Technical Committee - defines and stewards technical direction

Board of Directors - provides strategic and financial oversight of Foundation Resources

User Committee - created to represent enterprise, academic and service provider users

8

OpenStack History and CadenceHavana

(Oct 2013)

Grizzly(Apr 2013)

Folsom(Oct 2012)

Essex(Apr 2012)

Diablo(Sep 2011)

Cactus(Apr 2011)

Bexar(Feb 2011)

Austin(Oct 2010)

Ceilometer & Heat integrated(Oct 2013)

Maturation of Quantum and Cinder, focus on upgrade support(Apr 2013)

Quantum (Networking) full inclusion, Volume Service added(Oct 2012)

Dashboard and Identity service released, Quantum incubated(Apr 2012)

First “Production Ready” release(Sep 2011)

Scaling enhancements, support for many hypervisors(Apr 2011)

OpenStack Compute ready, initial release of Image Service(Feb 2011)

Initial release, Object Storage Production Ready, Compute in testing

6-month cadence

9

OpenStack Trends, Growth & MilestonesHavana

(Oct 2013)

Grizzly(Apr 2013)

Folsom(Oct 2012)

Essex(Apr 2012)

Diablo(Sep 2011)

Cactus(Apr 2011)

Bexar(Feb 2011)

Austin(Oct 2010)

First OpenStack release with Red Hat code

OpenStack Foundation formed(Sep 2012)

47 committers acrossthe top ten companies

Red Hat assigns its first developer to the OpenStack community(Aug 2011)

71 committers

230committers

Largest OpenStack Summit to date, enterprise customer keynotes

Data extracted from Google Trends (keyword: OpenStack) and Bitergia reports.“Committers” shown above indicate the number of individual committers across the top ten contributing companies.

10

#3 OVERALL CODE

CONTRIBUTOR TOESSEX

(Apr 2012)

11

#2 OVERALL CODE

CONTRIBUTOR TOFOLSOM

(Oct 2012)

12

#1 OVERALL CODE

CONTRIBUTOR TOGRIZZLY

(Apr 2013)

13

Red Hat Contributors

Many contributors. One community.No coincidence.

14

Agenda

✦ Introduction to OpenStack

✦ OpenStack Architecture

✦ Understanding the Elastic Cloud

✦ OpenStack in the Real World

15

OpenStack Design and Architecture

DASHBOARD(Horizon)

COMPUTE

(Nova)

IDENTITY SERVICE

(Keystone)

Modular architecture

Based on a (growing) set of core services

Designed for Scalability and Elasticity

BLOCK STORAGE

(Cinder)

OBJECT STORE

(Swift)

NETWORKING

(Quantum)

IMAGE SERVICE

(Glance)

16

OpenStack Dashboard (Horizon)

Provides simple self service UI for end-users

Basic cloud administrator functions (No infrastructure management)

Define users, tenants and quotas

DASHBOARD(Horizon)

COMPUTE

(Nova)

IDENTITY SERVICE

(Keystone)

BLOCK STORAGE

(Cinder)

OBJECT STORE

(Swift)

NETWORKING

(Quantum)

IMAGE SERVICE

(Glance)

17

OpenStack Compute (Nova)

Supports multiple hypervisors (KVM, Xen, LXC, Hyper-V, ESX)

Native OpenStack API and Amazon EC2 API support

Distributed controller services handle scheduling, API calls, etc.

DASHBOARD(Horizon)

COMPUTE

(Nova)

IDENTITY SERVICE

(Keystone)

BLOCK STORAGE

(Cinder)

OBJECT STORE

(Swift)

NETWORKING

(Quantum)

IMAGE SERVICE

(Glance)

18

OpenStack Block Storage (Cinder)

Provides block storage for virtual machines (persistent disks)

Similar to Amazon EBS service

Plugin architecture for vendor extensions

DASHBOARD(Horizon)

COMPUTE

(Nova)

IDENTITY SERVICE

(Keystone)

BLOCK STORAGE

(Cinder)

OBJECT STORE

(Swift)

NETWORKING

(Quantum)

IMAGE SERVICE

(Glance)

19

OpenStack Networking (Quantum)

Network Service

Provides framework for Software Defined Networking (SDN)

Allows integration of hardware and software based network solutions

DASHBOARD(Horizon)

COMPUTE

(Nova)

IDENTITY SERVICE

(Keystone)

BLOCK STORAGE

(Cinder)

OBJECT STORE

(Swift)

NETWORKING

(Quantum)

IMAGE SERVICE

(Glance)

20

OpenStack Image Service (Glance)

Stores and retrieves disk images (virtual machine templates)

Supports Raw, QCOW, VMDK, VHD, ISO, OVF & AMI/AKI

Backend storage : Filesystem, Swift, Amazon S3

DASHBOARD(Horizon)

COMPUTE

(Nova)

IDENTITY SERVICE

(Keystone)

BLOCK STORAGE

(Cinder)

OBJECT STORE

(Swift)

NETWORKING

(Quantum)

IMAGE SERVICE

(Glance)

21

OpenStack Object Storage (Swift)

Modeled after Amazon's S3 service

Provides simple service for storing and retrieving arbitrary data

Native API and S3 compatible API

DASHBOARD(Horizon)

COMPUTE

(Nova)

IDENTITY SERVICE

(Keystone)

BLOCK STORAGE

(Cinder)

OBJECT STORE

(Swift)

NETWORKING

(Quantum)

IMAGE SERVICE

(Glance)

22

OpenStack Identity Service (Keystone)

Common authorization framework

Manages users, tenants and roles

Pluggable backends (SQL, PAM, LDAP, etc)

DASHBOARD(Horizon)

COMPUTE

(Nova)

IDENTITY SERVICE

(Keystone)

BLOCK STORAGE

(Cinder)

OBJECT STORE

(Swift)

NETWORKING

(Quantum)

IMAGE SERVICE

(Glance)

23

Connecting the Components

DASHBOARD(Horizon)

COMPUTE

(Nova)

IDENTITY SERVICE

(Keystone)

OpenStack uses message queues for communicationbetween components

Supported queueing backends: RabbitMQ, Qpid and ZeroMQ

BLOCK STORAGE

(Cinder)

OBJECT STORE

(Swift)

NETWORKING

(Quantum)

IMAGE SERVICE

(Glance)

Message Queue

24

Incubation Project: Ceilometer

Metering & Monitoring(Ceilometer)

- Collects meter data (CPU, network, etc)

- Designed for integration and extensibility

- Data collected is made available via REST API

- Message signing provides non-repudiation

Graduated from Incubation to Integrated status for the Havana release

25

Incubation Project: Heat

Application Orchestration(Heat)

- Provides template driven cloud application orchestration

- Modeled after AWS CloudFormation

- Targeted to provide advanced functionality such as high availability and autoscaling

- Introduced by !

Graduated from Incubation to Integrated status for the Havana release

26

Spin me up a VM!

And make it LARGE!

Umm, Do I know you? I need to

see some papers!!

Keystone

Ok, we need to find a place to build

this VM.Nova

Tag - you’re it!

VM

capacity capacity

capacity Papers are good. Time to get to work!Nova

Node

Quantum, I need a network with all

the trimmings!Quantum

Here’s your IP, default route and

FW settings.

Cinder, have that volume

ready for me?

Node

Indeed I do. Don’t forget to

mount it!

SwiftGlance

Hey Glance, can I get the RHEL 6.4

image?

Node

8)

Let’s Follow a Request..

Thank you OpenStack!!

8)

It’s rendering

time!

27

Agenda

✦ Introduction to OpenStack

✦ OpenStack Architecture

✦ Understanding the Elastic Cloud

✦ OpenStack in the Real World

28

Elastic Cloud != Enterprise Virtualization

On-demand self-serviceBroad network access

Resource poolingRapid elasticity

Measured service

Many applications on each serverMaximum server utilization

Minimum server count

29

Workload Evolution

CLOUD WORKLOADS

✦Smaller stateless VMs

✦Lifecycle measured in hours to months

✦Applications scale out horizontally with new VMs

✦Applications expect failure

TRADITIONAL WORKLOADS

✦Larger stateful VMs

✦Lifecycle measured in years

✦Applications scale up (more vCPU, vRAM)

✦Applications NOT designed to tolerate failure

30

“Pets vs Cattle” (Scale Up vs Scale Out)

The above adapted from Tim Bell, CERNhttp://www.slideshare.net/noggin143/20121017-openstack-cern-accelerating-cienceOriginal “Pets vs. Cattle” is attributed to Bill Baker, Microsoft Distinguished Engineer.

http://mem-pass.org/summits/2011/pdfs/DBA-302-HD.pdf

“Future application architectures should use Cattle but Pets with strong configuration management are viable and still needed”

- Tim Bell, CERN“

Scale Up- Servers are like pets.

Pets are given names, are unique, lovingly hand raised and cared for. When they get ill, you nurse them back to health

Scale Out- Servers are like cattle.

Cattle are given numbers and are almost identical to each other. When they get ill, you get another one.

31

Why the Elastic Cloud Is Needed

Our Data is too large

✦We are past the point of single computers being able to efficiently handle our data

Service Requests are too large

✦Client devices are more plentiful than ever

Or, BOTH..

32

Why the Elastic Cloud Is Needed

Servers fail - Deal with it!1

✦Assume you could start with super reliable servers(MTBF of 30 years!)

✦If you build a system with 10,000 of those servers

✦You will watch one fail every day

Fault-tolerant software is inevitable!

[1] Adapted from Jeff Dean’s presentation on Designs, Lessons and Advice from Building Large Distributed Systemshttp://www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf

33

“Fault Tolerance” (survival) in Nature

elastic, agile, scaled with small failure domains

34

Pets in the Water

35

but.. Unnatural Acts Can Be Impressive

36

Agenda

✦ Introduction to OpenStack

✦ OpenStack Architecture

✦ Understanding the Elastic Cloud

✦ OpenStack in the Real World

37

OpenStack’s Increasing Maturity

We are seeing organizations struggle with elastic cloud adoption

✦Unfortunately, Frankenclouds are being white boarded daily!

✦“Pet friendly” strategies are underway

✦Green field deployments work well

Knowledge capture is early but evolving

✦OpenStack Operations Guide

✦OpenStack Security Guide

38

OpenStack Deployment Considerations

OpenStack’s two most important deployment questions:

✦What does the NETWORK look like?

✦“The 90’s called. They want their network architecture back.”

✦ What does the cloud HARDWARE look like?

✦Amazon and Google have figured it out: embrace and extend!

39

Question: The Network?

40

Network Elasticity is Required..

NODE NODE NODE NODE NODE NODE NODE NODE

NODE NODE

NODE NODE

NODE NODE NODE NODE NODE NODE NODE NODE

NODE NODE

NODE NODE

NODE NODE NODE NODE NODE NODE NODE NODE

NODE NODE

NODE NODE

NODE NODE NODE NODE NODE NODE NODE NODE NODE

BLOCKSTORE

BLOCKSTORE

NODE

NODE NODE NODE NODE NODE NODE NODE

BLOCKSTORE

BLOCKSTORE

NODE

NODE NODE NODE NODE NODE NODE NODE

NODENODE

NODE

BLOCKSTORE

BLOCKSTORE

BLOCKSTORE

BLOCKSTORE

Elastic Cloud Resource Map

NODE

NODE

41

Because your cloud will grow..

Each unit here could be a server, or a rack of servers.

42

And continue to grow..

1,152 servers shown? Or 1,152 racks with 16 servers each (18,432 servers)?

hint: it shouldn’t matter

43

Spine and Leaf Topology

Ask your friendly network vendor for guidance

✦Cisco, ARISTA, Brocade, Juniper, Force10, etc.

http://bradhedlund.com/2012/01/25/construct-a-leaf-spine-design-with-40g-or-10g-an-observation-in-scaling-the-fabric/

44

Question: The Hardware?

45

1/1

1/2

1/4

1/8

n1-standard-8-d

n1-standard-4-d

n1-standard-2-d

n1-standard-1-d

m1.xlarge

m1.large

m1.medium

m1.small

m1.class

n1-s

tand

ard.

clas

s

xlarge

large

medium

small

Public Cloud VM Instances Exposed!

46

Deployment: Sizing a Compute Node

xlarge

large medium

small

Solve for the biggest VM inthe class

Smaller VMs are fractional proportions of the largest. This facilitates efficient hardware use and scheduling.

Compute Hardware Node (m1.class)128GB memory, (16) 1TB disks, (2) E5-2670 CPU

1/1 1/2 1/4 1/8

47

Deployment: Sizing a Compute Node

xlarge

Compute Hardware Node (m1.class)128GB memory, (16) 1TB disks, (2) E5-2670 CPU

xlarge

small

small

small

small

small

small

small

small

medium medium

medium medium

large

xlarge xlargelarge

small

small

small

small

small

small

small

small

Given the machine config above, it would support:

(4) n1-standard-8-d, (8) n1-standard-4-d, (16) n1-standard-2-d, (32) n1-standard-1-d

(8) m1.xlarge, (16) m1.large, (32) m1.medium, (64) m1.small

48

Plan for the Resource Service Level

Compute/StorageNetwork Fabric

Cloud Controller

ResourceService

Level

49

OpenStack Deployment Profiles

50

PayPal

Profile Highlights:

✦113 million registered accounts; PCs or mobile devices in 190 different markets with 25 different currencies

✦Targeting 90 percent coverage for several thousand nodes in nine to 12 months

✦DIY team leveraging OpenStack community support

51

OpenStack Accelerating Science

Profile Highlights:

✦Using OpenStack to support particlephysics research

✦Currently running 500 nodes and 2000 VMs

✦Immediate plans:

✦Deploying production OpenStack running Grid software

✦Intends to use Ceilometer, bare metal for tenants and LBaaS

✦Ramping to 15,000 hypervisors with 100k - 300k VMs by 2015

52

National Security Agency

Profile Highlights:

✦One of the NSA's largest hosting platforms

✦Number of users, systems, servers, storage, applications users: [REDACTED]

✦Agility, flexibility and scalability providing better support for mission systems

✦Big Data is truly big

✦Has been working with OpenStack since Cactus

53

In Summary

54

OpenStack ...

✦Is open source software and vibrant community

✦Provides a framework for an elastic cloud.

✦Requires fresh thinking for deployments

✦Is being deployed successfully at scale today

55

And finally..

56

57

Thank You!

Red Hat IaaS Overview & RoadmapAndrew Cathrow — Sr. Virt. Product Manager, Red HatSimon Grinberg — Principal Product Manager, Red HatWednesday, June 123:40 pm - 4:40 pm

Red Hat OpenStack Performance & ScaleMark Wagner — Senior Principal Engineer, Red HatWednesday, June 124:50 pm - 5:50 pm

OpenStack ArchitectureRussell Bryant — Principal Software Engineer, Red HatThursday, June 133:40 pm - 4:40 pm

Check out these sessions!

58