01.19.2011 AIIT InfoTalk on OpenStack

OpenStack - Towards an Open CloudAIIT InfoTalk #26 2011

Adam JohnsonTwitter: @adjohn

1Friday, January 21, 2011

Application Platforms Undergoing A Major Shift

2

Applications are quickly moving to the cloud•#1 priority is virtualization•#2 is cloud computing[Based on a Gartner Study]


Midokura Overview

3

• Founded in January 2010• Based in Tokyo• 9 employees (and growing)• Former Google, Amazon engineers

• Developing “Cloud enabling technologies”• Virtualized networking• Cloud storage• Distributed architecture and management

• We think - OpenStack is going to be HUGE


What is OpenStack?

4

Overview of the project


OpenStack: The Mission

5

Provision virtual machines on standard hardware at massive scale

Software to reliably store billions of objects distributed across standard hardware

creating open source software to build public and private clouds


OpenStack: The Mission

6

"To produce the ubiquitous Open Source cloud computing platform that will meet the needs of public and private cloud providers regardless of size, by being simple to implement and massively

scalable."


7


OpenStack History

8

2005 2010

March May June July

Rackspace Cloud

developed

Rackspace decides to Open SourceCloud Software

NASA OpenSources Nebula

Platform

OpenStack formedb/w Rackspace and

NASA

First designSummit in Austin, TX


OpenStack History

9

2011July October November February

OpenStack launches with 25+

partners

First ʻAustinʼ coderelease with 35+

partners

First public DesignSummit in San

Antonio

Second ʻBexarʼ coderelease planned


OpenStack Founding Principles

10

•Apache 2.0 license (OSI), open development process

•Open design process, 2x year public Design Summits

•Publicly available open source code repository

•Open community processes documented and transparent

•Commitment to drive and adopt open standards

•Modular design for deployment flexibility via APIs


Community with Broad Commercial Support

11


OpenStack in Japan

12

•Japan OpenStack User Group (JOSUG)•website: http://openstack.jp•google group: openstack-ja


http://openstack.jp

http://openstack.jp

Why is OpenStack Important?

13

•Open eliminates vendor lock-in

•Working together, we all go faster

•Freedom to federate, or move between clouds


HOW TO: Turn Racks of Standard Hardware Into a

Cloud with OpenStack


Start with an open, scalable platform

OpenStack Compute OpenStack Object Storage

CLOUD OS

OpenStack Image Service


User Control Panel

TicketingSystem

NetworkManagement

MonitoringSystems

Host Server Management

ECOSYSTEM


CLOUD OS


Add 3rd party tools from the ecosystem


User Control Panel

TicketingSystem

NetworkManagement

MonitoringSystems


AccountBilling

Admin CLITools

Live ChatSupport

AccountManagement

ECOSYSTEM

PUBLIC CLOUD


CLOUD OS



User Control Panel

TicketingSystem

NetworkManagement

MonitoringSystems


ECOSYSTEM

Admin ControlPanel

Dept. Accounting Chargeback

UserManagement

Enterprise SoftwareIntegration Systems

PRIVATE CLOUD


CLOUD OS


Integrate with existing enterprise systems18Friday, January 21, 2011

OpenStack Compute

19

Software to provision virtual machines on standard hardware at massive scale


Asynchronous eventually consistent

communication

REST-based API

Horizontally and massively scalable

Hypervisor agnostic: support for Xen ,XenServer, Hyper-V,

KVM, UML and ESX is coming Hardware agnostic: standard hardware, RAID not required

OpenStack Compute Key Features


API: Receives HTTP requests, converts commands to/from API format, and sends requests to cloud controller

Cloud Controller: Global state of system, talks to LDAP, OpenStack Object Storage, and node/storage workers through a queue

User Manager

ATAoE / iSCSI

Host Machines: workers that spawn instances

Glance: HTTP + OpenStack Object Storage for server imagesOpenStack Compute


Server Groups1 GigE Connectivity

Dual Quad CoreRAID 10 Drives

Public Network

Private Network(intra data center)

Management

Example OpenStack Compute Hardware

(other models possible)


Compute Components

23

•API Server: Interface module for command and control requests•Designed to be modular to support multiple APIs•In current release: OpenStack API, EC2 Compatibility Module•Approved blueprint: Open Cloud Computing Interface (OCCI)

•Message Queue: Broker to handle interactions between services•Currently based on RabbitMQ

•Metadata Storage: ORM Layer using SQLAlchemy for datastore abstraction

•In current release: MySQL•In development: PostgreSQL

•User Manager: Directory service to store user identities•In current release: OpenLDAP, FakeLDAP (with Redis)

•Scheduler: Determines the placement of a new resource requested via the API

•Modular architecture to allow for optimization•Base schedulers included in Austin: Round-robin, Least busy


Compute Components

24

•Compute Worker: Manage compute hosts through commands received on the Message Queue via the API

•Base features: Run, Terminate, Reboot, Attach/Detach •Volume, Get Console Output

•Network Controller: Manage networking resources on compute hosts through commands received on the Message Queue via the API

•Support for multiple network models•Fixed (Static) IP addresses•VLAN zones with NAT

•Volume Worker: Interact with iSCSI Targets to manage volumes•Base features: Create, Delete, Establish

•Image Store: Manage and deploy VM images to host machines


OpenStack Object Storage

25

Software to reliably store billions of objects distributed across standard hardware


REST-based API Data distributed evenly throughout system

Hardware agnostic: standard hardware, RAID not required

OpenStack Object Storage Key Features

No centraldatabase

Scalable to multiple petabytes, billions of objects

Account/Container/Object structure (not file system, no nesting) plus Replication (N copies of accounts, containers, objects)


Object Storage Architecture

27


5 Zones2 Proxies per 25Storage Nodes

10 GigE to Proxies1 GigE to

Storage Nodes24 x 2TB Drives

per Storage Node

Public Internet

Example OpenStack Object Storage Hardware

Load Balancers (SW)

Example only – many configurations possible28Friday, January 21, 2011

Object Storage Components

29

•The Ring: Mapping of names to entities (accounts,containers, objects) on disk.

•Stores data based on zones, devices, partitions, and replicas

•Weights can be used to balance the distribution of partitions

•Used by the Proxy Server for many background processes

•Proxy Server: Request routing, exposes the public API

•Replication: Keep the system consistent, handle failures

•Updaters: Process failed or queued updates

•Auditors: Verify integrity of objects, containers, and accounts


Object Storage Components (cont)

30

•Account Server: Handles listing of containers, stores as SQLite DB

•Container Server: Handles listing of objects, stores as SQLite DB

•Object Server: Blob storage server, metadata kept in xattrs, data in binary format

•Recommended to run on XFS

•Object location based on hash of name & timestamp


Object Storage Software Dependencies

31

Object Storage should work on most Linux platforms with the following software (main build target for Austin release is Ubuntu 10.04):

•Python 2.6•rsync 3.0

And the following python libraries:•Eventlet 0.9.8•WebOb 0.9.8•Setuptools•Simplejson•Xattr•Nose•Sphinx


Building a Cloud

32

OpenStack Powered


OpenStack Isn’t Everything

33


Technical Prerequisites

34

CLOUD HAS A MINIMUM PRACTICAL SCALE

DATACENTER MADE “CLOUD READY”

EQUIPMENT BUILT FOR CLOUD

•Proof of Concept: 5+ Servers•Pilot: 20+ Servers•Production: 40+ Servers

•Networking•Power

•CPUs with virtualization and power management support•Storage platforms with flexible workload capabilities


Cloud Ready Datacenter Requirements

35

HOMOGEOUS CONFIGURATION

INCREASED POWER DENSITY

FAT TREE / MESH NETWORKS

“LIGHTS OUT” OPERATION


Deploying the Cloud

36


OpenStack GUI Options

37

• Cappuccino Web Panel

• iOS Control Panel

• Android

• Django Based Control Panel

•SimCloud - Sim City Like GUI

• Etc..


OpenStack Development Process

38


OpenStack Release Process: Four Phases

39

Design* Development QA Release

*Design phase and Design Summit occur every other release, 2x per year


OpenStack Releases

40

Cactus:April 2011

Bexar: February 2011

Austin:October 2010

• OpenStack Object Storage production-ready• OpenStack Compute developer preview, ready for testing and proofs of concept

• OpenStack Compute ready for enterprise private cloud deployments and mid-size service provider deployments• Enhanced documentation• Easier to install and deploy

•OpenStack Compute ready for large service provider scale deployments

We are here!


Openstack Compute: Austin Release

41

•Multi-hypervisor support: KVM, QEMU, User-Mode Linux, Xen and XenServer

•Introduces official OpenStack API, while maintaining EC2 API option

•New image registry and delivery service, called the Glance project

•Support for two network models on compute nodes: VLANs with DHCP and flat

with either static IP pools or DHCP

•Addition of base scheduling service

•Implements WSGI to create a standard API layer with reusable components

•Support for user-friendly naming

•Refactored ORM and networking code for simpler code that is easier to understand

•Addition of SQLAlchemy Database toolkit so users can leverage existing SQL

infrastructure


Openstack Object Storage: Austin Release

42

•Addition of a stats system that produces per-account hourly summaries of system

usage

•Ability for users to set ACL’s and grant public access to containers

•Support for API access to account and container metadata

•Rate limiting was extended to allow requests to be slowed down and support stair

stepped rate limits based on container size

•WSGI support was improved and pulled into middleware


What’s next for OpenStack?

43

Future releases and features in the next versions of OpenStack


Openstack Compute: Bexar Release

44

•IPv6/IPv4 Dual Stack Support

•Sheepdog Support

•I18N Japanese messages support

•Microsoft Hyper-v support

•raw-disk images

•Web-based serial console

See all the blueprints at https://blueprints.launchpad.net/nova/bexar


https://blueprints.launchpad.net/nova/bexar

https://blueprints.launchpad.net/nova/bexar

Beyond Bexar

45

•Block Storage by Sheepdog

•Plugin Network Architecture

•Virtualized Networking with Midonet


Sheepdog

46

Distributed block storage for KVM

http://www.osrg.net/sheepdog/






Current Networking Model

47


Network as a First Class Service

48


Virtualized Networking Architecture

49


Midonet - flexible virtual networks

50

Fully Distributed Virtual Networking

• L2/L3 virtual distributed switches

• Distributed virtual load balancing

• Distributed virtual firewalls

• Scale out

• and more!


Midonet - flexible virtual networks

51


Questions?

52

Developing, Using, Future Trends, ...


Sign up for the Beta

53

If you want to help us build a better

cloud, you can sign up for our beta and

try our upcoming products.

http://midokura.com/beta.html




Want to know more?

54

• http://openstack.org

• Japan OpenStack User Group

• Contact the Midokura Team

•[email protected]

• OpenStack Community


mailto:[email protected]

mailto:[email protected]

55

Life at Midokura


But wait there’s more!

56

• Free OpenStack t-shirts!

• Midokura is hiring!

• More Questions?

• Meet the Midokura team afterwards


Thank You!

57

Join the OpenStack revolution!

Special thanks to others who generously provided presentation content:http://www.theregister.co.uk/2011/01/08/openstack/http://www.slideshare.net/bpiatt/openstack-tutorialhttp://www.slideshare.net/annegentle/openstack-overview-for-austin-cloud-user-grouphttp://adrianotto.com/2010/09/openstack-os-is-great-for/


http://www.theregister.co.uk/2011/01/08/openstack/

http://www.theregister.co.uk/2011/01/08/openstack/

http://www.slideshare.net/bpiatt/openstack-tutorial

http://www.slideshare.net/bpiatt/openstack-tutorial

http://www.slideshare.net/annegentle/openstack-overview-for-austin-cloud-user-group

http://www.slideshare.net/annegentle/openstack-overview-for-austin-cloud-user-group

Technology

01.19.2011 AIIT InfoTalk on OpenStack