OpenStack  Private  Cloud  at  eBay  

1  

OpenStack  Design  Summit  –  Nov  2013  

Marc Carges - CTO

John Donahoe- CEO

Devin Wenig – President eBay NA

Bob Swan - CFO

Analyst Conference 2013

QA DEV

Prod

PCI

QA DEV

DEV QA

Secure

Prod

4

5/1/2013 6/1/2013 7/1/2013 8/1/2013 9/1/2013 10/1/2013

Users Projects

5/1/13 6/1/13 7/1/13 8/1/13 9/1/13 10/1/13

VMs Created VMs Deleted

5/1/2013 6/1/2013 7/1/2013 8/1/2013 9/1/2013 10/1/2013

Cores

Frictionless on-demand access

Multi-tenant, isolated

One cloud on shared infra

billmelater.com

brands4friends.de

corp.ebay.com

ebay-kleinanzeigen.de

ebay.ca

ebay.com

ebay.de

gumtree.com

hunch.com

marketplats.ne

milo.com

5  

Globally Shared infra (Compute, Network, Storage)

VPC VPC

project project … project project …

Availability Zone AZ

VPC

project …

Availability Zone

VPC

project …

VPC

project …

Region Region

6  

Virtual Networks Block, Object Storage Flavors classes

Large scale networks Multi tiers storage Commodity H/W

VPC

Class of Service (PCI, DEV, PROD, …)

OpenStack APIs

Project -> VPC Network -> VPC Dashboard VPC selection

Capacity based Network selection

Managed images

Win/Linux

Images -> VPC

Dashboard AZ selection

7  

OpenStack  is  not  cloud  

hDp://www.subbu.org/blog/2013/07/openstack-­‐is-­‐not-­‐cloud  

OpenStack  APIs  

Network  design  

Net/info  sec  

Builds  

CI  

Infra  onboarding  

Config  management  

High  availability  

Log  processing  

Metrics  

Cloud  monitoring  

AlerRng  

Incident  resoluRon   User  

experience  

Customer  support  

SLA  

Upgrades  

Capacity  planning  

Scale  out  

RemediaRon  

Autoscaling  

Metering  and  

chargeback  Monitoring  and  alerRng  

Cloud  is  a  service!  

9  

Monitoring  

10  

nova  boot        -­‐-­‐image  a4a3d552-­‐4f92-­‐4426-­‐82d9-­‐ff2226d72d08        -­‐-­‐flavor  medium  myvm  

Is  the  cloud  broken?  

User  

You  ?  

11  

RabbitMQ   Capacity  issues  

Glance  down  

Didn’t  get  DHCP  in  Ame  for  cloud  init  

Metadata  Aming  out  

Quantum  plugin  failures  

vm_state=ACTIVE  does  not  really  mean  success  

12  

Log  source   logstash   ØMQ  

ØMQ  

ØMQ  

ØMQ  

ElasRcSearch  

ElasRcSearch  

ElasRcSearch  

Log  source   logstash   ØMQ  

Log  source   logstash   ØMQ  

Log  source   logstash   ØMQ  

1   Grok  paDerns  3   A  cluster  that  can  

scale  horizontally  

kibana  

logstash  

logstash  

logstash  

2   Low  latency  

13  

StatsD  

Log  source   logstash   ØMQ  

ØMQ  

ØMQ  

ØMQ  

ElasRcSearch  

ElasRcSearch  

ElasRcSearch  

Log  source   logstash   ØMQ  

Log  source   logstash   ØMQ  

Log  source   logstash   ØMQ  

kibana  

logstash  

logstash  

logstash  

Graphite  

Zabbix  

Metrics   Monitoring   Alerts  

14  

StatsD  

Log  source   logstash   ØMQ  

ØMQ  

ØMQ  

ØMQ  

ElasRcSearch  

ElasRcSearch  

ElasRcSearch  

Log  source   logstash   ØMQ  

Log  source   logstash   ØMQ  

Log  source   logstash   ØMQ  

kibana  

logstash  

logstash  

logstash  

Graphite  

Zabbix  OpenStack  nodes,  processes,  network,  

storage,  …  

Is  the  cloud  broken?   May  be?  

15  

StatsD  

Log  source   logstash   ØMQ  

ØMQ  

ØMQ  

ØMQ  

ElasRcSearch  

ElasRcSearch  

ElasRcSearch  

Log  source   logstash   ØMQ  

Log  source   logstash   ØMQ  

Log  source   logstash   ØMQ  

kibana  

logstash  

logstash  

logstash  

Graphite  

Zabbix  OpenStack  nodes,  processes,  network,  

storage,  …  

StackMetrics  

StackWatch  

Simulate  KPIs  

•  Create,  delete  •  Volumes  •  Ping  success  •  Ping  latency  •  …  

hDp://www.subbu.org/blog/2013/10/monitoring-­‐and-­‐alerRng-­‐for-­‐openstack  

16  

Scale  out  

17  

Capacity  

Availability  

One  Cloud  User  Experience   Availability  for  Business  

1.  Nova  “availability  zones”  

2.  Keystone  “regions”  

3.  Nova  “cells”  

18  

Region  A  Availability  

Zone  

Availability  Zone  

Availability  Zone  

Region  B  Availability  

Zone  

Availability  Zone  

AWS  

nova API

cinder API

neutron API

swift API

Block storage backend

network driver + agents + infra

Hypervisors

storage

keystone

glance Availability  Zone  

19  

Dashboard Keystone

Nova

Glance

Quantum Cinder Swift

region services

az services

infra

Region BDashboard Keystone

Nova

Glance

Quantum Cinder Swift

region services

az services

infra

Region C

GlanceDashboard Keystone

Nova Quantum Cinder Swift

region services

az services

infra

Region A

Dashboard behind a global load balancer VIP

user chosen replication of selected resources

Region  ≈  A  fully  decoupled  OpenStack  deployment  

Availability  Zone    ≈  Keystone  “region”  

Nova  availability  zone?  Nova  cells  –  an  implementaRon  detail  

20  

Build  and  Deployment  

21  

Ubuntu,  apt,  fabric  

Ubuntu,  apt,  puppet  and  foreman  

1.  Bare-­‐metal  provisioning  for  hypervisors,  controllers,  NVP,  …  

2.  Virt  env  tarballs,  puppet  and  foreman  3.  A  full  working  deployment!  

CaDle  test  clouds  Patches    ParRal  upgrades  Rollbacks  

Pet  test  clouds  

1.  Change  orchestraRon  2.  Foreman  UI  “ClickOps”  

22  

• Puppet  automaRon  • Full  topology  with  virtual  environments  • ProducRon  ready  

• Zabbix  templates    • StackWatch  and  StackMetrics  • Contribute  on  VPCs  • Contribute  on  scale-­‐out  paDerns  

Our  Commitment  to  the  Community  

JC  MarRn    

@jchmarRn  

23  

Subbu  Allamaraju    @sallamar