At Rakuten, the Rakuten OpenStack Platform and Beyond Oct/25/2014 Kentaro Sasaki, Koichi Yoshigoe Global Operations Department, Rakuten, Inc.

2

Agenda • OpenStack • Design • Customize • Impressions • Roadmap • OSS Activity

http://goo.gl/3Qitfy

3

Presenter Kentaro Sasaki Storage Architect since 2010 Research fields:

Condensed Matter Physics Skills and Experience:

Python, C, Golang, OpenStack, NetApp, ZFS, EMC, Object Storage

4

Presenter Koichi Yoshigoe Cloud Architect since 2012 Research fields:

High Energy Astrophysics Skills and Experience:

Python, C, Fortran, OpenStack, VMware, Xen, Open Compute

5

100% OSS cloud management software: OpenStack project began in 2010. NASA Nebula -> Nova Rackspace Cloud Files -> Swift

10 core components in Icehouse(2014.1)

OpenStack

6

OpenStack Project

19303"people

144"countries

Community

7

REST API

Abstract Infrastructure

Generalize / OSS

Why we choose OpenStack?

API

OSS

Internal

Agility

Extensibility

Tech Driven

8

DesignComplicated, not sure yet, no best practice

Introduce current system architecture

9

Middleware for OpenStack OpenStack Components:

Keystone (Icehouse) Nova (Icehouse) Cinder (Icehouse) Glance (Icehouse) Neutron (Icehouse) Horizon (Icehouse) http://goo.gl/FxLCYi

10

All nodes are on VMware. 1DB / 1server

Auth = LDAP Glance = ZFS LB = A10 Min = 20servers

System Design Overview

keystone API nodes

nova/cinder/neutron

horizon

API nodes glance

ZFS

A10 LB

nova compute node

nova misc. nodes scheduler, etc

cinder volume

A10 LB A10 LB

A10 LB

LDAP

region1

VMware RabbitMQ

neutron-vcenter

Mamcached

each components MariaDB

keystone MariaDB

cinder misc. nodes scheduler

Complicated !!

11

Pros: High SLA A few compute

Cons: Complex HW Far from defect High Cost!!

Design Key Factor

vSphere 5.5

3PAR, Pure Storage

Ubuntu 14.04 LTS

A10 Load Balancer

OpenStack RabbitMQ MariaDB 10 Splunk

12

Region and Availability Zone

AZ: development IOPS: 500

AZ: staging IOPS: 500

region1 region2

region3

keystone

AZ: production IOPS: 500, 4000

AZ: sandbox Disposable environment

IOPS: 100

horizon

common

Flavor Spec Catalog: standard or high (defined by IOPS) Region limitation: 5000VMs / 1Region

13

Pros: Much less power

Cons: High overcommit rate

IaaS in numbers AZ VMs HVs

Sandbox 438 2

Development 2039 44

Staging 3000 93

Production 6909 371

CentOS: 80%, Ubuntu(used in paas): 20%

Tenant: 600, User = 2000

14

Redundancy Concept Overview Concept Overview:

1Process / 1server except for API nodes Use Load Balancer for API nodes

Use VMware HA for active / standby

15

Redundancy Concept Scale out with Load Balancer:

nova-api cinder-api neutron-server glance-api/registry keystone horizon

API node

Load Balancer

API node API node

16

Scale out with servers: nova-conductor nova-scheduler cinder-scheduler

Redundancy Concept

nova-conductor nova-conductor nova-conductor

nova-scheduler nova-scheduler nova-scheduler

cinder-conductor cinder-conductor cinder-conductor

1process / 1server, No Load Balancer necessary

17

Redundancy stack of API nodes

Active / Active: A10 Load Balancer Active / Standby: VMware HA

VMware HA

keystone nova cinder glance neutron horizon

A10 Load Balancer

Active / Standby

Active / Active

18

Deployment Automation flow Local PC

CI environment

Chef server

Chef cookbook

git push

hook

deploy

OpenStack nodes

MariaDB RabbitMQ

http://goo.gl/MGIUwA

http://goo.gl/hRogWM

SERVERSPEC http://goo.gl/PPEj46

http://goo.gl/WIFg8A

Chef cookbook

19

CustomizeBasically community based, but… still have

difficulty in default, develop by ourselves

20

Customizing Customizing is scared…, but required!!:

Difficult to upgrade / apply patch / maintain Rakuten specific configuration / General Bugs

keystone ldap driver

neutron-vcenter-agent

nova vmware driver

neutron ml2 vds driver

glance_store driver

cinder driver

21

neutron-vcenter-agent Official neutron vDS: Single port group

Rakuten neutron vDS: Multiple port group

ESXi ESXi ESXi

vSphere Distributed Switch

port group

ESXi ESXi ESXi

vSphere Distributed Switch

port group port group port group

22

Config Drive and cloud-init start creation

boot instance

mount config drive

extend root LVM

set static IP

Register DNS record

Use per-instance module in cloud-init

Mount /dev/disk/by-label/config-2 Call in-house DNS API

finish creation

23

ImpressionsBasically community based, but… still have

difficulty in default, develop by ourselves

24

What a difficult…!!!

Complicated: • So many components

No best practice: • Different from one by one

25

What a difficult…!!!

Bugs & Bugs: • Read source code!!

Big changing on culture: • Most difficult thing is mind-shift

26

What a fun…!!! Full Stack Skills:

• Read / Write source codes • Broader technology range

Be a pioneer: • Eternal Beta • Cutting Edge

Let’s Begin!!

27

What a fun…!!! Contribution to World Wide Project:

• Everyone will be happy!! Community activities:

• Knowledge sharing • Regular workshop Let’s Join!!

28

Roadmap

29

Roadmap #1 LBaaS / DNSaaS: Introduce “Official” XaaS project

Current

In-house LBaaS Neutron LBaaS

Designate In-house DNSaaS

Future

30

Expand to the other Regions: [soon] 4th Region

Roadmap #2

Introduce new Network / Storage / Compute technology

Current

region1: sdbx

region2: dev/stg

Future

region4: prod2

region3: prod1

31

Roadmap #3 Online Migration of existing VMs:

Migration method had been established!!

Run only DB INSERT at nova boot

UPDATE nova’s DB to be active

SELECT VM’s instance_id

from vCenter

32

Hybrid Cloud: Abstract Private / Public with paas and Terraform

Roadmap #4

Private Cloud

OpenStack Microsoft Azure Amazon Web Service Public Cloud

Cloud Foundry v2 / Terraform

33

Auto Scaling: • Validate Heat component • Hook by monitoring • Scheduled-based • Event-based

Roadmap #5

VM

VM

VM

VM VM

VM VM

VM VM

34

Challenges at OpenStack activity KVM / Multi-Hypervisor Other Components OpenStack x LeoFS

KVM VMware Region / AZ

OpenStack

http://goo.gl/VUqPnb http://goo.gl/Ot8Ef4

35

OSS Activity

36

Open Source Strategy at Rakuten OPS

http://goo.gl/a3S7Sp

“Take advantage of global developers and open source” • Follow open source mainstream • Don’t fork unless required • Build plugin when needed • Contribute back

37

Upstream training Training for contributing newbie:

• Faster integration • Contribute real world patch • Master the technical tools • Understand workflow

Educating Contributors Makigaki @Atlanta, 2014 Summer

38

Contribution at Juno (2014.2) Release 24061

20503

16851

12563

8749

3900 3474 2718 5

Contribution by companies

Mirantis

HP

Red Hat

Rackspace

IBM

Cisco

Independent

VMware

Rakuten

http://goo.gl/N3c5pT (at Sep 4 2014)

Rakuten started commitment from last quarter.

39

GitHub Our source code in GitHub!!:

• Our Chef recipe • Bootstrap script • GPLv3 • By the winter of 2014

http://goo.gl/DRxKxT

40

Wrap Up We started OpenStack!:

• Contributed back • OSS activity is a lot of fun! • Our knowledge will be shared in public

41

Thank you