[email protected]

Openstack@eBayPractical SDN Deployment using Quantum

Not a public cloud, but …

Copyright eBay Inc. 2012 2

QADEV

Prod

PCI

QA DEV

DEVQA

Secure

Prod


Principles Any Application Anywhere

Dedicated physical environments cause fragmentation Soft Cabling

Datacenter reconfiguration is costly and cannot be automated Shared Standardized Infrastructure

Simplifies automation and improves supply chain efficiency Virtualize everything

White space between applications and infrastructure helps agility Automate everything

Automation helps agility and efficiency


Class of Service

• Translation of physical environment properties into configurations• Assigned to projects (logical environments), drives scheduling and policies

• For example, network selection

Obligations Restrictions Capabilities

QA Approved Builds No Login Access Core DB access

Prod OS version No Corp Access 24/7 Incident Mgt

Monitoring No QA Access Site traffic Access

Production

Obligations Restrictions Capabilities

Certified OS versions Limited Prod Access

Full root

Limited QA Access

No site Traffic Filtered Internet

DEV

Obligations Restrictions CapabilitiesNo Prod Access Private DB

Certified OS Versions No Corp Access 24/7 Incident Mgt

Monitoring No QA Access Site traffic Access

External


Infrastructure designed for scale

Spine

Leaves

Core

M servers2x1Gb

N leaves

4 spines

Flat L3 (all switches are routers too)Line rate from any server to any server (oversubscription = 48/40)OSPF/ECMP to advertise routes

(48x1Gb)

(Nx10Gb)

48 -> N “½ racks”


Isolation options: L2

Production QA

VLAN trunk

vlan 1

vlan nQA

Prod

Dedicated Network VLAN Based

- physical network build out- Fragmentation- coarse grained isolation

- Limited scale (n = 4096) - Large fault domain (STP)

+ Physical isolation+ fool proof

+ L2 isolation+ somewhat soft Cabling


Isolation options: L3 with Security Groups

Security Groups or Virtual Firewall

- Difficult to combine provider policies and user policies- Management of rules- Impact of group membership modification- Aggregation/summarization difficult/impossible

+ no/minimal infrastructure requirement+ good for user policies (ip tables)


Isolation Options: Virtual L2 Networks

Cloud Fabric

QA

ProdOther

Networks

+ L2 isolation+ compatible with large scale networks+ can be fully automated+ firewall can be interposed betweenvirtual networks

Virtual Networks using Software Defined Networks

+ Can complement L3 isolation+ large number of networks (n>4096)- Tunnel overhead- L2 size limited by # of tunnels and their mgt

Overlay 1

Overlay n


ControllerThe Switch/Router

All you need to know about SDN

The Network

Routing/switching engine

Logic

controls

Netw

ork protocols

Traditional SDN

The Network

The Switch/Router

Routing/switching engine

Logic

controls

Netw

ork protocols

API


SDN ‘levels’

Nerdy

Wizard

Ninja

Virtual SwitchesOverlay Networks

Physical SwitchesTraffic Engineering

ARP + L2 protocols

OSPF/ECMP,…

Virtual + Physical switchesOverlay Networks


Dev Cos : aka Dev Cloud A logical environment defined as a class of service on top of shared infrastructure

Self Service VM for developers. Access must be similar to their desktops (access to QA, Corp, …) Should allow collaboration

Implemented as a set of L2 networks (/24) with in a given L3 (/20) No private networks : all developers on same shared networks No private IP space: traffic is routed within core, no need for floating Ips

Isolated from infrastructure Overlay network using OpenVswitch / STT tunneling Nicira NVP controllers integrated with Quantum (Essex) Routed out through perimeter firewall


Niciracontrollers

NiciraServiceNodes

N

vswitch

gtw-xxxxM

Active Gateway

vswitch

HypervisorC

Eth1/vlan 1

vif

Eth0/vlan 2

S Q

Eth1/vlan 1

Eth0/vlan 2

N

vswitch

gtw-xxxx

gtw-xxxxM

Standby Gateway

10.9.0.110.9.0.1010.9.1.0/24

10.9.2.0/24

10.9.1.1

10.9.2.1

NiciraServiceNodes

Niciracontrollers

Corp

Internet

QA

Dev Cloud : 10.9.0.0/20

trunk

10.9.0.0/20 ->10.9.0.10From 10.9.1.0/24 default->10.9.0.1From 10.9.2.0/24 default->10.9.0.1

default->10.9.2.1

K AN:Nova-network+dnsmasqC:Nova-computeS:Nova-schedulerM:Metadata

K:Ubuntu + KVMA:Nova-apiQ:Quantum

Infrastructure/Internal

Infrastructure/External

Virtual network

Copyright eBay Inc. 2012

13

eBay IaaS

eBay Cloud Portal

Create instance (COS,OS, size)

DNSManagement

Create DNS (A,PTR)

Nova API

Boot Instance(Image ID,Flavor, NIC)

Get Free Networks

1

2

34

Nova Network

Nova-manage

Create network(project = admin,Cidr=10.9.x.0/24)

novadb

Quantum

Nicira Controller

Create lswitch

Create port

Nova Compute

Nova Scheduler

Create routes

Gateway

Create gtw-xxxx

AdminDeveloper

Get IPCreate port


0

50

100

150

200

250

0102030405060708090100

SuccessFailedrate

Instance Requests

15

What works/What doesn’t

Good Perimeter firewalls configured once, not

dependent on the instance creation/deletion/movement

Network are pre-created using nova-manage, good for provider networks

Can be extended with other COS using same pattern

Stability of both Nicira NVP and Openstack + Ubuntu + KVM

Looking forward to new features in Folsom – Quantum v2

Bad No capacity/policy based assignment of

networks – had to be implemented outside. Moving it to nova scheduler.

One network flavor supported in Essex. Cannot have, e.g., one gateway per network, with different behavior (dhcp)

Scale out requires bigger links out of the gateway, or more gateways

Upset the separation of concern requirement: Netsec + Networking + Sys Admins in same box = ‘interesting’

16

What’s Next New classes of service

External : private networks + VIP and Floating IP on the Internet Production : Bridged network

Scale out 80 today, going to a lot more More gateways/10Gb

Folsom upgrade L3 Routers Load Balancers

Cleaner Openstack integration Network Allocation DNS configuration AuthN/AuthZ


We are Hiring !

http://www.ebaycareers.com/

Documents

[email protected]