36
Xen*, SDN and Apache Cloudstack Sebastien Goasguen, Apache CloudStack Citrix EMEA August 28 th 2012 Xen Summit

Xen*, SDN, and Apache CloudStack

Embed Size (px)

DESCRIPTION

Apache CloudStack has already proven its abilities to scale to tens of thousands of physical XenServer nodes, but a number of traditional networking schemes don't work. VLANs max out at 4096, Layer3 isolation (aka Security Groups) helps scale to a degree, but you lose a lot of flexibility in the process. XCP and XS6.0 have the ability to make use of openvswitch to build software defined networks for increasing flexible networks with little or none of the legacy restrictions.

Citation preview

Page 1: Xen*, SDN, and Apache CloudStack

Xen*, SDN and Apache Cloudstack

Sebastien Goasguen, Apache CloudStack Citrix EMEA

August 28th 2012 Xen Summit

Page 3: Xen*, SDN, and Apache CloudStack

Apache CloudStack

• IaaS solution to build a private/public cloud

• Hypervisor agnostic:

– Xen, XS, XCP

– KVM

– VMware

• Object store support

– Swift

– Upcoming support from Caringo

• EC2/S3 compatibility

• Java application

• Ant build but moving to maven (via new contributor)

• First Apache release 4.0

coming Sept 26th

Page 4: Xen*, SDN, and Apache CloudStack

Participating in CloudStack

• Apache incubator project • http://www.cloudstack.org • #cloudstack on irc.freenode.net • @CloudStack on Twitter • http://cloudstack.org/discuss/mailing-lists.html

Welcoming contributions and feedback, Join the fun !

Page 5: Xen*, SDN, and Apache CloudStack

A Very Flexible IaaS Platform

Compute

XenServer VMware KVM Oracle VM Bare metal

Hypervisor

Storage

Local Disk iSCSI NFS Fiber

Channel Swift

Block & Object

Network

Network Type

Isolation Load

balancer Firewall VPN

Network & Network Services

Primary Storage Secondary Storage

http://www.slideshare.net/cloudstack/cloudstack-architecture

Page 6: Xen*, SDN, and Apache CloudStack

NaaS ?

• “Cloud Servers”

– On-demand, Elastic, Measured server provisioning

• Cloud Storage

– Scalable/fault tolerant storage with object stores

• Cloud Networks

– How to do on-demand, elastic, measured networking provisioning ?

– How to program the network ?

Page 7: Xen*, SDN, and Apache CloudStack

A very extensive API • CloudStack orchestrates

your network:

– Provisioning

– Configuration

– Updates

• For multi-tenants isolation

• Using hardware and software devices

• At scale: O(10^4) Hyp, O(10^5) VMs…

Page 8: Xen*, SDN, and Apache CloudStack

Software Defined Networking

• Enable innovation, experimentation, optimization and customization of networks

• Move control of the network to software. i.e Programmable network

• Virtualize the network

• Vendor-agnostic, standard protocol for control: OpenFlow

Page 9: Xen*, SDN, and Apache CloudStack

OpenFlow

• Leading SDN protocol

• Decouples control and data plane by giving a controller the ability to install flow rules on switches.

• Hardware or software switches can use OpenFlow

• Spec driven by ONF

• Google achieved 95% utilization of WAN backbone by using SDN

Page 10: Xen*, SDN, and Apache CloudStack

OpenFlow

• OpenFlow rules can drop, rewrite, forward packets

10 September 2012 10

Switch Port

MAC src

MAC dst

Eth type

VLAN ID

IP Src

IP Dst

IP Prot

TCP sport

TCP dport

Rule Action Stats

1. Forward packet to port(s)

2. Encapsulate and forward to controller

3. Drop packet

4. Send to normal processing pipeline

Packet + byte counters

Diagram Src: http://www.openflow.org/wp/documents/

Page 11: Xen*, SDN, and Apache CloudStack

OF Controllers

• Several controllers out there (NOX, POX,Trema…)

• Floodlight from Big Switch. Apache license

Page 12: Xen*, SDN, and Apache CloudStack

OpenVSwitch

• “Open vSwitch is a production quality, multilayer virtual switch licensed under the open source Apache 2.0 license. It is designed to enable the massive network automation through programmatic extension…”

Page 13: Xen*, SDN, and Apache CloudStack

OpenVSwitch • Default bridge in XenServer

and XCP

• Supported in Xen but not integrated in toolstack

• Enables:

– VLAN tagging

– Rate limiting

– GRE tunnels

– OpenFlow controller

– …

• High Performance ( http://networkheresy.com/category/o

pen-vswitch )

Page 14: Xen*, SDN, and Apache CloudStack

e.g OVS rate limiting

• Can enforce QoS with rate limiting controls • ovs-vsctl set Interface tap0 ingress_policing_rate=1000

• ovs-vsctl set Interface tap0 ingress_policing_burst=100

Page 15: Xen*, SDN, and Apache CloudStack

e.g OVS VLAN tagging

• ovs-vsctl add-br br0

• ovs-vsctl add-port br0 eth0

• ovs-vsctl add-port br0 tap0 tag=1

• ovs-vsctl add-port br0 tap1 tag=2

• Complement on host2…

Page 16: Xen*, SDN, and Apache CloudStack

e.g OVS and GRE tunnels

• No Cookbook on OVS page • ovs-vsctl add-port br1 gre1 -- set interface gre1 type=gre

options:remote_ip=192.168.1.152

Page 17: Xen*, SDN, and Apache CloudStack

OVS and Openflow

• Point OVS switches to an OF controller: $ovs_vsctl set-controller br0 tcp 192.168.1.33:6633

• Install rules on switch

– Proactively (before any packet flows)

– Reactively (unknown packets forwarded to controller, who pushes flow mod on switch, then operates at line rate)

• Can do SDN with OpenFLOW but also with straight up OVS and managing mappings/rules in CloudStack db.

Page 18: Xen*, SDN, and Apache CloudStack

OpenNebula

• Supports VLAN tagging and rate limiting through “hooks” that call ovs_vsctl

• Scripts executed on an hypervisor before a VM is launched

• Potentially also executed after VM shutdown for cleanup

• Also supports OpenFlow

Page 20: Xen*, SDN, and Apache CloudStack

API key to customization of the network

• You dream it, CloudStack orchestrates it

Page 21: Xen*, SDN, and Apache CloudStack

Terminology Zone: Availability zone, aka Regions. Could be worldwide. Different data centers Pods: Racks or aisles in a data center Clusters: Group of machines with a common type of Hypervisor Host: A Single server Primary Storage: Shared storage across a cluster Secondary Storage: Shared storage in a single Zone

Page 22: Xen*, SDN, and Apache CloudStack

Router

L3 Core Switch

Access Layer

Switches

… … … … …

Availability Zone

Servers

CloudStack Mgmt Server Cluster

Secondary Storage

Pod 1 Pod 2 Pod 3 Pod N

MySQL

Load Balancer

Operations Admin and Cloud API

Users

Physical Network

Slide from Chiradeep Vittal, http://www.slideshare.net/cloudstack/cloudstack-networking

Page 23: Xen*, SDN, and Apache CloudStack

Layer-2 Guest Virtual Network 1 VLAN per guest network

Public Network/Internet

Guest Virtual Network 10.1.1.1/8 VLAN 100

Gateway address 10.1.1.1

DHCP, DNS NAT Load Balancing VPN

Public IP 65.37.141.11

10.1.1.1

Guest VM 1

10.1.1.3

Guest VM 2

10.1.1.4

Guest VM 3

10.1.1.5

Guest VM 4

CS Virtual Router

Public Network/Internet

Guest Virtual Network 10.1.1.1/8 VLAN 100

Private IP 10.1.1.112

DHCP, DNS

Public IP 65.37.141.112

10.1.1.1

Guest VM 1

10.1.1.3

Guest VM 2

10.1.1.4

Guest VM 3

10.1.1.5

Guest VM 4

NetScaler Load

Blancer

Private IP 10.1.1.111

Public IP 65.37.141.111

Juniper SRX

Firewall

CS Virtual Router provides Network Services External Devices provide Network Services Network Hardware exposing API can be controlled

CS Virtual Router

Slide from Chiradeep Vittal, http://www.slideshare.net/cloudstack/cloudstack-networking

Page 24: Xen*, SDN, and Apache CloudStack

Opportunity for Xen

• Opportunity to create highly specialized networking services appliances using

– OpenMirage VMs

– HalVM

• See talks in Monday’s session

Page 25: Xen*, SDN, and Apache CloudStack

Networking challenges in a private Cloud

• Multi-tenants on hypervisors => isolation between guest networks

• VLANs in the datacenter is hard and limit at 4096 VLANs.

• Hardware switches may not do it very well or have a lower limit

Page 26: Xen*, SDN, and Apache CloudStack

Networking trend • Move to software switches

• Move to L3 isolation

• Use tunnels between OVS (GRE tech preview)

• Program the network through API

• Encapsulation virtualizes the network, between overlays on overlays on overlays..

• L3 on L2 on GRE on L3 on L2…

• Then you bring the WAN and you have:

• L3 on L2 on GRE on L3 on L2 on GRE on L3 on L2 ….Euhhhh !!!

Page 27: Xen*, SDN, and Apache CloudStack

Back of the enveloppe

• ~10,000 hypervisors in your data center

• ~100,000 VMs

– x10 or x100 if you use HalVM or Openmirage.org

• (10,000*9,999)/2 tunnels for a full mesh

– 50x10^6 tunnels to keep track of ?

Page 28: Xen*, SDN, and Apache CloudStack

DB Security Group

Web Security Group

Layer 3 cloud networking

… …

Web VM

Web VM

Web VM

Web VM

DB VM

Web VM

DB VM

Web VM

Ingress Rule: Allow VMs in Web Security Group access to VMs in DB Security Group on Port 3306

Slide from Chiradeep Vittal

Page 29: Xen*, SDN, and Apache CloudStack

L3 isolation with distributed firewalls Tenant 1 VM 1

10.1.0.2

Tenant 2 VM 1

10.1.0.3

Tenant 1 VM 2

10.1.0.4

Public Internet

10.1.0.1

Public IP address 65.37.141.11 65.37.141.24 65.37.141.36 65.37.141.80

Load Balancer

L3 Core

Pod 1 L2 Switch

Pod 3 L2 Switch

10.1.16.1

… 10.1.8.1

Pod 2 L2 Switch

Slide from Chiradeep Vittal

Page 30: Xen*, SDN, and Apache CloudStack

L3 isolation with distributed firewalls Tenant 1 VM 1

10.1.0.2

Tenant 2 VM 1

10.1.0.3

Tenant 1 VM 2

10.1.0.4

Tenant 1 VM 3

10.1.16.47

Tenant 1 VM 4

10.1.16.85

Public Internet

10.1.0.1

Public IP address 65.37.141.11 65.37.141.24 65.37.141.36 65.37.141.80

Load Balancer

L3 Core

Pod 1 L2 Switch

Pod 3 L2 Switch

10.1.16.1

… 10.1.8.1

Pod 2 L2 Switch

Slide from Chiradeep Vittal

Page 31: Xen*, SDN, and Apache CloudStack

L3 isolation with distributed firewalls Tenant 1 VM 1

10.1.0.2

Tenant 2 VM 1

10.1.0.3

Tenant 1 VM 2

10.1.0.4

Tenant 2 VM 2

10.1.16.12

Tenant 2 VM 3 10.1.16.21

Tenant 1 VM 3

10.1.16.47

Tenant 1 VM 4

10.1.16.85

Public Internet

10.1.0.1

Public IP address 65.37.141.11 65.37.141.24 65.37.141.36 65.37.141.80

Load Balancer

L3 Core

Pod 1 L2 Switch

Pod 3 L2 Switch

10.1.16.1

… 10.1.8.1

Pod 2 L2 Switch

Slide from Chiradeep Vittal

Page 32: Xen*, SDN, and Apache CloudStack

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

… VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

VM

VM

VM …

A Million Firewalls?

Slide from Chiradeep Vittal

Page 33: Xen*, SDN, and Apache CloudStack

Problem: Manage the state of 100s of thousands of firewalls

Solution: Well-known software scaling techniques • Message queues • Consistency tradeoffs • Idempotent configuration & retries

CloudStack uses • special purpose queues • optimized for large security groups • eventual consistency for rule updates

Slide from Chiradeep Vittal

Page 34: Xen*, SDN, and Apache CloudStack

Problem: Firewall (iptables) rules explosion on the host firewall

ipset –N web_sg iptreemap ipset –A web_sg 10.1.16.31 ipset –A web_sg 10.1.16.112 ipset –A web_sg 10.1.189.5 ipset –A web_sg 10.21.9.77 -A FORWARD –p tcp –m tcp –dport 3060 –m set –match-set web_sg src -j ACCEPT

Solution: Use ipsets:

Slide from Chiradeep Vittal

Page 35: Xen*, SDN, and Apache CloudStack

Conclusions

• Programmable networking is here

• Software switches are key enabler to network virtualization

• Opens the door for scalable, on-demand, ephemeral networks

• OVS is the default switch in Xen, and supported in XenServer and XenCP.

• CloudStack implements highly scalable network structures and leverages OVS capabilities

Page 36: Xen*, SDN, and Apache CloudStack

Participating in CloudStack

• Apache incubator project • http://www.cloudstack.org • #cloudstack on irc.freenode.net • @CloudStack on Twitter • http://cloudstack.org/discuss/mailing-lists.html

Welcoming contributions and feedback, Join the fun !