BayLISA: MidoNet Overlay Based Network Virtualization for IaaS Clouds

MidoNet: Overlay-based Virtual Networking for IaaS Clouds

March 21, 2013

Adam Johnson General Manager, Midokura

@adjohn

Copyright ©2012 Midokura All rights reserved

Requirements

2

Tenant/Project A

Network A1

VM1 VM3

Network A2

VM5

Tenant/Project B

Network B1

VM2 VM4

uplink

Provider Virtual Router (L3)

Tenant AVirtual Router

Tenant BVirtual Router

VM6

Virtual L2 Switch B1

Virtual L2 Switch A1


TenantB office

Tenant BVPN Router

Office Network


Tenant/Project A

Network A1

VM1 VM3

Network A2

VM5

Tenant/Project B

Network B1

VM2 VM4

uplink




VM6




TenantB office

Tenant BVPN Router

Office Network

Requirements

3

Isolated tenant network (virtual

data center)


Tenant/Project A

Network A1

VM1 VM3

Network A2

VM5

Tenant/Project B

Network B1

VM2 VM4

uplink




VM6




TenantB office

Tenant BVPN Router

Office Network

Requirements

4

L3 isolation (similar to VPC and VRF)


Tenant/Project A

Network A1

VM1 VM3

Network A2

VM5

Tenant/Project B

Network B1

VM2 VM4

uplink




VM6




TenantB office

Tenant BVPN Router

Office Network

Requirements

5

Isolated L2 networks


Tenant/Project A

Network A1

VM1 VM3

Network A2

VM5

Tenant/Project B

Network B1

VM2 VM4

uplink




VM6




TenantB office

Tenant BVPN Router

Office Network

Requirements

6

Redundant, optimized and fault-tolerant

paths to the Internet (e.g. via BGP)


Tenant/Project A

Network A1

VM1 VM3

Network A2

VM5

Tenant/Project B

Network B1

VM2 VM4

uplink




VM6




TenantB office

Tenant BVPN Router

Office Network

Requirements

7

Fault-tolerant devices and links


Tenant/Project A

Network A1

VM1 VM3

Network A2

VM5

Tenant/Project B

Network B1

VM2 VM4

uplink




VM6




TenantB office

Tenant BVPN Router

Office Network

Requirements

8

NAT, LB, and Filtering

NAT, LB, and Firewalls


Tenant/Project A

Network A1

VM1 VM3

Network A2

VM5

Tenant/Project B

Network B1

VM2 VM4

uplink




VM6




TenantB office

Tenant BVPN Router

Office Network

Requirements

9

L3 (and L2) VPNs


Tenant/Project A

Network A1

VM1 VM3

Network A2

VM5

Tenant/Project B

Network B1

VM2 VM4

uplink




VM6




TenantB office

Tenant BVPN Router

Office Network

Requirements

10

Minimize ARP broadcasts by exploiting CMS config

RESTful API for CMS integration and direct

tenant access

Solid integration with leading open CMS: ��

OpenStack, CloudStack DHCP, DNS and other

services


Requirements: recap

11

l  Multi-tenancy l  Scalable, fault-tolerant

devices (or device-agnostic network services).

l  L2 isolation l  L3 routing isolation

u  VPC u  Like VRF (virtual

routing and fwd-ing) l  BGP gateway l  Scalable control plane

u  ARP, DHCP, ICMP l  Floating IP

l Stateful NAT

u  Port masquerading u  DNAT

l ACLs l Stateful (L4) Firewalls

u  Security Groups l LB health checks l VPNs at L2 and L3

u  IPSec l REST API l Integration with CMS

u  OpenStack u  CloudStack


How to build it?

12

1. Virtualized physical devices

2. Centrally controlled OpenFlow-based hop-by-hop switching fabric

3. Edge to edge overlays


Virtualized physical devices

13

l  4096 limit on number of unique tags l  Large spanning trees terminating on many hosts l  High churn in switch control planes due to MAC learning

l Each VM is separate virtual MAC! l  Need MLAG for L2 multi-path (vendor specific)

1

VLAN VLAN1

VLAN2



14

l  L2 isolation l  What about L3 and Internet access? l  Use VRF or virtual appliances?

1

VLAN (more) VLAN1

VLAN2



15

1

l  Not scalable to cloud scale l  Expensive hardware l  Not fault tolerant (HSRP?) l  L2 and L3 isolation. What about NAT, LB, FW?

出典：http://infrastructureadventures.com/tag/vrf-lite/

VRF

Core VLAN 10 VLAN11 VLAN12

Product VLAN 20 VLAN21 VLAN22

Sales VLAN 99

VRF VRF VRF


OpenFlow hop-by-hop switch fabric

16

2

OpenFlow Switches

OpenFlow Controller (Cluster)

l  Fabric extends to the compute host software switch? • State in each switch is proportional to the virtual network state • Need to update all switches in path when provisioning new virtual devices or updating them. • Not scalable, slow and non-atomic switch updates.


OpenFlow hop-by-hop switch fabric (more)

17

2

OpenFlow Switches

OpenFlow Controller (Cluster)

l  Flow rules for VM flows (microflows)? l  Flow rules for virtual device simulation?


　Edge-to-Edge Overlays

18

3

VM

VM Edge

Edge Edge

Edge Edge

Edge

Use scalable IGP (iBGP, OSPF) to build multi-path underlay



19

3

VM

VM Edge

Edge Edge

Edge Edge

Edge

IP encapsulation provides isolation



20

3

VM

VM Edge

Edge Edge

Edge Edge

Edge

Virtual network processing at ingress host, decoupled from

physical network



21

3

VM

VM Edge

Edge Edge

Edge Edge

Edge

Virtual network changes don't affect

underlay state



22

3

• Packet processing on x86 CPUs (at edge) •  Intel DPDK facilitates packet processing •  Number of cores in servers increasing fast

• Clos Networks (for underlay) • Spine and Leaf architecture with IP • Economical and high E-W bandwidth

• Merchant silicon (cheap IP switches) •  Broadcom, Intel (Fulcrum Micro), Marvell •  ODMs (Quanta, Accton) starting to sell directly •  Switches are becoming just like Linux servers


Overlays are the right approach!

But not sufficient... We still need a scalable control plane.

23


MidoNet SDN Solution

24

Logical Topology

vPort

Provider Virtual Router

Tenant A Virtual Router

Tenant B Virtual Router

Virtual Switch A1

Virtual Switch A2

Virtual Switch B1

vPort

vPort

vPort

vPort

vPort



25

Logical Topology

Private IP Network

MN

MN

MN

Internet

BGP Multi

Homing

Physical Topology

MN VM

VM

MN VM

VM

MN VM

VM

BGP To ISP3

BGP To ISP2

BGP To ISP1

vPort

Provider Virtual Router

Tenant A Virtual Router

Tenant B Virtual Router

Virtual Switch A1

Virtual Switch A2

Virtual Switch B1

vPort

vPort

vPort

vPort

vPort

Network State Database

MN MN MN

Tunnel



26

Distributed State

MidoNet REST API

Dashboard



27

Distributed State

Linux Kernel + OVS KMOD

VM1 MidoNet

Ctrl

HW


VM2 MidoNet

Ctrl

HW

Host A Host B

Lazy state propagation



28

Distributed State


VM1 MidoNet

Ctrl

HW


VM2 MidoNet

Ctrl

HW

Host A Host B

VM sends first packet; table miss; NetLink

upcall to MidoNet



29

Distributed State


VM1 MidoNet

Ctrl

HW


VM2 MidoNet

Ctrl

HW

Host A Host B

MidoNet agent locally processes packet (virtual layer simulation); installs

local flow (drop/mod/fwd)



30

Distributed State


VM1 MidoNet

Ctrl

HW


VM2 MidoNet

Ctrl

HW

Host A Host B

Packet tunneled to peer host; decap; kflow table miss; Netlink notifies

peer MidoNet agent



31

Distributed State


VM1 MidoNet

Ctrl

HW


VM2 MidoNet

Ctrl

HW

Host A Host B

MN agent maps tun-key to kernel datapath port#; installs fwd flow

rule



32

Distributed State


VM1 MidoNet

Ctrl

HW


VM2 MidoNet

Ctrl

HW

Host A Host B

Subsequent packets matched by flow rules

at both ingress and egress hosts



33

• Distributed and scalable control plane Ø Handle all control packets at local MidoNet agent

adjacent to VM • Scalable and fault tolerant central database

Ø Stores virtual network configuration Ø Dynamic network state

² MAC learning, ARP cache, etc Ø Cached at edges on demand

• All packet modifications at ingress Ø One virtual hop

² No travel through middle boxes Ø Drop at ingress



34

• Scalable edge gateway interface to external networks • Multihomed BGP to ISP • REST API and GUI • Integration with popular open source cloud stacks • OpenStack • Removes SPOF of network node • Scalable and fault tolerant NAT for floating IP • Implements security groups efficiently

• CloudStack



35

Deep OpenStack Integration

• Quantum Plugin • L2 isolation, of course • Also… • L3 isolation (without VM / appliance) • Security groups (stateful firewall) • Floating IP (NAT) • Load balancing (L4)


HorizonWeb GUI

Quantum Plugin

MidoNet Manager (Web GUI)

Nova API

MidoNet API

RabbitMQ - Passing Queue

Compute Host

NovaCompute

Libvirt driver

Datapath

Inst

an

ce

A1

Inst

an

ce

B1

MidoNetAgent

Quantum

API

MidoNet Plugin

MidoNet Distributed

State

MidoNet Distributed

State

MidoNet Distributed

State

MidoNet EdgeAgent

MidoNet EdgeAgent

MidoNet EdgeAgent

INTERNET

MidoNetVIF Driver

KeystoneAuthentication

OpenStack Integration



37

Future Directions

• Scalable L7 virtual appliances • Content aware load balancer • MPLS VPN termination Ø Interconnect with carrier backbones • multiple data center federation Ø Virtual L2 between sites • LISP Ø Global IP mobility between sites



38

Conclusions

• IaaS clouds require new networking model • Edge to edge overlays are the right

approach • Servers are good enough at packet

processing Ø Can use them for edge gateways • Multipath IP network fabric is cheap and

easy to build

Questions?

[email protected]

We’re hiring http://midokura.com/careers/



40

Backup Slides



41

MN

Packet Encapsulated

Tunnel

Drop/Block

Packet from VM, VPN, or external BGP peer

enters kernel datapath



42

MN

Packet Encapsulated

Tunnel

Drop/Block

One flow rule reflecting the outcome of the virtual layer

simulation AND the mapping of egress vport to

peer host decides to drop or fwd


Spine and Leaf Network Architecture

43

L3 Switch Spine

Leaf

L3 Switch L3 Switch L3 Switch

L3 Switch L3 Switch L3 Switch x32

x4

48x10G

1536 x 10G

4x40G

e.g Force10 Z9000

IBGP and ECMP

e.g Arista 7050T

e.g Force10 Z9000

Technology

BayLISA: MidoNet Overlay Based Network Virtualization for IaaS Clouds