Presenters Jack McCann Rajeev Grover ... - OpenStack · Rajeev Grover Swaminathan Vasudevan Vivekanandan Narasimhan ... Introduction High level architecture and DVR ... DVR High level

Presenters

Jack McCann Rajeev Grover

Swaminathan Vasudevan Vivekanandan Narasimhan

Paris OpenStack® Summit

Agenda

Introduction High level architecture and DVR

Configuration

North-South Routing East-West Routing OVS Rules

Architectural Overview of Distributed Virtual Routers in OpenStack Neutron Paris OpenStack® Summit

Scheduling Services API Changes and DB Extensions

Future Plans for DVR

Legacy Routing in Neutron Network node provides:

• IP forwarding

– Inter-subnet (east-west) traffic between VMs

– Floating IP (north-south) traffic between external and VM

– Default SNAT (north-south) traffic from VM to external

• Metadata Agent

– access to Nova metadata service

Issues:

• Performance bottleneck

• Scalability limitations

• Single Point of Failure


Compute node

Network node

Compute node

VM1 VM2

VM1 VM2

Virtual

Physical

Router A

RtrA

Distributed Routing in Neutron Compute node provides:

• IP forwarding for local VMs

– Inter-subnet (east-west) traffic between VMs

– Floating IP (north-south) traffic between external and VM

• Metadata Agent for local VMs

– access to Nova metadata service

Advantages:

• Bypass network node improves performance

• Scales with size of compute farm

• Limited failure domain (per compute node)

Limitations:

• Default SNAT function is still centralized


Compute node

Network node

Compute node

VM1 VM2

VM1 VM2

Virtual

Physical

Router A

RtrA Default

SNAT RtrA’’ RtrA’

High level requirements for DVR

• Help close the parity gap with Nova (multi-host)

• Provider feature

– Tenants should not have to know or care

• Configurable on a per-router basis (centralized or distributed)

– Default router type set by a global config knob

• Can be deployed into existing environments

• Centralized routers and Distributed routers can coexist in same cloud

• Ability to migrate a router from Centralized to Distributed

• Minimize overhead use of public IP addresses

• Leverage existing code base


DVR High Level Architecture and Configuration


Network/Service Node(s)

Compute Node(s)

VM VM

VM

l3-agent ovs-agent

Controller Node

Neutron controller/plugin L3 Plugin

Network/Service Node(s)

l3-agent ovs-agent

dhcp-agent qr

qr qr

agent_mode=dvr

router_distributed=True

enable_distributed_routing=True

agent_mode=dvr_snat

enable_distributed_routing=True • One FIP namespace supports multiple routers

• Legacy OpenStack Routers (non-distributed) can exist on the network/service node

• Enhanced l3-agent works in different modes “dvr_snat”, “legacy” and “dvr”.

FIP

SNAT

Architectural Overview of Distributed Virtual Routers in OpenStack Neutron

metadata

metadata

qr qr

DVR High level Architecture


Configuring DVR



“neutron.conf”

router_distributed=True

Plugin Configuration ( Default Router Option)

“l3_agent.ini”

agent_mode=dvr_snat ( Network/Service Node)

agent_mode=dvr (Compute Node only)

agent_mode=legacy ( Network/Service Node)

router_delete_namespaces=True ( Enable namespace

cleanup)(Optional)

L3 Agent Configuration and Modes

DVR Configuration Overview


“ml2_conf.ini”

enable_distributed_routing = True enable_tunneling = True

local_ip = <Local Data Network IP (or) TEP IP>

tunnel_types = vxlan

l2_population=True

L2/OVS Agent Configuration

DevStack Configuration “local.conf” Q_DVR_MODE=dvr_snat/dvr/legacy

DVR Configuration Overview

br-int br-int br-int

br-tun br-tun br-tun br-ex

VM1 VM2 VM3 VM4 DHCP Agent L3 Agent

OVS Agent

Q_DVR_MODE=dvr_snat Q_DVR_MODE=dvr Q_DVR_MODE=dvr

L3 Agent

L3 Agent

qr qr qr

snat

Network/Service Node Compute Node Compute Node

DVR Deployment without FIP

br-ex br-ex


qr

Legacy Deployment

Q_DVR_MODE=legacy

br-int br-int

br-tun br-tun

VM1 VM2 VM3 VM4

OVS Agent


L3 Agent

L3 Agent

qr qr

fip fip

br-ex br-ex

DVR Deployment with FIP

br-int

br-tun br-ex

qr

DHCP Agent

L3 Agent

qr

snat

Network/Service Node Compute Node Compute Node


VM5

External Network

North-South Routing


Overview of North-South Routing

External Network

vm1 vm3

br-int

br-tun

vm5

br-ex

• IP Rules

•Routing Entries

• Iptable based NATing rules

• /32 addresses for the FIP

• One Floating IP Namespace per CN

• Interface on external network

• Host routes to qr for FIP

• Proxy ARPs enabled for FIP

qr

FIP

L3 Agent

Data Network



Default SNAT Traffic flow


East-West Routing


L3 Agent

Rules ------ ------

Rules ------ ------

vm1 vm2

br-int

br-tun

qr

LMAC

East-West routing is accomplished through a combination of the following in each CN:

•qr : a namespace that forwards traffic among routed subnets using route table entries.

•LMAC: a MAC address unique to each CN for use as source address in forwarding routed frames from one CN to another.

•A set of rules in the OVS bridges that prevent frames with src=gateway MAC from egressed tunnel bridge . These rules also cause peer bridges to substitute/restore gateway MACs with LMACs while routed frames traverse

through br-tuns and underlay network.

br-ex

Compute Node entities for enabling the DVR East West Routing

L3 Agent


vm1

br-int-cn1

br-tun-cn1

qr

vm2

br-int-cn2

br-tun-cn2

qr

Tenant has two VMs - vm1 that is in RED Net & vm2 that is in GREEN Net CN1 and CN2 are two compute nodes. qr represents a dvr owned by Tenant 1

Data Network

CN1 CN2

PING REQUEST from vm1 to vm2

vm1 ip qr red mac vm1 mac vm2 ip

vm1 ip qr red mac

vm2 ip

vm1 ip vm2 mac qr grn mac vm2 ip

vm1 ip vm2 mac l_mac vm2 ip

vm1 ip vm2 mac l_mac vm2 ip

1

2

3

4

5

grn vni

grn 2 vlan

vm1 ip vm2 mac qr grn mac vm2 ip 6

vm1 mac

Flow of DVR routed packets (example)


Table 0: Dispatch from in_port from patch-int: goto table 1 default: drop

Table 1: DVR process Table If packet is from dvr-router-intf-mac, replace SRCMAC with DVR-LMAC and forward to table 2

Table 2: Local VLAN to TUN Table If unicast frame, forward to table 20 Table 20: Unicast to Tunnel Table

Pre-populated L2-pop rules that bind dst-mac in frame to a single output tunnel

Frame out to single tunnel

Table 0: Local Switching Table from patch-tun: Is src-mac == DVR-Unique-MAC - goto table 1 default: forward packet to ports - NORMAL action

1

2 to dvr router

Integration Bridge (br-int)

Tunnel Bridge (br-tun)

4 5

6

3 from dvr router

7

8


Egress to Cloud


Table 0: Dispatch from in_port from vxlan-port: forward to table 4 default: drop

Table 4: VXLAN Tun to Local Vlan Table match tun_id: Set local vlan-id, forward to table 9 default: drop

Table 9: DVR Learning blocker If src_mac == DVR-Unique-MAC frame, don’t learn, just output to patch-int default: forward to table 10

4

Integration Bridge (br-int)

Tunnel Bridge (br-tun)

1

2

3

Ingress from Cloud Table 0: Local Switching Table from patch-tun: Is dvr_routed - goto table 1 default: forward packet to ports - NORMAL action

Table 1: DVR to LMAC Table match DVR-Unique-MAC: strip off dvr-unique-mac, add local dvr-router-intf-mac, output direct to vm-port default: drop

5

vm1


6

Scheduling

Achitectural Overview of Distributed Virtual Routers in OpenStack Neutron Paris OpenStack® Summit

br-int

br-tun

VM3 VM2

OVS Agent

Q_DVR_MODE=dvr_snat Q_DVR_MODE=dvr

L3 Agent

qr

br-ex

Scheduling a “qr/snat” in Service/Compute Node

br-int

br-tun br-ex

qr

DHCP Agent L3 Agent

qr

snat

Schedule Events Create a Router Add one or more subnets

with VMs Set a default Gateway for

the Router

Compute Node Service/Network Node


DHCP DHCP

VM1

DHCP

External Network

Internal Network

br-int

br-tun

VM3 VM2

OVS Agent

Q_DVR_MODE=dvr_snat Q_DVR_MODE=dvr

L3 Agent

qr

fip

br-ex

Compute Node

Scheduling a “fip” in Compute Node

Schedule Events Make sure you have a router

configured with External Network or gateway.

Create a Floating IP Associate a Floating IP to a

VM


br-int

br-tun br-ex

qr

DHCP Agent L3 Agent

qr

snat

Service/Network Node

DHP DHCP DHCP

VM1

External Network

Internal Network

Un-Scheduling Routers and FIP

Un Schedule Events FIP Namespace cleanup Last VM holding the FIP determines the FIP namespace cleanup Delete a FloatingIP Disassociate a Floating IP

Router Namespace cleanup When no more DVR related ports ( Includes Compute, VIP, DHCP etc.,) are serviced by the

routed subnet, the router namespaces are cleaned up. Both l2 agent and l3 agent takes part in the clean up.

SNAT Namespace cleanup When a gateway is removed from a router the snat namespace will be cleaned up. NOTE: Namespace cleanup should be enabled “router_delete_namespaces=True” in l3_agent.ini


Services


Services support with DVR

Services Support in DVR as of Juno LBaaS

FWaaS

• North-South only ( External Traffic) • East-West ( Internal Traffic)

MetadataService VPNaaS is still supported with Centralized Routers. VPNaaS support for DVR is Work in Progress.


br-int br-int

br-tun br-tun

VM1 VM2 VM3 VM4

OVS Agent


L3 Agent

L3 Agent

qr qr

fip fip

br-ex br-ex

Firewall Firewall

LBaaS

Agent VIP

br-int

br-tun br-ex

qr

DHCP Agent

L3 Agent

qr

snat

Network/Service Node

Firewall


Compute Node Compute Node

Firewall

Services deployment with DVR

Meta data

Meta data


VM5

External Network

Internal Network

API Changes and DB Extensions



API Extensions Adds ‘distributed’ attribute to the ‘router’ object

Router Create: Router Update: Can be set by admin user through the above API Global default is set as “router_distributed” in neutron.conf The attribute is only visible to admin tenant in GET. NOTE: Migrating or Converting a Legacy Router to Distributed is Work in Progress.


neutron router-create --distributed=True/False

neutron router-update --distributed=True/False


DB Changes for DVR ml2_dvr_port_bindings


Future Plans



Future Plans for DVR VPNaaS support for DVR Full migration support for DVR routers. HA for Service Node IPv6 Support VLAN Support L3 Agent Refactor Distributed DHCP Performance tuning. Distributed SNAT


Questions?


THANK YOU


Documents

Presenters Jack McCann Rajeev Grover ... - OpenStack · Rajeev Grover Swaminathan Vasudevan Vivekanandan Narasimhan ... Introduction High level architecture and DVR ... DVR High level