1
VXLAN Fundamentals, Architecture & Roadmap
2
Table of ContentsTable of Contents
1. Data Center IP Fabric ‘Building a strong Foundation’
2. What is ‘Network Virtualization’?
3. VXLAN Overview
4. VXLAN Packet details
5. VXLAN Terminology
6. VXLAN Host Discovery
7. VXLAN BUM Traffic Handling
8. VXLAN Layer 2 & Layer 3 Terminologies
9. VXLAN Arista Architecture & Vision
10.VXLAN Roadmap
11.VXLAN Visbility
3
Data Center – ‘IP Fabric’
Building A Strong Foundation
Challenges with current network architectureChallenges with current network architecture
Scalability Scales up and not scales out Dependent on specific hardware (mix &
match) Not scalable to 40GbE / 100GbE
Latency High latency Low predictability
Mobility What happens if my “IP” changes? What happens if traffic pattern changes?
Cost As multiple layers, it can get $$$
Oversubscription Ports on devices are oversubscribed ~ 8:1 Higher Oversubscription as traffic traverses
north ~ 20:1
Layer 2 Domain
Layer 2 Domain
Legacy Data Center Model
Multiple points of management, rampant oversubscription, wasteful
cost model
Layer 2 Domain
Layer 2 Domain
Data Center ‘IP Fabric’Data Center ‘IP Fabric’
Support for East/West 80:20 traffic pattern
Scale up to 64-way ECMP Spine designs
All uplinks from ToR are Active/Active
Support 100’000s of host ports
Non-blocking / Non-oversubscribed architecture
Deploy L3 routing protocols between leaf & spine i.e. BGP, OSPF, or ISIS
Everything is only 3 hops away!
Provide network mobility via ‘Overlay Network’
6
Arista – Spine/Leaf “IP Fabric” Arista – Spine/Leaf “IP Fabric” ArchitectureArchitecture
VTEP1VTEP1
IP Fabric
Spine Tier
Leaf Tier
A 1A 1 B 1B 1 A2A2 B2B2 Bare Metal Servers
Bare Metal Storage
HYPERVISOR 1HYPERVISOR 1 HYPERVISOR 2HYPERVISOR 2
VTEP2VTEP2
VTEP3VTEP3 VTEP4VTEP4
Network core is an IP fabric laid out in a Leaf-Spine architecture running ECMP between the two tiers- Leaf switches - Arista 7150-x or 7050Q-x models are deployed at the TOR connecting
virtualized servers, bare-metal servers, storage arrays and other devices
- Spine switches – Arista 7500’s are deployed at the core
- Routing Protocol – Either EGP (BGP) or IGP (OSPF / ISIS) is run in the IP fabric
7
What is Network Virtualization?
8
What is Network Virtualization?What is Network Virtualization?
Network Virtualization is not the same as Server Virtualization!
9
Overlays v UnderlaysOverlays v Underlays
Network virtualization: ability to separate, abstract and decouple the physical topology from a ‘logical’ or ‘virtual’ topology by using encapsulated tunneling.
This logical network topology is often referred to as an ‘Overlay Network’.
Overlay Network
Physical Infrastructure i.e. Underlay Network
VXLAN disassociates workloads from physical networks, allowing for possible transition to cloud based providers
10
Types of ‘Overlay’ TechnologiesTypes of ‘Overlay’ Technologies
Any Overlay technology uses Location & Identity separation
Location
Identity
Fabric Path
VXLAN OTV LISP
Underlay Protocol IS-IS BGP, OSPF, IS-IS BGP, OSPF, IS-IS BGP, OSPF, IS-IS
Location Switch-ID IP address IP address IP address
Identity Client MAC Client MAC Client MAC Client IP / Mac
Identity Learning Flooding Flooding / Dynamic learning
IS-IS Mapping DB
Vendor Proprietary Yes Non Yes Non
Intra & / or Inter DC Intra Both Both Inter
11
VXLAN Overview
12
Virtual Extensible Local Area Network (VXLAN)Virtual Extensible Local Area Network (VXLAN)
Ethernet in IP overlay network • Entire L2 frame encapsulated in
UDP
• 50 bytes of overhead
Include 24 bit VXLAN Identifier• 16 M logical networks
VXLAN can cross Layer 3
Tunnel between ESX hosts• VMs do NOT see VXLAN ID
IP multicast used for L2 broadcast/multicast, unknown unicast
Technology submitted to IETF for standardization• With Arista, Vmware, Red Hat, Citrix,
Cisco, and Others
Outer MACDA
Outer MACSA
Outer 802.1Q
Outer IP DA
Outer IP SA
Outer UDP
VXLAN ID (24 bits)
Inner MAC DA
InnerMACSA
Optional Inner
802.1Q
Original Ethernet Payload
CRC
VXLAN Encapsulation Original Ethernet Frame
13
Virtual eXtensible LAN: How does it Virtual eXtensible LAN: How does it work?work?
MAC&IP are UDP Encapsulated
Encapsulation at VTEP node is transparent to IP ECMP fabric
VM-110.10.10.1/24
VM-210.10.10.2/24
Subnet-A Subnet-B
Layer 2 Domain between the VMvWire- VNI 10
HW VTEPEncap/Decap
VXLAN Frames
SW VTEPEncap/DecapVXLAN VTEP
VTEP VTEP
14
VXLAN BenefitsVXLAN Benefits
Feature Benefits- Eliminates current networking challenges in the way of on-demand,
virtual environment:- VLAN Sprawl
- Single fault domains
- Scalability beyond 4096 segments
- Proprietary fabric solutions
- IP mobility
- Physical cluster size and locality
- Enables multi-tenancy at scale
- Decouples logical networks from physical infrastructure so that applications can be deployed without worrying about physical rack location, IP address or VLAN
- Based on open and well known standards
15
VXLAN Use CasesVXLAN Use Cases
Physical to Virtual internetworking
Multi-hypervisor connectivity and integration
Multi-tenant Cloud environments
HA clusters across failure domains
Dynamic growth
Dynamic resource management
16
VXLAN Packet Details
17
VXLAN PacketVXLAN Packet
VXLAN is a MAC-in-IP encapsulation
18
VXLAN HeaderVXLAN HeaderVXLAN Header is a 8 Byte field comprising of:
(a)Flags (8 Bits)
(b)VxLAN Network Identifier (VNI) (24 Bits)
(c)Reserved (24 & 8 Bits) – Always set to zero.
Reserved (24 & 8 Bits) – Always set to zero.
Flags (8 Bits) – I flag is set to 1 for a valid VxLAN Network ID (VNI). The remaining 7 bits (designated "R") are reserved fields and set to zero.
VxLAN Network Identifier (VNI) (24 Bits) – Used for identification of the individual VxLAN overlay network on which the communicating VMs are situated. VMs in different VxLAN overlay networks cannot communicate.
19
VXLAN Terminology
20
VXLAN Segments
VXLAN Terminology – Physical TopologyVXLAN Terminology – Physical Topology
Software VTEP
Hardware VTEP
VTEP1VTEP1
IP Fabric
Spine Tier
Leaf Tier
A 1A 1 B 1B 1 A2A2 B2B2Bare Metal
ServersBare Metal
Storage
HYPERVISOR 1HYPERVISOR 1 HYPERVISOR 2HYPERVISOR 2
VTEP2VTEP2
VTEP3VTEP3 VTEP4VTEP4
VTIVXLAN Gateway
VTI
VXLAN 10001 VXLAN 10001
VXLAN 10002VXLAN 10002
21
VNI
B2B2
VTEP 4VTEP 4
A2A2
10.100.1.0/24 10.100.2.0/24
.10 .11.2 .3
VXLAN 10001 VXLAN 10002
.10.2
VARP Default
Gateway:10.100.1.1
VARP Default
Gateway:10.100.1.1
VARP Default
Gateway:10.100.2.1
VARP Default
Gateway:10.100.2.1
ExternalHost
ExternalHost
DataCenter
Network
B1B1
VXLAN Terminology – Logical VXLAN Terminology – Logical TopologyTopology
A1A1
Bare Metal Storage
Bare Metal Servers
VTEP 1VTEP 1 VTEP 3VTEP 3 VTEP 1VTEP 1
VXLAN Segment VXLAN SegmentVARP
Default Gateway:
10.100.1.1
VARP Default
Gateway:10.100.1.1
VARP Default
Gateway:10.100.2.1
VARP Default
Gateway:10.100.2.1
.1 .1 .1 .1
22
VXLAN Terminology ExplainedVXLAN Terminology Explained
VTEP: VXLAN Tunnel End Point- VXLAN encapsulation and decapsulation happens at the VTEP
VXLAN Gateway - A device which bridges traffic from VXLAN and non-VXLAN environments. - VXLAN gateways allow for physical and non virtualized devices to communicate
with VXLAN networks
- A VXLAN gateway can be either a hardware or software device
VNI: Virtual Network Identifier - a 24-bit number is also called the VXLAN segment ID. The system uses the
VNI, along with the VLAN ID, to identify the appropriate tunnel.
VXLAN Header – is an 8-byte header that contains the 24-bit VNI value. It lives in between the UDP header and the inner MAC frame being carried over the VTI.
VTI: VTEP Tunnel Interface - a switchport linked to a UDP socket that can be shared between many VLANs. Packets bridged through a vlan into the VTI are sent out the UDP socket with a VXLAN header including a VNI. The socket is bound to a fixed local port, but is not connected to any particular destination port or IP address; logically, we use sendto() (not send()) to transmit VXLAN-encapsulated frames on the socket. Packets arriving on the VTI (via the UDP socket, based on their UDP destination port) are demultiplexed into a VLAN for bridging. A 24-bit VNI within the packet determines which VLAN the packet is mapped to for bridging.
VXLAN Segment - is a Layer 2 overlay network over which VMs communicate. Only VMs within the same VXLAN segment can communicate with each other.
23
VXLAN Visibility
24
VXLAN Visibility - Arista’s vmTracerVXLAN Visibility - Arista’s vmTracer
Full physical to virtual visibility
Network audit to ensure reachability
Automated provisioning
Workflow without finger pointing
Other awesome capabilities
25
Monitoring VXLANs with vmTracerMonitoring VXLANs with vmTracer
Physical
Virtualization
VMware NSXHyperviso
r
VTEP VTEP VTEP VTEP
Rapidly correlate vlan to VNI switch5#:show vmtracer vxlan interface Ethernet48
Ethernet48: esx1.aristanetworks.com/ndsTest/dvuplink1
VM Name VLAN vWire Network Multicast
--------------------------------------------------------------------------------------------
Exchange 5 Corp 172.20.20.0 239.20.20.0
Apache 6 web 182.10.0.0 220.10.10.0
MySQL 7 ERP 172.20.30.0 239.20.30.0
view VNIs across the data center from the CLIswitch9#:show vmtracer vxlan all
7150s R1: Ethernet 48:esx1/vwTest/dvUplink 1
vWire:Corp -- VLAN:5
vWire:ERP -- VLAN:7
7150s R2: Ethernet 40:esx2/vwTest/dvUplink 1
vWire:Corp -- VLAN:5
vWire:web -- VLAN:6
26
Automate Learning of VNI StateAutomate Learning of VNI State
OVSDB
VNI, VXLAN, VNI ID
VM- OskiVNI - CalBears
New VNI - CalBearsMulticast Group - 224.0.14.13VNI ID - 650782
Interface Ethernet 24 VXLAN VTEP VNI CalBears
Interface Loopback0 VXLAN VTEP Gateway VNI Calbears IP Address 204.181.40.1/24
<--Network
NSX Controller
27
Where is my VM now?Where is my VM now?
Aubie WarEagle vshield vm-tiger
VNI ‘Test’: 224.0.0.12
spine0
leaf1 leaf2
esx10 esx11
spine0: show vmtracer vxlan
VNI-Name VNI #VTEPs Learning Mcast Group Status Subnet Auburn 5096 4 Flood 224.0.1.95 Up 204.181.40.0/24 foo 15893425 5 Flood 224.0.4.84 Up 128.218.56.0/24 bar 65456 45 Flood 224.5.1.92 Down 192.168.10.0/20
VNI Name: AuburnVNI Segment ID: 5096 VTEP Type Status Inside Outside Learning Mcast Grp PIM-RP Switch Port ModelESX1 VMware Up 3 VNICs 204.181.21.5 Flood 224.0.1.95 204.181.1.16 ar16 eth15 7050Sar24 Arista Up/GW 204.181.40.1 204.181.1.16 Flood 224.0.1.95 204.181.1.16 ar24 loop0 7150Sar22 Arista Up/Up 1 MAC/IPs 204.181.3.67 Flood 224.0.1.95 204.181.1.16 ar22 eth2 7150SESX4 VMware Up 4 VNICs 204.181.1.5 Flood 224.0.1.95 204.181.1.16 ar2 eth23 7050T
spine0: show vmtracer vxlan vni Auburn
28
Where is my VM now?Where is my VM now?
128.218.11.x128.218.10.x
spine0
leaf1 leaf2
esx1 esx11
Aubie WarEagle vshield vm-tiger
spine0: show vmtracer interface vxlan Auburn
VTEP: ESX1 Role: vSwitch Switch/Port: ar16.foo.com/eth15 Name VNIC Status State IP Address Aubie Network Interface 1 Up/Up vMotion 204.181.40.2WarEagle Network Interface 2 Up/Up VM-FT-A 204.181.40.3BooBama Network Interface 1 Up/Down -- 204.181.40.5
VTEP: ar24 Role: Router Switch/Port: ar24.foo.com/loopback0NAT/PATStatus#ARPs IP Address No Up/Up 45 204.181.40.1
VTEP: ar22 Role: Port-VTEP Switch/Port: ar22.foo.com/eth2 FQDN IP MAC VLAN Statusisilon16.foo.com 204.181.40.190 00-00-45-ab-12-fe 5 Up/Up
29
THANK YOU