Upload
jongho-seo
View
2.295
Download
0
Embed Size (px)
Citation preview
클라우드 데이터센터 네트워크 아키텍처
VXLAN Deployment Models –Cisco-
KrDAG
2014.12
Cisco BRKDCT-2404
http://sola99.tistory.com
Overlay Network Evolution
[ Overlay Network 발전 방향 ]
Hybrid Overlays
Network DB
A p p O S
A p p O S
Virtual Physical
Network Overlays
Protocols
Physical Physical
V M O S
V M O S
Virtual Virtual
V M O S
V M O S
Host Overlays
Flooding
[ VXLAN Evolution ]
Overlay Network 발전방향과 VXLAN Evolution
Edge(장비), OTV/VPLS/LISP등 Edge(서버), VXLAN/NVGRE/STT Edge(물리+가상), 통합/연동/오픈/표준
Multicast Independent Head-end replication enables unicast-only mode Control Plane provides dynamic VTEP discovery
Protocol Learning prevents floods Workload MAC addresses learnt by VXLAN NVEs Advertise L2/L3 address-to-VTEP association information in a protocol
External Connectivity VXLAN HW Gateways to other encaps/networks VXLAN HW Gateway redundancy Enable hybrid overlays
IP Services VXLAN Routing Distributed IP Gateways
http://sola99.tistory.com
VXLAN Evolution: Multicast Independent
Using a Control Protocol: VTEP Discovery
VTEP이 BGP를 통해 자신의 VNI 정보를 광고하고 BGP RR을 통하여 전파: VTEP 자동 발견됨 [ VTEP Discovery ]
VTEP IP B
POD2
VTEP IP C
POD3
VTEP IP A
POD1
BGP consolidates and propagates VTEP list for VNI
2
4 VTEP can perform Head-End Replication
Overlay Neighbors POD3 , IP C POD2 , IP B
3 VTEP obtains list of VTEP neighbors for each VNI
1
VTEPs advertise their VNI membership in BGP
BGP Route Reflector
1
1
http://sola99.tistory.com
VXLAN Evolution: Multicast Independent
VXLAN Unicast Mode: Head-end replication
단말이 *BUM 트래픽 발생 시 VTEP에 등록된 네이버 VTEP에게 복제하여 Unicast로 각각 전달 [ Head-end replication ]
Unicast-Only Transport
VTEP IP B
POD2
VTEP
A host sends a L2 BUM* frame
POD1 VXLAN Encap 4
3 VTEP performs Head-End Replication
VTEP
Overlay Neighbors POD3 , IP C POD2 , IP B
2 VTEP retrieves the list of Overlay Neighbors**
BUM Frame 1
IP A => IP B BUM Frame
IP A BUM Frame
*Broadcast, Unknown Unicast or Multicast = BUM 트래픽
5 Frames are unicast to the neighbors
IP C
POD3
IP A => IP C
http://sola99.tistory.com
VXLAN Evolution: Protocol Learning prevents floods
Using a Control Protocol: Protocol Learning
호스트 라우팅 정보(IP+MAC)를 BGP를 통하여 광고하고 BGP RR을 통하여 전파됨 [ Protocol Learning ]
POD2
VTEP IP C
POD3
VTEP IP A
POD1
BGP propagates routes for the host to all other VTEPs
2
VTEP IP B
Overlay Forwarding Table Host1 <MAC,IP> , VTEP IP A Host2 <MAC,IP> , VTEP IP B
3
1
VTEPs advertise host routes (IP+MAC) to local hosts
BGP Route Reflector
Overlay Forwarding Table Host1 <MAC,IP> , VTEP IP A
3 VTEPs obtain host routes for remote hosts and install in RIB/FIB
http://sola99.tistory.com
VXLAN Evolution: Protocol Learning prevents floods
VXLAN Control Plane: Host and Subnet Route Distribution
Leaf 장비가 MP-BGP를 통하여 호스트 정보 및 외부 네트워크 정보를 광고 및 전파 [ Host and Subnet Route Distribution ]
Route-Reflectors deployed for scaling purposes
Aggregation RR RR
Access
iBGP Adjacnencies
Host Route Distribution decoupled from the Underlay protocol
Use MP-BGP on the leaf nodes to distribute internal host/subnet routes and external
reachability information
iBGP 내에서 Spine(Aggregation)에 BGP RR을 통하여 전파
http://sola99.tistory.com
VXLAN Evolution: Protocol Learning prevents floods
VXLAN Control Plane: Host Advertisement
Leaf 장비가 MP-BGP를 통하여 호스트 정보 및 외부 네트워크 정보를 광고 및 전파 [ Host Advertisement ]
1) Host Attaches : 단말 생성
2) Attachment NVE advertises host’s MAC (+IP) through BGP RR : NVE(Network Virtualization
Edge)가 host’s MAC과IP 정보를 BGP를 통하여 광고
3) Choice of encapsulation is also advertised : VXLAN 인지, NVGRE인지 선택하여 광고
NLRI: Host MAC1, IP1 NVE IP 1 VNI 5000 Ext.Community: Encapsulation: VXLAN, NVGRE
Sequence 0
MAC IP VNI Next- Hop
Encap Seq
1 1 5000 IP1 VXLAN 0
Access
Aggregation RR RR
VNI 5000
Host 1 VLAN 10
http://sola99.tistory.com
VXLAN Evolution: Protocol Learning prevents floods
VXLAN Control Plane: Host Move
Leaf 장비가 MP-BGP를 통하여 호스트 정보 및 외부 네트워크 정보를 광고 및 전파 [ Host Move ]
1) Host Moves to NVE3 : 호스트가 NVE3 아래쪽으로 이동
2) NVE3 detects Host1 and advertises H1 with seq#1 : NVE3이 호스트 정보를 광고하고 변경을 알
리는 시퀀스 넘버를 1 증가하여 전파(즉, 최신 정보라는 뜻)
3) NVE1 sees more recent route and withdraws its advertisement : NVE1은 이동을 인지
NLRI: Host MAC1, IP1 NVE IP 3 VNI 5000 Ext.Community: Encapsulation: VXLAN, NVGRE
Sequence 1
MAC IP VNI Next- Hop
Encap Seq
1 1 5000 IP3 VXLAN 1
Access
Aggregation RR RR
VNI 5000
Host 1 VLAN 10
http://sola99.tistory.com
VXLAN Evolution: External Connectivity
VXLAN L2 and L3 Gateways: Connecting VXLAN to the broader network
VXLAN과 외부간 연결 시 L2 or L3 로 연결 가능 [ Connecting VXLAN to the broader network ]
Destination is in another segment. Packet is routed to the new segment
VXLANORANGE VXLANBLUE
Ingress VXLAN packet on Orange segment
VXLAN Router
L2 Gateway: VXLAN to VLAN Bridging VXLANORANGE
Ingress VXLAN packet on Orange segment
Egress interface chosen (bridge may .1Q tag the packet)
VXLAN L2 Gateway
SVI
Egress interface chosen (bridge may .1Q tag the packet)
L3 Gateway: VXLAN to X Routing • VXLAN • VLAN VLAN100 VLAN200
N1Kv N7K w/F3 LC Nexus 3K N5K/6K N9K CSR 1000V ASR1K/ASR9k
L2 Gateway VXGW on 1110 Yes Yes Yes Yes N/A Yes
L3 Gateway CSR 1000V Yes No Roadmap Roadmap Yes Yes
http://sola99.tistory.com
VXLAN Evolution: External Connectivity
The Multi-encapsulation Gateway
다양한 오버레이 프로토콜 지원, L2/L3 Gateway 지원 [ The Multi-encapsulation Gateway ]
Destination is in another segment. Packet is routed to the new segment
VXLANORANGE
Encap Bridge
SVI
Encap Router
VX
LA
NBLU
E
LIS
PG
REEN
NVGREORANGE
VLANORANGE
MPLS
GREEN
Multi-encapsulation Gateway:
• VXLAN, NVGRE, MPLS, LISP, VLAN, OTV
Bridging (L2 Gateway)
Routing (L3 Gateway)
http://sola99.tistory.com
VXLAN Evolution: IP Services
VXLAN Routing: L2 VXLAN Fabric
Leaf Switch와 물리서버에서 VXLAN(VTEP)이 동작하여 내부에 단말 이동성 보장 [ L2 VXLAN Fabric ]
VXLAN L2 Gateway
VXLAN L2 Gateway
VM
OS
VM
OS
VXLAN L2 Gateway
VXLAN L2 Gateway
VXLAN L2 Gateway
VXLAN L2 Gateway
http://sola99.tistory.com
VXLAN Evolution: IP Services
VXLAN Routing: Centralized Routing in a L2 VXLAN Fabric
코어 네트워킹(혹은 외부)경우 상단에 VXLAN L3 장비(이중 배치)를 통하여 통신 [ Centralized Routing in a L2 VXLAN Fabric ]
VXLAN L3 Gateway
VXLAN L3 Gateway
VXLAN L2 Gateway
VXLAN L2 Gateway
VM
OS
VM
OS
VXLAN L2 Gateway
VXLAN L2 Gateway
HSRP
SVI SVI
VXLAN L2 Gateway
Inter-V(X)LAN and Core Routing
VXLAN L2 Gateway
IP Core
http://sola99.tistory.com
VXLAN Evolution: IP Services
VXLAN Routing: VTEP Redundancy in a VXLAN Fabric
vPC는 MAC 정보 동기화, VTEP 이중화를 위하여 VTEP IP를 Anycast로 제공(HSRP peer 사용) [ Centralized Routing in a L2 VXLAN Fabric ]
L3 Fabric
L2 Gateway redundancy based on vPC (anycast VTEP address)
vMAC -> Emulated VTEP VM VM
VXLAN L3 Gateway
VXLAN L3 Gateway
VXLAN L2 Gateway
VXLAN L2 Gateway
L3 GW redundancy based On vPC and HSRP (2 nodes)
http://sola99.tistory.com
VXLAN Evolution: IP Services
Distributed IP Gateways
기존 L2 환경과 L2/L3 Fabric 환경 비교 [ Traditional L2 & L2/L3 Fabric(or overlay) ]
App
OS
App
OS
L3 Boundary
L3 Boundary
App
OS
App
OS
Virtual Physical
L2/L3 Fabric
Virtual Physical
Traditional L2 - centralised L2/L3 boundary
• Always bridge, route only at an aggregation point
• Large amounts of state converge
• Scale problem for large# of L2 segments
• Traditional L2 and L2 overlays
L2/L3 fabric (or overlay)
• Always route (at the leaves), bridge when necessary
• Distribute and disaggregate necessary state
• Optimal scalability
• Enhanced forwarding and L3 overlays
http://sola99.tistory.com
VXLAN Evolution: IP Services
Distributed IP Gateways: Distributed IP Anycast Gateway
호스트는 어느 Leaf 장비 아래에 있어도 동일한 GW IP(MAC)을 발견(이후 외부 통신) [ Distributed IP Anycast Gateway ]
VXLAN L3 Gateway
L3 Fabric
VXLAN L3 Gateway
VM
OS
VM
OS
VXLAN L3 Gateway
VXLAN L3 Gateway
VXLAN L3 Gateway
SVI IP Address MAC: 0000.dead.beef IP:
10.1.1.1
SVI IP Address MAC: 0000.dead.beef IP:
10.1.1.1
The same“Anycast” SVI IP/MAC is used at all VTEPs/ToRs
A host will always find its SVI anywhere it moves
http://sola99.tistory.com
VXLAN Evolution: IP Services
Distributed IP Anycast Gateway: IP Forwarding within the Same Subnet aka Bridging (ARP)
동일 서브넷 2개의 단말이 통신 과정 소개 [ Distributed IP Anycast Gateway ]
VXLAN L3 Gateway
VXLAN L3 Gateway
L3 Fabric
V M O S
V M O S
VM1 10.10.10.10 PM1 10.10.10.20
1
CPU 2
3
rib
PM1 ARP Cache 10.10.10.1 -> GW_MAC
MAC IP L2 VNI L3 VNI
PM1_MAC 10.10.10.20 10000 50000
1) PM1 sends an ARP request
for Default Gateway –
10.10.10.1
2) The ARP request is
suppressed at TOR and
punted to the Supervisor,
where MAC and IP is learned
and distributed
3) TOR response with Gateway
MAC to PM1
Standard behavior of End-Host (virtual or physical) to ARP for the Default Gateway
http://sola99.tistory.com
VXLAN Evolution: IP Services
Distributed IP Anycast Gateway: IP Forwarding within the Same Subnet aka Bridging (ARP)
VM1단말이 PM1단말에 arp 요청을 leaf 에서 처리 [ Distributed IP Anycast Gateway ]
VXLAN L3 Gateway
VXLAN L3 Gateway
L3 Fabric
V M O S
V M O S
VM1 10.10.10.10 PM1
10.10.10.20
4
CPU 5
6
rib
VM1 ARP Cache 10.10.10.20 -> PM1_MAC
MAC IP L2 VNI L3 VNI
VM1_MAC 10.10.10.10 10000 50000
If there is Unicast RIB miss on TOR, ARP request will be forwarded to all ports except the
original sending port (ARP snooping). ARP response will be punted to Supervisor of destination
TOR for Unicast RIB population (learn) and subsequently forwarded to source TOR.
4) VM1 sends an ARP request for
PM1 – 10.10.10.20
5) The ARP request is suppressed
at TOR and punted to the
Supervisor, where MAC and IP
is learned and distributed
6) Assuming PM1 is known and a
valid route does exist in the
Unicast RIB, TOR responds to
ARP with PM1 MAC as Source
MAC. VM1 can build its ARP
cache
http://sola99.tistory.com
VXLAN Evolution: IP Services
Distributed IP Anycast Gateway: IP Forwarding within the Same Subnet aka Bridging (Data)
leaf 에서 PM1 MAC을 L2 Lookup 이후 VXLAN 패킷으로 오른쪽 leaf 로 전달 이후 PM1에 도착 [ Distributed IP Anycast Gateway ]
In case of VM1 is not known to PM1, PM1 would ARP for VM1. Destination TOR would Proxy
for VM1. No Silent-Host discovery problem.
VXLAN L3
Gateway
VXLAN L3
Gateway
VM1 10.10.10.10 PM1
10.10.10.20
VLAN 123
DIP:
10.10.10.20 SIP
: 10.10.10.10
DVTEP: DTOR_L0
SVTEP : STOR_L0
VNI 10000
LDM3ACF:
PMa1b_MrAiCc
SMAC: VM1_MAC
DIP: 10.10.10.20
SIP : 10.10.10.10
V M O S
SMAC: VM1_MAC V
M O S
7
DMAC: PM1_MAC
9
VLAN 123 <-> VNI 10000 PM1_MAC -> DTOR_L0, 10000
8
SIP : 10.10.10.10
DIP: 10.10.10.20
VLAN 123
SMAC:
VM1_MAC
DMAC:
PM1_MAC
10
VLAN 123 <-> VNI 10000 PM1_MAC -> eth1/23
7) VM1 generates a data packet with
PM1_MAC as destination MAC
8) TOR receives the packet and performs
Layer-2 lookup for the destination
9) TOR adds VXLAN-Header information
(Destination VTEP, VNI, etc) and
forwards the packet across the Layer-3
fabric, picking one of the equal cost
paths available via the multiple Spines
10) The destination TOR receives the
packet, strips off the VXLAN header
and performs lookup and forwarding
toward PM1
http://sola99.tistory.com
VXLAN Evolution: IP Services
Distributed IP Anycast Gateway: IP Forwarding within the 다른 Subnet aka Routing (ARP)
VM1이 GW ARP 요청을 Leaf응답 [ Distributed IP Anycast Gateway ]
VXLAN L3
Gateway
VXLAN L3
Gateway
L3 Fabric
V M O S
V M O S
SVI IP Address (VRF Blue) MAC: 0000.dead.beef
IP: 20.20.20.1
VM1 10.10.10.10
VM1 ARP Cache 20.20.20.20 -> GW_MAC
PM2 20.20.20.20
1
CPU 2
3
rib
SVI IP Address (VRF Blue) MAC: 0000.dead.beef
IP: 10.10.10.1
MAC IP L2 VNI L3 VNI
VM1_MAC 10.10.10.10 10000 50000
1) VM1 sends ARP request for
Default Gateway –10.10.10.1
2) The ARP request will be
received at TOR and punted
to the Supervisor, where MAC
and IP is learned and
distributed
3) TOR acts as regular Default
Gateway and sends ARP
response with GW_MAC to
VM1
http://sola99.tistory.com
VXLAN Evolution: IP Services
Distributed IP Anycast Gateway: IP Forwarding within the 다른 Subnet aka Routing (Data)
VM1이 데이터를 전송하면, leaf는 L3 Lookup 이후 VXLAN 을 통하여 전달 [ Distributed IP Anycast Gateway ]
4) VM1 generates a data packet
destined to PM2 IP (20.20.20.20)
with GW_MAC as destination
MAC
5) TOR receives the packet and
performs Layer-3 lookup for the
destination (known)
6) TOR adds VXLAN-Header
information (Destination VTEP,
VNI, etc) and forwards the
packet across the Layer-3 fabric,
picking one of the equal cost
paths available via the multiple
Spines
7) The destination TOR receives
the packet, strips off the VXLAN
header and performs lookup
and forwarding toward PM2
VXLAN L3
Gateway
VXLAN L3
Gateway
PM2 20.20.20.20
DVTEP: DTOR_L0
SVTEP : STOR_L0
VNI 50000
LDM3AC:FDT
aORb_MrAicC
SMAC: STOR_MAC
DIP: 20.20.20.20
SIP : 10.10.10.10
V M O S
V M O S
VM1 10.10.10.10
4
DMAC: GW_MAC
SMAC: VM1_MAC
VLAN 123
DIP: 20.20.20.20
SIP : 10.10.10.10
6
20.20.20.20 -> DTOR_L0, 50000
5
SIP : 10.10.10.10
DIP: 20.20.20.20
VLAN 321
SMAC: GW_MAC
DMAC:
PM2_MAC
7
20.20.20.20 -> PM2_MAC PM2_MAC -> eth1/32
http://sola99.tistory.com
SVI/VNI/VLAN Scoping and Provisioning
Orchestration leads to scale optimization : 규모 및 배포
일반적인 L2/L3 overlay 환경과 SDN(Cisco ACI) 환경 비교 [ 2가지 모델 ]
L3 Fabric
L3 GWY L3 GWY L3 GWY L3 GWY L3 GWY L3 GWY
L3 Fabric
L3 GWY L3 GWY L3 GWY L3 GWY L3 GWY L3 GWY
Mgmt
All VNIs/SVIs everywhere
• Umbrella catch-all provisioning
• Full ARP state on all Leaf Nodes
• Can be manually provisioned up-front
• Open to L2 Flooding everywhere
VNIs/SVIs scoped as hosts attach
• Provision on host attach/policy
• ARP state only for local subnets
• Requires orchestration
• L2 Flooding is scoped
http://sola99.tistory.com
Integration with Orchestartors and Host Attachment
호스트 생성 시 통합 관리 시스템과 연동
자동으로 호스트 발견 및 관련 정책 적용 [ Host Attach ]
Virtual Access
Leaf
Hosts
VM
OS
VM
OS
Virtual Physical
Trigger
Tunnel end-point
Overlay
Network
Manager /
Controller
vCloud
Director
API
Spine
Compute
Orchestration
Network
Orchestration
Trigger
TOR TOR
VM
OS
VM
OS
1
2
3
LDAP
4
VLAN, SVI, VRF, BGP
Compute Controller (e.g. vCloud Director) integrated
overlay provisioning
Integrates physical and virtual end-points Topology discovery with triangulation
VM 생성 이후 TOR에 이벤트발생(단말 학습: New MAC with
VLAN, VDP with VNI, VMTracker, LLDP with NIC MAC,
CLI with VNI or VLAN)이후 TOR LDAP 쿼리이후 config 다운
http://sola99.tistory.com
Multi-Data Center Connectivity
LAN Extensions and IP Mobility
독립적인 Fabric 간에 이더넷 확장, IP 트래픽은 최적 라우팅(no hair-pinning) [ LAN Extensions and IP Mobility ]
N7K/ASR
VLAN 20 VLAN 30
L3 Fabric
L3 Domain
N7K/ASR
VLAN 20 VLAN 30
L3 Fabric
Data Center 1 Data Center 2
VXLAN L3 Gateway
VXLAN L3 Gateway
VXLAN L3 Gateway
VXLAN L3 Gateway
VXLAN L2/L3 Gateway
VXLAN L2/L3
Gateway
VXLAN L3 Gateway
VXLAN L3 Gateway
VXLAN L3 Gateway
VXLAN L3 Gateway
VXLAN L2/L3
Gateway
VXLAN L2/L3 Gateway LAN Extensions (OTV/EVPN)
IP Mobility (LISP)
VNI 5000
http://sola99.tistory.com
Multi-Data Center Connectivity
LAN Extensions: 데이터센터 도메인 영역 분리 : 운영 및 영향도 분리
2개의 데이터센터간 연동: 데이터센터 내 통신(VXLAN), 센터 간 통신(OVT/EVPN) [ LAN Extensions ]
N7K/ASR
VNI 5000 VLAN 20 VLAN 30
L3 Fabric
N7K/ASR
VLAN 20 VLAN 30
L3 Fabric
DC1 DC2
VXLAN L3 Gateway
VXLAN L3 Gateway
VXLAN L3 Gateway
VXLAN L3 Gateway
L2 Gateway L2 Gateway
VXLAN L3 Gateway
VXLAN L3 Gateway
VXLAN L3 Gateway
VXLAN L3 Gateway
L2 Gateway L2 Gateway
L3 Domain
VXLAN ✓OTV/EVPN VXLAN
VXLAN
Domain Boundary:
Failure and Event Containment Clear Administrative Delineation
✗
http://sola99.tistory.com
Multi-Data Center Connectivity
Scoped Control Plane
데이터센터 내(iBGP 통한 광고/전파), 데이터센터 간(eBGP를 통한 광고/전파) [ Scoped Control Plane ]
L3 Domain
VXLAN
L3 Gatewa y
VXLAN
L3 Gatewa y
VXLAN
L3 Gatewa y
VXLAN
L3 Gatewa y
VXLAN
L3 Gatewa y
VXLAN
L3 Gatewa y
DataCenter #1 DataCenter #2
eBGP
eBGP iBGP iBGP
PM2
20.20.20.20
PM1
10.10.10.10
MAC IP VTEP L2 VNI L3 VNI
PM1_MAC 10.10.10.10 STOR_LO 10000 20000
PM2_MAC 20.20.20.20 DTOR_LO 30000 20000
http://sola99.tistory.com
Multi-Data Center Connectivity
End-to-end Data Plane
데이터센터 간 트래픽 전달 [ End-to-end Data Plane ]
L3 Domain
2
VXLAN 20.20.20.20 -> DTOR_L0, 20000 L3
Gatewa y
VXLAN L3
Gatewa y
VXLAN L3
Gatewa y
VXLAN L3
Gatewa y
20.20.20.20 -> PMV2X_LAMNAC L3
PM2_MAC -> eGtha1te/w3a2
y
VXLAN L3
Gatewa y
DVTEP: DTOR_L0 SVTEP :
STOR_L0 VNI 20000 DMAC:
DTOR_MAC SMAC:
STOR_MAC DIP: 20.20.20.20
SIP : 10.10.10.10
DataCenter #1
DataCenter #2
eBGP
VXLAN VNI 10’000 VXLAN VNI 20’000 VXLAN VNI 30’000
PM2 20.20.20.20
PM1 10.10.10.10
1
DMAC:
GW_MAC
SMAC:
PM1_MAC VLAN
123 DIP:
20.20.20.20 SIP
: 10.10.10.10
3
SIP :
10.10.10.10
DIP:
20.20.20.20
VLAN 321
SMAC:
GW_MAC
DMAC:
PM2_MAC
4