BRKDCT-2840 Data Center Networking Taking Risk Away From Layer 2 Interconnects

8/4/2019 BRKDCT-2840 Data Center Networking Taking Risk Away From Layer 2 Interconnects

1/114

BRKDCT-2840

Minimizing the Risks With EnterpriseMulti-Site Data Center L2 Connectivity

David Jansen CCIE 5952Technical Solutions Architect Data [email protected]


2/114

2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2840 2

Reference Sessions BRKDCT-2011 - Design and Deployment of Data Center

Interconnects using (Advanced) A-VPLS, Amit Singh.

BRKDCT-2048 - Deploying Virtual Port Channel in NXOS,Francis Guillier.

BRKDCT-2049 - Introduction to Overlay Transport Virtualization:Extending the Data Center Layer 2 Connectivity, Natale Ruello.

BRKDCT-2081 - Cisco FabricPath Technology and Design, Tim

Stevenson.

BRKSAN-2704 - Storage Area Network Extension Design andOperation, Mark Allen.

BRKDCT-3060 - Deployment Challenges with Interconnecting DataCenters, Max Ardica & Patrice Bellagamba.

BRKDCT-3103 - Advanced OTV - Configure, Verify and TroubleshootOTV in Your Network, Bhanu Vemula.

BRKCRS-3045 LISP, Dino Farinacci, & Greg Schudel.

BRKDCT-9131 - Mobility and Virtualization in the Data Center with

LISP and OTV, Victor Moreno.


3/114


Session BRKDCT-2840 Abstract

Data Center Networking: Taking Risk Away from Layer 2

InterconnectsThis intermediate session details a solution for providing a means of Layer 2communications adjacency to support operating system clustering, file systemclustering, virtual machine mobility, symmetric traffic flows, and more in a highlyresilient multisite data center infrastructure. Starting from the building blocks ofspanning-tree implementations and considerations, the session continues with details

on how to control the Layer 2 control and data planes to limit negative effects presenttoday in geographically diverse Layer 2 domains. The emphasis is on multisite datacenter interconnect and specifics of service advertisement and site failover.Considerations are given for tying users to either site in an active/standby,active/active per application, and active/active within an application relationship.Transport mechanisms such as tag switching, Ethernet over MPLS, Virtual PrivateLAN Service, MPLSoGRE, OTV, Virtual Ethernet, ServerFarm to User First HopRedundancy, User to ServerFarm redundancy with Route Health Injection, 802.1s andw, load sharing multisite traffic on intra-data center VLANs, global site load balancing,and others. This session compares alternatives with direct Layer 2 links on dedicatedservices or DWDM lambdas, point-to-point and multipoint scenarios, configurationsusing existing RPVST or MST deployments within a data center site, sharing Layer 2and Layer 3 services, and operations and administration considerations.


4/114


Goals of This Session

Present alternatives for interconnecting multipleData Center locations

Present tested methods in production forminimizing the risks associated with meeting theseconnectivity requirements.


5/114


Session Agenda

Data Center Interconnection Common Scenariosand Terms

Dark Fiber / DWDM Solutions

Label Based Solutions

IP Based Solutions

Encryption

Recommended Designs for Optimizing Traffic Flows

Q & A


6/114

Data Center InterconnectionCommon Scenarios and Terms

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

6


7/114 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2840 7 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

7

Data Center InterconnectionCommon Scenarios and Terms


8/114 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2840 9

Layer 2 Use Cases

Extending Operating System / File System clusters

Extending Database clusters

Virtual machine mobility

Physical machine mobility

Physical to Virtual (PtoV) Migrations

Legacy devices/apps with embedded IP addressing

Time to deployment and operational reasons

Extend DC to solve power/heat/space limitations

Data Center co-location



Layer 2 Risks

Flooding of packets between data centers

Spanning Tree (STP) is not easily scalable and riskgrows as diameter grows

STP has no domain isolation issue in single DC

can propagate First hop resolution and inbound service selection

can cause verbose inter-data center traffic

In general Cisco recommends L3 routing for

geographically diverse locations

This session focuses on making limited L2connectivity as stable as possible



Layer 2 Solution Types

Light customer owned fiber to build an extended L2 networkNo STP isolation between sites

Virtual Switching System (VSS) / Virtual Port Channel (vPC)

FabricPath (no STP)

Purchase multiple wavelengths from SPCost rises, still nothing to offer STP isolation

Redesign data center STP domain using Multiple SpanningTree (MST) regions

STP domain concept

Fundamental change requiring large time investmentOperational differences and MST database management



Implement a L2 solution to virtualize transport over L3

EoMPLS for point to point (possible STP isolation issues)

Multipoint bridging using Virtual Private LAN Services (VPLS)

MPLSoGRE

Overlay Transport Virtualization (OTV)

Advanced VPLS (A-VPLS)

Layer 2 Solution Types (Cont)



Session Agenda




IP Based Solutions

Encryption

Recommended Designs for Optimizing Traffic Flows Q & A


13/114


1



Layer 2 Prerequisites for All Options

This session assumes a fairly detailed knowledgeof Spanning Tree Protocol

Items we leverage in this solution:

802.1w

802.1sPort Fast

BPDU Filter

BPDU Guard

Root Guard

Loop Guard

Bridge Assurance (Catalyst 6500, Nexus 5000/5500 and7000)

L 2 E t i With t



Layer 2 Extension WithoutTunnels/Tags (vPC/VSS)

6500 with Virtual Switching System cluster (Supporteddistances at 80km (ZR) Dark Fiber)

Nexus 7000 with Virtual Port-Channels (Supported distancesat 80km (ZR-X2) Dark Fiber)

All traffic flows to a vPC/VSS member node

Hub-and-spoke topology from a layer 2 perspective

Dedicated links to vPC/VSS members from each data centeraggregation switch

Can consume lambda or fiber strands quickly

Data plane rate limiting in L2 still needs protection

STP domains are not isolated unless we BPDU-filter at allvPC/VSS aggregation switches



vPC / VSS Design

vPC / VSS vPC / VSS

L2 LH Fiber/DWDM

L3 LH Fiber/DWDM

L2 Local Fiber

L3 Local Fiber

Data Center #1 Data Center #2



vPC / VSS L2 View

vPC/VSS vPC/VSS

L2 LH Fiber/DWDM

L2 Local Fiber


BPDU-FilteringBPDU-Filtering

- vPC/VSS Domain ID for facing vPC/VSS layers should be different- BPDU Filter on the edge devices to avoid BPDU propagation- STP Edge Mode to provide fast failover times- No Loop must exist outside the vPC/VSS domain- No L3 peering between Nexus 7000 devices (i.e. pure layer 2)



vPC / VSS Design

VSS/vPC

vPC / VSS vPC / VSS

VSS

12 Lambda/24 Strand Example4 Additional Lambda/8 Strands per new DCL2 Service Only from Provider


Data Center #3 L2 LH Fiber/DWDML3 LH Fiber/DWDML2 Local Fiber

L3 Local Fiber



vPC / VSS L2 View

vPC/VSS

VSS VSS

VSS

L2 LH Fiber/DWDM

L2 Local FiberAll links are port channels to Central VSS


Data Center #3

BPDU Filtering

BPDU FilteringBPDU Filtering



vPC and Layer 3


vPC

P P

vPC

L2 LH Fiber/DWDM

L3 LH Fiber/DWDM

L2 Local Fiber

L3 Local Fiber

P L3 Peer

Nexus 7000 configured for L2 Transport only SVI passive-interface (no IGP peering)


21/114


vPC and Layer 3


vPC

P P

vPC

Peering over a vPC inter-connection on parallel routed interfaces SVI passive-interface (no IGP peering)

P P

L2 LH Fiber/DWDM

L3 LH Fiber/DWDM

L2 Local Fiber

L3 Local Fiber

P L3 Peer

F b i P th D i (P ti l/F ll/Ri T l )


22/114


FabricPath Design (Partial/Full/Ring Topology)

Data Center #1

Data Center #2

FabricPathCore

Agg w/vPC+

FabricPath

Data Center #3

ClassicEthernet

Leverage vPC+

Brownfield / Greenfield DC

STP Integration

Conversational MACLearning

Native VLAN Pruning

TTL / RPF

ECMP for L2

STP (CE)

FabricPath


23/114


Session Agenda




IP Based Solutions

Encryption

Recommended Designs for Optimizing Traffic Flows Q & A


24/114

MPLS Solutions

2


25/114


EoMPLS (Ethernet Over MPLS)

Encapsulates Ethernet frames inside MPLS packets to passlayer 3 network

EoMPLS has routing separation from metro core devicesproviding connectivity CE flapping routes wont propagate

inside MPLS Point to point links between locations


PE PECE CE

EoMPLS Is a Pseudo-Wire

MPLS


26/114


Virtual Private LAN Service (VPLS)

VPLS defines an architecture that allows MPLS networks tooffer Layer 2 multipoint Ethernet Services

Metro Core emulates an IEEE Ethernet bridge (virtual)

Virtual Bridges linked with EoMPLS Pseudo Wires


PE PECE CE

VPLS Multipoint Services

CE

MPLS

VFI

VFI

VFI


27/114


Virtual Forwarding Instance (VFI)

IOS Representation of Virtual Switch Interface

Flooding / Forwarding

MAC table instances per customer (port/VLAN) for each PE

VFI will participate in learning and forwarding process

Associate ports to MAC, flood unknowns to all other ports

Address Learning / Aging

LDP enhanced with additional MAC List TLV (label withdrawal)

MAC timers refreshed with incoming frames

Loop Prevention

Create full-mesh of Pseudo Wire VCs (EoMPLS)

Unidirectional LSP carries VCs between pair of N-PE Per

VPLS Uses split horizon concepts to prevent loops


28/114


Calculating Core MTU Requirements

Core MTU Edge MTU + Transport Header +(MPLS Label Stack * MPLS Header Size)

Edge MTU is the MTU configured in the CE-facingPE interface

Examples (all in Bytes):

1526

1522

Total

421500EoMPLS VLAN Mode

421500EoMPLS Port Mode

MPLSHeader

MPLSStack

Edge

18

14

Transport


29/114


End to End VPLS and EoMPLS Design

Access Access

Agg Agg

DC Core DC Core

Layer 3 CoreIntranet

L2 Links (GE or 10GE)


Server Farm Server Farm

VPLS / EoMPLSDomain

Loss of Link/Node

Po1

Po1

WAgg1

WAgg2

WCore1 WCore2 ECore2ECore1

EAgg1

EAgg2

WMC1

WMC2

EMC1

EMC2


30/114


Access to Aggregation Connections

Rapid-PVST is existing protocol,and no desire to force a change

Aggregation switches are root for allintra-DC VLANs

Aggregation ARP and CAM Timers

The peer aggregation switch issecondary root

HSRP tested for first hopredundancy from server (more later) Server Farm

Agg

Access


31/114


Layer 3 Aggregation andCore Connections

Layer 3connections from DCCore to Enterprise Core

Aggregation switch L3

connected to DC Core Hanging L3 links in

diagram, are to MetroCore switches which areEthernet over MPLS links

Hanging L3 links are forpeering the DC Cores ineach location in a point-to-point scenario

DC Core

Layer 3Enterprise Core

Agg

Bidirectional forwarding detection (BFD)interval 100 min_rx 100 multiplier 3

If dual supervisor modules, need non-stop forwarding (NSF)

under routing process


32/114


EoMPLS / VPLS Infrastructure

Loopbacks chosen as peering points for

EoMPLS and VPLS xconnects Horizontal links represent 10GE on DWDM

service between data centers (alternatepaths)

Vertical links represent intra-DC 10GE

connections

MPLS LDP enabled globally (not a full P / PEMPLS implementation)

LDP NSF/SSO mpls ldp graceful-restart

Links to/from aggregation switches for Layer2, are storm-control limited for broadcastsand multicasts to 1% (protect data plane)

MTU increased to 1522 bytes on the L3MPLS links for the MPLS tagging

Metro Core Metro Core

VPLS / EoMPLSDomain


33/114


Metro Switch Interconnectivity


L3 Links (10GE)

IGP Routing ProcessconnectingMPLS PEs

- Link debounce timers- Aggressive-UDLD- Carrier-delay timers

- Link debounce timers- Aggressive-UDLD- Carrier-delay timers


34/114


EoMPLS for Layer3

Access Access

Agg Agg






DC Core DC Core

EoMPLS

METRO CORE

PW Pseudo Wires


35/114


DC Core DC Core

VPLS for Layer2

Access Access






Agg Agg

Metro Core

METRO CORE

PW Pseudo Wires

VFI


36/114


DC Core DC Core

VPLS for Layer2

Access Access






Agg Agg

Metro Core

METRO CORE

PW Pseudo Wires

l2 vfi vlan3700 manual

vpn id 3700neighbor 192.168.255.250 encapsulation mplsneighbor 192.168.255.251 encapsulation mplsneighbor 192.168.255.253 encapsulation mpls

l2 vfi vlan3700 manualvpn id 3700neighbor 192.168.255.250 encapsulation mplsneighbor 192.168.255.252 encapsulation mplsneighbor 192.168.255.253 encapsulation mpls

l2 vfi vlan3700 manualvpn id 3700neighbor 192.168.255.250 encapsulation mplsneighbor 192.168.255.251 encapsulation mplsneighbor 192.168.255.252 encapsulation mpls

l2 vfi vlan3700 manual

vpn id 3700neighbor 192.168.255.251 encapsulation mplsneighbor 192.168.255.252 encapsulation mplsneighbor 192.168.255.253 encapsulation mpls


37/114


DC Core DC Core

VPLS for Layer2

Access Access






Agg Agg

Metro Core

METRO CORE

PW Pseudo Wires

interface Vlan3700no ip addressload-interval 30xconnect vfi vlan3700




VLAN3700


38/114


Spanning Tree

Spanning-Tree BPDUs will NOT traverse betweenthe Data Centers It isnt needed (and blocked)with VPLS

We still need to control data plane layer 2 events(i.e., limit the traffic)

Since enterprises want dual N-PE devices, andVPLS blocks BPDUs, we require method to blockwithin a local DC

End-to-End L2 View


39/114


Access

Agg

DC Core

Server Farm

End-to-End L2 View

Without layer 2 link between Metro Switches there is aloop. Each side has a U shape with Metro and Aggswitches, broadcast storms.

Access

Agg

DC Core



L3 Links (GE or 10GE)Server Farm

VPLS / EoMPLSDomain

RSTPRSTP

X X XX


Broadcast, Multicast,Unknown Unicast


40/114


Access

Agg

DC Core

Server Farm

Spanning Tree Option: MSToNPE

Access

Agg

DC Core




Server Farm

VPLS / EoMPLSDomain

RSTP RSTP

MST MST

Single L2MST BridgeSingle L2MST Bridge

Root Bridge in West DCfor all VLANs that GoBetween Data Centers

Root Bridge in East DCfor all VLANs that Go

Between Data CentersLayer 3 CoreIntranet

X X


41/114


Spanning-Tree

MST (802.1s) represents Metro Cores as single bridge

Blue Layer 2 link is access port channel with a VLANthat represents the MST0 instance to make the MSTgroup

MST bridge priority set to 0 (Metro Core will be rootof Inter-DC VLANs)

Spanning tree root-guard enabled on Metro Corestoward aggregation switches (protects in case theblue MST link fails)

Only inter-DC VLANs allowed on trunks to/fromaggregation switches

Set spanning-tree VLAN cost to set the priorities on theagg switches links to metro core will allow us to putsome VLANs on upper Metro Core, some on lower bydefault

Single L2MST Bridge


42/114


Access

Agg

DC Core

Server Farm

Spanning Tree Option: MSToNPE

Access

Agg

DC Core



Server Farm

VPLS / EoMPLSDomain

RSTPRSTP

MST MST

Single L2MST BridgeSingle L2MST Bridge

XXX X

X X

XX

X

XX

X

interface Port-channel4description Port Channel to WestMetroCore2spanning-tree vlan 3700,3704,3712,3716 cost 8

interface Port-channel4

description Port Channel to WestMetroCore1spanning-tree vlan 3702,3706,3710,3714,3718 cost 8

STP Option: Multi Chassis Link Aggregation Group (MC LAG)


43/114


Access

vPC

DC Core

Server Farm

STP Option: Multi-Chassis Link Aggregation Group (MC-LAG)

Access

vPC

DC Core




Server Farm

VPLS / EoMPLSDomain

RSTP

Root Bridge in East DC forall VLANs that Go

Between Data Centers


RSTP

Root Bridge in West DC forall VLANs that Go Between

Data Centers

ICCPICCP


44/114



Leverages VSS MEC for DCI

L2/L3/L4 Flow Based Balancing

Simplified Edge Redundancy

Optimal Bandwidth Utilization

PFC on SUP720 treats as a normal Ethernet port

Flexibility to trunk VLANs over either an MPLS or IPtransport easily

A new interface type: interface virtual-ethernet x Takes switchport commands just like a normal

physical Ethernet port


45/114



Integration with existing VPLS solutions

MPLS Fast Re-Route (FRR) for very fast failover

MPLS Traffic Engineering (TE)

Requires SIP-400 / ES40+ (12.2.33SXJ1) 10GE

IOS Version 12.2.33SXI4

Sub-1 second fail-over

4,000 VLANs

32 Sites

Unified Control-Plane (Single nPE Per Location)


46/114



VSS is recommended but not required. If VSS isused then the modules need to be compatible withVSS. Ie. 67xx modules.

Scalability is 32k VCs; the number of VCs equals

the number of neighbors * number of VLANs The solution supports MPLS L3 VPNs at the same

time; MPLS L3 VPNs can exist side by side on thesame PEs to provide a complete solution.


47/114


Leveraging VSS for Dual-Homing

nPE

Agg

Agg

nPE

VSS system

Agg

Agg

IP/MPLS Cloud

AggAgg

VSL VSL

VSS system

Leveraging VSS at the DCI edge provides nPE redundancyUse of VSS is transparent to the VPLS cloudEquivalent to having the sites single attached (single virtual PE)


48/114


The Label Setup Example

nPE

Agg

Agg

nPE

Agg

Agg

AggAgg

VSL VSL

One Tunnel Label Per ECMP Exit

OSPF

Loop0:1.1.1.1 Loop0:2.2.2.2


49/114


The Label Setup Example

nPE

Agg

Agg

nPE

Agg

Agg

AggAgg

VSL VSL

Loop0:1.1.1.1 Loop0:2.2.2.2

Targeted LDPPW Lbl2 PW Lbl2VLAN20 VLAN20

PW Lbl1VLAN10 PW Lbl1 VLAN10

Single tLDP per neighbor


50/114


Multi-Pathing with A-VPLS

nPE

Agg

Agg

nPE

VSS system

Agg

Agg

IP/MPLS Cloud

AggAgg

VSL VSL

VSS system

Up to 8 equal cost paths between any two sites

A label is assigned to each equal cost path based on routing reachability of neighbor

Simplified CLI: Virtual Ethernet interface

Loadbalancing at L2/L3/L4

LSP/GRETunnel

A-VPLS PseudowireSingle Virtual Ethernet Interface across Multiple Interfaces


51/114


A-VPLS Solution

nPE

Agg

Agg

nPE

VSS system

Agg

Agg

AggAgg

VSL VSL

VSS system

L2/L3/L4 LBbetween all sites

Want to add a 3rd site?

VSL

Split horizon betweenall neighbors for loopavoidance, multipoint support.

C S


52/114


Configuration A-VPLS

pseudowire-class cl1

encap mpls

! enable ML PW (ECMP LB)load-balance flow

! enable FAT PWflow-label enable

interface virtual-ethernet 1

transport vpls meshneighbor 2.2.2.2 pw-class cl1neighbor 3.3.3.3 pw-class cl1switchportswitchport mode trunkswitchport trunk allowed vlan 10, 20

interface TenGigabitEthernet1/1/3/0ip address 10.1.1.1 255.255.255.0mpls ip

Egress physical interface:

IP/MPLS

PE1 (1.1.1.1)

PE2 (2.2.2.2) PE3 (3.3.3.3)

S S S


53/114


End to End VPLS and EoMPLS Design A-VPLS

Access Access

Agg Agg

DC Core DC Core





VPLS / EoMPLSDomain

Loss of Link/Node

Po1

Po1

WAgg1

WAgg2


EAgg1

EAgg2

WMC1

WMC2

EMC1

EMC2

A VPLS R d/IRB PW


54/114


A-VPLS Routed/IRB PW

Access Access

Agg Agg

DC Core DC Core

MPLS Cloud


L3 Links (GE or 10GE)Server Farm Server Farm

Loss of Link/Node

Ten3/0/0

Ten4/0/0 Ten4/0/0

Ten4/0/0

Po1

VSS

WAgg2


EAgg1

VSS

A-VPLS Virtual Ethernet Configuration

A-VPLS with Integrated Routing and Bridging L2 Boundary does not extend beyond Aggregation layer

SIP-400 or ES40+Core Interfaces

VSL VSL

St C t l


55/114


Storm Control

Traffic storms when packets flood the LAN

Traffic storm control feature prevents LAN ports frombeing disrupted by broadcast or multicast flooding

Rate limiting for unknown unicast (UU) must be handledat Data Center aggregation; unknown unicast flood rate-limiting (UUFRL):

mls rate-limit layer2 unknown rate-in-pps [burst-size]

Storm Control is configured as a percentage of the linkthat storm traffic is allowed to use.

storm-control broadcast level 1.00 (% of b/w may vary need tobaseline)

storm-control multicast level 1.00 (% of b/w may vary need tobaseline)

3 M D t C t L ti


56/114


3 or More Data Center Locations

EoMPLS will allow multiple point to point links

between any 2 sites

Can build a full mesh of links to interconnect layer 3devices

VPLS scales by adding peer xconnects under theVFI in the IOS configuration

Split horizon with MST local to data center willmake for simple growth

Limits dependant on amounts of L2 traffic especiallymulticast, as these are replicated on each PW

3 Site Drawing With EoMPLS PWs for L3


57/114


3 Site Drawing With EoMPLS PWs for L3


Server Farm



3 Site Drawing With VPLS PWs for L2


58/114


3 Site Drawing With VPLS PWs for L2


Server Farm



S f T i S ti


59/114


Summary of Tagging Section

EoMPLS well suited for Router-Router links

VPLS well suited for Switch-Switch links

Straightforward to scale to multiple Data Centerlocations

MST and MC-LAG both work well

One tradeoff is QinQ support against number of VLANs to pass

Another is the root of the spanning tree for inter-DC VLANs

A-VPLS

Backwards CompatibleLoad Balancing Enhancements

Simplified Configuration

Single virtual nPE

S i Ag d


60/114


Session Agenda

Data Center Interconnection Common Scenarios

and Terms



IP Based Solutions

Encryption


Q & A


61/114

IP Based Solutions

6

EoMPLS/VPLSoGRE Reason for oGRE


62/114


EoMPLS/VPLSoGRE Reason for oGRE

IP Only Core

Need a solution to stand up VC with a LDP label

GRE provides routing separation from metro coredevices providing connectivity Customer Edge (CE)flapping routes wont propagate inside IP network

Point to point links between locations

Wide range of hardware support including 6500, 7600,ASR

IPSec securing of tunnel straightforward Data plane rate limiting in L2 still needs protection

* Please note the 7600 does not support VPLSoGRE

What Is EoMPLS and VPLS Over GRE?


63/114


What Is EoMPLS and VPLS Over GRE?

EoMPLS connectivity over IP-onlynetwork.

EoMPLS VCs are established overMPLSoGRE Tunnels

Requires SIP-400 on the 6500 withSUP720

VPLS connectivity over IP-onlynetwork.

VPLS VCs are established overMPLSoGRE Tunnels.

Requires SIP-400 on the 6500 withSUP720

MPLSoGRETunnels

PEPE

EoMPLSinstance EoMPLS

instance

MPLSoGRETunnels

PEPE

PE

VPLSinstance VPLS

instance

VPLSinstance

IP GRE Tunnels that provide MPLS connectivity overIP-only network.

MPLS LDP session is established through the GRE tunnel

Layer 2 Extension EoMPLSoGRE


64/114


yCatalyst 6500

L2

L3

SiSiSiSi

MCEC with Nexus 7000 vPC

nPE nPE

Aggregation

Access VSL SiSi

VSL

MEC

SiSi

L2

L3

SiSiSiSi

SiSi

L2 Etherchannelas VSS is viewedas one device

nPE nPE

Aggregation

Access

Backup EoMPLS Pseudo-wireinto Core

Per VLANalternate path

SiSi

Per VLANVC/GRE



Layer 2 Extension EoMPLSoGRE - Catalyst 6500


65/114


y y

Int vlan 10

Xconnect 11.11.11.2 10 encapsulation mpls

mtu 9216

interface Loopback0description tunnel sourceip address 10.10.10.1 255.255.255.0

interface Loopback1

description LDP Router IDip address 11.11.11.1 255.255.255.255

interface Loopback0description tunnel source

ip address 10.10.10.2 255.255.255.0

interface Loopback1

description LDP Router IDip address 11.11.11.2 255.255.255.255

Interface Tunnel 10ip address 192.168.10.1 255.255.255.0tunnel-source 10.10.10.1tunnel-destination 10.10.10.2mpls ip

ip route 11.11.11.2 255.255.255.255 Tunnel 10

Interface gig 1/0SwitchportSwitchportmode accessSwitchportaccess vlan10mtu 9216

interface GigabitEthernet3/0/1

description SIP-400 Interfacemtu 9216ip address 192.168.33.3 255.255.255.0bfd interval 100 min_rx 100 multiplier 3

!

Int vlan 10

Xconnect 11.11.11.1 10 encapsulation mpls

mtu 9216


ip route 11.11.11.1 255.255.255.255 Tunnel 10

Interface gig 1/0SwitchportSwitchportmode accessSwitchportaccess vlan10mtu 9216

interface GigabitEthernet3/0/1

description SIP-400 Interfacemtu 9216ip address 192.168.33.4 255.255.255.0bfd interval 100 min_rx 100 multiplier 3

Layer 2 Extension VPLSoGRE


66/114


yCatalyst 6500

VSL

L2

L3

SiSiSiSi

L2 Etherchannelas VSS is viewedas one Device

nPE nPE

Aggregation

VSLSiSi

VSL

MEC

SiSi

L2

L3

SiSiSiSi

SiSi

nPE nPE

AccessSiSi

L2

L3

SiSiSiSinPE nPE

Aggregation

Access


Per VLANalternate path

Access

Per VLANVFI/GRE

SiSiSiSi



Aggregation


Layer 2 Extension VPLSoGRE - Catalyst 6500


67/114


l2 vfi vfi-vlan10

vpn id 10

neighbor11.11.11.2 encapsulation mpls

interface Vlan 10

xconnectvfi vfi-vlan10

mtu 9216


interface Loopback1description LDP Router IDip address 11.11.11.1 255.255.255.255


interface Loopback1description LDP Router IDip address 11.11.11.2 255.255.255.255


ip route 11.11.11.2 255.255.255.255 Tunnel 10

Interface gig 1/0SwitchportSwitchport mode accessSwitchport access vlan10mtu 9216

interface GigabitEthernet3/0/1description SIP-400 Interface

mtu 9216ip address 192.168.33.3 255.255.255.0bfd interval 100 min_rx 100 multiplier 3

l2 vfi vfi-vlan10

vpn id 10

neighbor11.11.11.1 encapsulation mpls

interface Vlan 10

xconnectvfi vfi-vlan10

mtu 9216


ip route 11.11.11.1 255.255.255.255 Tunnel 10

Interface gig 1/0SwitchportSwitchport mode accessSwitchport access vlan10mtu 9216

interface GigabitEthernet3/0/1description SIP-400 Interface

mtu 9216ip address 192.168.33.4 255.255.255.0bfd interval 100 min_rx 100 multiplier 3



68/114



Ethernet LAN Extension over any Network

Ethernet in IP MAC routing Multi-datacenter scalability

Simplified Configuration & Operation

Seamless overlay - no network re-design

Single touch site configuration

High Resiliency

Failure domain isolation

Seamless Multi-homing

Maximizes available bandwidth

Automated multi-pathing

Optimal multicast replication

OTV Interface Types


69/114


OTV Interface Types

Edge Device

Internal Interfaces

External Interface

Join Interface

Overlay Interface

OT

V

InternalInterfaces

CoreL2 L3

JoinInterface

Overlay

Interface

OTV Control PlaneN i hb Di d Adj F i


70/114


Neighbor Discovery and Adjacency Formation

Before any MAC address can be advertised theOTV Edge Devices must:

Discover each other

Build a neighbor relationship with each other

The neighbor relationship can be built over atransport infrastructure, that can be:

multicast-enabled

unicast-only

Technology Benefit: OTV can leverage anynetworking capability provided by the transportinfrastructure (multicast, fast-reroute, ECMP)

OTV Control PlaneN i hb Di ( M l i T )


71/114


Neighbor Discovery (over Multicast Transport)

The end result

Adjacencies are maintainedover the multicast group

A single update reaches allneighbors

The mechanism

Edge Devices (EDs) join anmulticast group in the transport, asthey were hosts (no PIM on EDs)

OTV hellos and updates areencapsulated in the multicast group

West

OTVOTV Control Plane

IP AEast

OTV

OTV Control Plane

IP B

Multicast-enableTransport

OTV Control PlaneNeighbor Discovery (Unicast Only Transport)


72/114


Neighbor Discovery (Unicast-Only Transport)

The end resultNeighbor Discovery is automated bythe Adjacency Server

All signaling must be replicated foreach neighbor

Data traffic must also be replicated atthe head-end

The mechanism

Edge Devices (EDs) register with anAdjacency Server ED

EDs receive a full list of Neighbors(oNL) from the Adjacency Server

OTV hellos and updates areencapsulated in IP and unicasttoeach neighbor

West

OTVOTV Control Plane

IP AEast

OTV

OTV Control Plane

IP B

Unicast-onlyTransport

Ideal for connecting two or three sites

With a higher number of sites a multicast transport is thebest choice

AdjacencyServer Mode

OTV Data PlaneEncapsulation


73/114


20B + 8B + 14B* = 42Byteof total overhead

OTV encapsulation adds 42 Bytes to the packet IP MTU size

Outer IP Header and OTV Shim Header in addition to original L2 Headerstripped off of the .1Q header

The outer OTV shim header contains information about theoverlay (VLAN, overlay number)

The 802.1Q header is removed from the original frame and the

VLAN field copied over into the OTV shim header

6B 6B 2B 20B 8B

DMAC SMACEtherType IP Header

Payload 4B

CRCOTV Shim

802.1QDMAC SMAC

EtherType

802.1Q

Encapsulation

14B*

Original L2 Frame

L2Header

802.1Q header removed

* The 4Bytes of .1Q header havealready been removed

OTV Data Plane: Unicast


74/114


Eth 4

Eth 3

MAC TABLE

VLAN MAC IF

100 MAC 1 Eth 2

100 MAC 2 Eth 1

100 MAC 3 IP B

100 MAC 4 IP B

MAC 2

MAC 1

OTV Data Plane: Unicast

Core

MAC 4

MAC 3

OTV

ExternalIP A

ExternalIP B

West East

L2 L3 L3 L2

OTV Inter-Site Traffic

MAC Table contains

MAC addresses reachable throughIP addresses

OTV

Encap

2

Layer 2Lookup

1

3 Decap4 MAC 1 MAC 3

6

MAC TABLE

VLAN MAC IF

100 MAC 1 IP A

100 MAC 2 IP A

100 MAC 3 Eth 3

100 MAC 4 Eth 4

Eth 1

Eth 2

Layer 2

Lookup

5

MAC 1 MAC 3

IP A IP BMAC 1 MAC 3 MAC 1 MAC 3IP A IP BMAC 1 MAC 3

STP BPDU Handling


75/114


STP BPDU Handling

When STP is configured at a site, an Edge Device will send andreceive BPDUs on the internal interfaces.

An OTV Edge Device will not originate or forward BPDUs on theoverlay network.

An OTV Edge Device can become (but it is not required to) a root ofone or more spanning trees within the site.

An OTV Edge Device will take the typical action when receivingTopology Change Notification (TCNs) messages.

OTV

Core

The BPDUsstop here

Data-plane Loop PreventionAED and Broadcast/Multicast Handling

Broadcast, Multicast,Unknown Unicast


76/114


AED and Broadcast/Multicast Handling

Broadcast/M-cast packets reach all Edge Devices within a site.

The AED for the VLAN is the only Edge Device that forwards b-cast/

m-cast packets onto the overlay network

The b-cast/m-cast packet is replicated to all the Edge Devices on the overlay.

Only the AED at each remote site will forward the packet from the overlayonto the site.

Once sent into the site, the b-cast/m-cast packet is replicated per regular

switching

Core

OTV

OTV

OTV

AEDAED

OTV

Multi-HomingPer VLAN Authoritative Edge Device


77/114


Per VLAN Authoritative Edge Device

OTV provides loop-free multi-homing by electing a designatedforwarding device per site for each VLAN

This forwarder is known as the Authoritative Edge Device(AED)

The Edge Devices at the site peer with each other on the

internal interfaces to elect the AED

A hash based on the VLAN-IDand the number of edgedevices on the site is usedto elect the AED

As sites merge and/orpartition, internal peeringis updated and AEDre-election happens

OTV

OTV

AED

Internal peering forAED election

Multi-HomingAED and Broadcast/Multicast Handling


78/114


AED and Broadcast/Multicast Handling

Broadcast/M-cast packets reach all Edge Devices within a site.

The AED for the VLAN is the only Edge Device that forwards b-cast/

m-cast packets onto the overlay network

The b-cast/m-cast packet is replicated to all the Edge Devices on the overlay.

Only the AED at each remote site will forward the packet from the overlayonto the site.

Once sent into the site, the b-cast/m-cast packet is replicated per regular

switching

Core

OTV

OTV

OTV

AEDAED

Bcast

pkt

Broadcaststops here

Broadcaststops here

OTV

Multi-HomingAED and Unicast Forwarding


79/114


AED and Unicast Forwarding

One AED is elected for each VLAN on each site

Different AEDs can be elected for each VLAN to balance traffic load

Only the AED forwards unicast traffic to and from the overlay

Only the AED advertises MAC addresses for any given site/VLAN

Unicast routes will point to the AED on the corresponding remotesite/VLAN

Core

OTV

OTV

OTV

OTV

AEDAED

AEDAED

MAC TABLE

VLAN MAC IF

100 MAC 1 IP A

201 MAC 2 IP B IP A

IP B

OTV Use Case


80/114


Two Sites Connected With Dark-Fiber

ConfigurationOTV over a Multicast Transport


81/114


West

OTV

OTV over a Multicast Transport

Minimal configuration required to get OTV up and

running

IP A IP B

IP C

East

South

OTV

OTV

feature otv

otv site-vlan 600

interface Overlay1

description WEST-DCotv join-interface e1/1

otv control-group 239.1.1.1

otv data-group 232.192.1.0/24

otv extend-vlan 100-150

feature otv

otv site-vlan 602

interface Overlay1

description EAST-DCotv join-interface e1/1.10

otv control-group 239.1.1.1


otv extend-vlan 100-150feature otv

otv site-vlan 601

interface Overlay1

description SOUTH-DC

otv join-interface Po16otv control-group 239.1.1.1



ConfigurationOTV over an unicast-only transport


82/114


West

OTV

OTV over an unicast only transport

Establishing a DCI has never been this simple

IP A IP B

IP C

East

South

OTV

OTV

feature otv

otv site-vlan 600

interface Overlay1

description WEST-DCotv join-interface e1/1

otv adjacency-server local


feature otv

otv site-vlan 602

interface Overlay1

description EAST-DCotv join-interface e1/1.10

otv adjacency-server 10.1.1.1


feature otv

otv site-vlan 601

interface Overlay1

description SOUTH-DC

otv join-interface Po16otv adjacency-server 10.1.1.1


Localized HSRP


83/114


Localized HSRPip access-list ALL_IPs10 permit ip any any

mac access-list ALL_MACs10 permit any any

ip access-list HSRP_IP10 permit udp any 224.0.0.2/32 eq 198520 permit udp any 224.0.0.102/32 eq 1985

mac access-list HSRP_VMAC10 permit 0000.0c07.ac00 0000.0000.00ff any20 permit 0000.0c9f.f000 0000.0000.0fff any

vlan access-map HSRP_Localization 10match mac address HSRP_VMACmatch ip address HSRP_IPaction drop

vlan access-map HSRP_Localization 20match mac address ALL_MACsmatch ip address ALL_IPsaction forward

vlan filter HSRP_Localization vlan-list 100-104,1100,1200,1300

mac-list OTV_HSRP_VMAC_deny seq 10 deny 0000.0c07.ac00 ffff.ffff.ff00mac-list OTV_HSRP_VMAC_deny seq 11 deny 0000.0c9f.f000 ffff.ffff.f000mac-list OTV_HSRP_VMAC_deny seq 20 permit 0000.0000.0000 0000.0000.0000

route-map OTV_HSRP_filter permit 10match mac-list OTV_HSRP_VMAC_deny

otv-isis defaultvpn Overlay0redistribute filter route-map OTV_HSRP_filter

otv site-vlan 601

OTV Summary


84/114


OTV Summary

STP Isolation: BPDUs are not forwarded over theoverlay

Multi-homing support

Optimal Multicast Replication

Control-plane MAC based learning and forwarding

Simplified Configuration

IP Based / Transport Agnostic


85/114


Calculating Core MTU Requirements

Edge MTU is the MTU configured in the CE-facing PEinterface

Examples (all in Bytes):

1532

1528

Total

241500MPLSoGRE PE to P

241500MPLSoGRE PE to PE

GREHeader

Edge

8 (2labels)

4 (1label)

MPLSLabel

PWoGRE PE to PE* (vLAN) 1554241500 30* 6 -srcmacaddr6 -dstmacaddr4 -VLAN information2 -Type field4 -Control word4 -VC label4 -Tunnel label

PWoGRE PE to PE* (port) 1550241500 26

OTV 1542421500 n/a


86/114


Session Agenda



Label Based Solutions IP Based Solutions

Encryption


Q & A


87/114

Encryption

8

Point-to-Point Encryption Solution


88/114


yp

Nexus 7000 Trustsec can be used to secure data across remote data-center ifLayer 2 and BPDU transparency is ensured (e.g. dark fiber or DWDM transport).

N7000-1 N7000-2

e1/25e1/25

802.1AE Link

Nexus 7000 Nexus 7000

55.5.5.1 55.5.5.2

DC-1 DC-2

Encryption Solution


89/114


Self-ManagedMPLS Core

yp

* Remote port shutdown (ASR Only)

N7000-1 N7000-2

e1/25e1/25

802.1AE Link

gi 0/0/3 gi 0/0/0gi 0/0/3

EoMPLS PW

gi 0/0/0

Nexus 7000 Nexus 7000

55.5.5.1 55.5.5.2

DC-1 DC-2

Nexus 7000 vPC Encryption Solution


90/114


yp

* Remote port shutdown (ASR)

DC1-Nexus7000-1

DC1-Nexus7000-2

Self-ManagedMPLS Core

DC2-Nexus7000-1

DC2-Nexus7000-2

vPC vPC

Conclusions


91/114


TrustSec SAP (Security Association Protocol)control plane is preserved through the EoMPLSpseudowire.

802.1AE connectivity can be achieved between the

two nexus 7000 through the ASR(s)/6500(s)devices with confidentiality and integrity.

Such solution can be deployed to preserve dataconfidentiality and integrity through Nexus 7000

when interconnecting remote data-centers over anEoMPLS network.

VSPA/ASR1000/ASA Solution Overview


92/114


Datacenter Interconnect with MPLSoGREoIPSec

Solution Objective

Provide a high speed Layer 2connection between two or moreDCs.. Two or more redundantlinks are used between the DCs.

VSPA Performance

Three VSPAs can drive a 10 GElink with IMIX traffic. Single

chassis can encrypt three 10 GElinks at IMIX rates.

ASR-1000 Performance

ASR1000-ESP5-1.8Gbps IPSec

ASR1000-ESP10-4Gbps IPSec

ASR1000-ESP20-8Gbps IPSec

ASR1006-2/ESP20-16GbpsIPSec

ASR1006-2/ESP40 25.8GbpsIPSec

ASA-5585-X Performance

IPSec 5Gbps

Leverage ECMP to load balance flows over multiple

GRE/IPSec Duplicate tunnels per VSPA allow redundant 10GE

links to be provisioned

Inherent crypto engine HA: Traffic will rebalance inthe event of a VSPA outage

DC 1 DC 2MPLSoGREoIPSec

S i A d


93/114


Session Agenda




IP Based Solutions

Encryption


Q & A


94/114

Flow Optimization and SymmetrySite Selection and Inbound Flows

First Hop Outbound

Optimizing Traffic Patterns and HA Design


95/114


Many tradeoffs in understanding flows in multi-DC design

Slides that follow are a specific recommendation that meets the following

requirements:

Minimize inter-DC traffic to maintenance/failure scenarios

Ability to extend clusters between locations (OS, FS, DB, VMware DRS, etc.)

Desire to keep flows symmetric in/out of a location for DC services (FW, LB, IPS, WAAS, etc.)

Site failure will allow failover, with IP mobility to resolve caching issues

Single points of failure in gear wont cause site failover

Indicate a location preference for a service to the Layer 3 network

If broadcast storm in DC, limit impacts to other DCs

If DCI Layer 2 adjacency fails

Ability to connect to services in both DC locations (active/active per application)

DNS to round-robin clients to DC

Allow backup server farms with same service VIP (for backup connections on site fail)

Localized HSRP (egress)

Inbound traffic draw via LISP (ingress)

This is a solution in production at some customers

Sample Cluster Service Normally in Left DCD f l G Sh d B Si


96/114


Cluster Node A

Layer3 Core

Cluster Node B

VLAN A VLAN A

Cluster VLAN D (L2 Only)

10.1.1.1 HSRP Group 1Priority 140 and 130


Default Gateway Shared Between Sites

Cluster VLAN C (L2 Only)

-Cluster VIP = 10.1.1.100-Default GW = 10.1.1.1

-Cluster VIP = 10.1.1.100 Preempt-Default GW = 10.1.1.1

10.1.1.0/24 advertised into L3Backup should main site go down

10.1.1.0/25 & 10.1.1.128/25 advertised into L3-EEM or RHI can be used to get very granular

Active/Standby Pairs:FWIPSNLBSSLWAN Accel

Active/Standby Pairs:FWIPSNLBSSLWAN Accel

Data Center 1 Data Center 2



Layer3 Core

Sample Cluster Broadcast Storm in Left DCB d M l i U k U i


97/114


Cluster Node A Cluster Node B

VLAN A VLAN A




Broadcast, Multicast, Unknown Unicast







Layer3 Core

Sample Cluster L2 Interconnect FailureB d t M lti t U k U i t


98/114


Cluster Node A

Layer3 Core

Cluster Node B

VLAN A VLAN A




Broadcast, Multicast, Unknown Unicast







Layer3 Core

Active/Active per Application (VIP at Either)


99/114


Cluster Node A

Layer3 Core

Cluster Node B

VLAN A VLAN A














DNS:www-hr.acme.com -> 10.1.1.100www-news.acme.com -> 10.1.2.100


Layer3 Core

Active/Active per Application (VIP at Both)


100/114


Cluster Node A

Layer3 Core

Cluster Node B

VLAN A VLAN A















DNS:www-hr.acme.com -> 10.1.1.100

10.1.2.100


Layer3 Core

Primary Service in Left DC DR/SRMMovement of VM announced via VCenter


101/114


144.254.1.100

VM= 10.1.1.100

Default GW = 10.1.1.1

VLAN A

Public Network

144.254.200.100

Access

Agg

Access

AggSNAT

SNAT

Layer3 Core

144.254.1.100144.254.200.100

MAC movedChange the IP@

144.254.1.0/24 is

advertised into L3

Stateful Firewall Services


102/114


VLAN A 10.1.1.x


VLAN A 10.1.1.x

VLAN B - Outside

VLAN C - Inside

ESX Node A ESX Node B

VLAN B - Outside

VLAN C - Inside

Layer3 Core

Localized First Hop


103/114


ESX Node A

Layer3 Core

ESX Node B

VLAN A 10.1.1.x




VLAN A 10.1.1.x1) Filter HSRP Message2) Filter vMAC

-VM IP Address = 10.1.1.100-VM Default GW = 10.1.1.1

Layer3 Core

Locator/ID Separation Protocol (LISP) and L2 Extension Workload Mobility

Client in LISP Site Client in non-LISP Site

C1 C2


104/114


ESX Server A

Layer3 Core

ESX Server B

VLAN A 10.1.1.0

FHRP: 10.1.1.1 FHRP: 10.1.1.1

VLAN A 10.1.1.0

L3 Router LISP Router or infrastructure device

A AB B

MS

MR PxTR

D E

OTV Server-to-Server L2 traffic

LISP: L3 Client-to-Server

Optimize L3 Routing providing granular location information

Optimized mobility within or across subnets

Scale the network so host routes are in mapping database

L2 Server-to-Server

Optimize LAN Extensions

Enable dispersion of app clusters

App discovery based on MAC level broadcast and link-localmulticast

General application communication may require L2 connectivity

-Virtual-Machine-A-IP Address = 10.1.1.100-Mask: 255.255.255.0-Default GW = 10.1.1.1

-Virtual-Machine-A-IP Address = 10.1.1.100-Mask: 255.255.255.0-Default GW = 10.1.1.1

Layer3 Core

Routing Based Ingress OptimizationLISP


105/114


Access

Agg

VM= 10.10.10.1

Default GW = 10.10.10.100

ISP AISP B

Access

Agg

Data Center 1

LAN Extension

Prefix(EID)

Route Locator(RLOC)

10.10.10.1 A, B

10.10.10.2 A, B

10.10.10.5 C, D

10.10.10.6 C, D

Ingress Tunnel

Router (ITR)

Moved to C, D

Decap

3

IP_DA = 10.10.10.1

1

ETR

LISP

A B C D

IP_DA = BIP_DA = 10.10.10.1

IP_DA = 10.10.10.1

4

5Decap

7

IP_DA = CIP_DA = 10.10.10.1

6Encap

2

Data Center 2

ETR

VM= 10.10.10.1

Default GW = 10.10.10.100

IP_DA = 10.10.10.1

VM IP Address10.10.10.1

Session Agenda


106/114


Session Agenda




IP Based Solutions

Encryption


Q & A

Summary


107/114


Summary

Discussed different deployment options andtransport options

Tightly coupled Data Center with FabricPath

Spanning-tree isolation

Traffic Optimization Egress and Ingress Symmetry

Encryption Solutions


109/114


NX-OS and Cisco Nexus Switching (ISBN:

1587058928), by David Jansen, Ron Fuller,Kevin Corbin. Cisco Press 2010.

Interconnecting Data Centers Using VPLS(ISBN-10: 1-58705-992-4; ISBN-13: 978-1-58705-992-6), by Nash Darukhanawalla,Patrice Bellagamba . Cisco Press. 2009.

MPLS Fundamentals (ISBN: 1-58705-319-5),by Luc De Ghein, Cisco Press. 2007.

Layer 2 VPN Architectures (ISBN: 1-58705-848-0), by Wei Luo, Carlos Pignataro, AnthonyChan, Dmitry Bokotey. Cisco Press. 2005.

Cisco LAN Switching Configuration Handbook(2nd Edition) (ISBN-1587056100; ISBN-13:978-1587056109), by Steve McQuerry, DavidJansen, David Hucaby, Cisco Press. 2009.

Recommended Reading

Available Onsite at the Cisco Company Store

Recommendations


110/114


Check the Recommended Readingflyer for suggested books

Additional Information on LISP:http://www.lisp4.net

http://lisp4.cisco.com

http://www.cisco.com/go/lisp

Available Onsite at the Cisco Company Store

Complete Your OnlineSession Evaluation


111/114


Receive 25 Cisco Preferred Access points for each sessionevaluation you complete.

Give us your feedback and you could win fabulous prizes. Points arecalculated on a daily basis. Winners will be notified by email afterJuly 22nd.

Complete your session evaluation online now (open a browserthrough our wireless network to access our portal) or visit one of theInternet stations throughout the Convention Center.

Dont forget to activate your Cisco Live and Networkers Virtualaccount for access to all session materials, communities, and on-

demand and live activities throughout the year. Activate your accountat any internet station or visit www.ciscolivevirtual.com.

Session Evaluation
http://www.ciscolivevirtual.com/http://www.ciscolivevirtual.com/


112/114

Visit the Cisco Store for RelatedTitles

http://theciscostores.com

1
http://theciscostore.com/http://theciscostore.com/


113/114



114/114

Thank you.

Documents

BRKDCT-2840 Data Center Networking Taking Risk Away From Layer 2 Interconnects