Quick Start Guide FabricPath

© 2013 Cisco and/or its affiliates. All rights reserved. 1

Quick Start Guide FabricPath

Architecture & Solutions GroupUS Public Sector Advanced ServicesMark Stinnette, CCIE Data Center #39151Date 1 August 2013Version 1.7.2


This presentation will provide end-to-end configurations mapped directly to commonly deployed data center architecture topologies. In this cookbook style; quick start guide; configurations are broken down in an animated step by step process to a complete end-to-end good clean configuration based on Cisco best practices and strong recommendations. Each QSG will contain set the stage content, technology component definitions, recommended best practices, and more importantly different scenario data center topologies mapped directly to complete end-to-end configurations. This QSG is geared for network engineers, network operators, and data center architects to allow them to quickly and effectively deploy these technologies in their data center infrastructure based on proven commonly deployed designs.

This Quick Start Guide (QSG) is a Cookbook style guide to Deploying Data Center technologies with end-to-end configurations for several commonly deployed architectures.


FabricPath ConfigurationFabricPath Designs – 2 SPINE (Routing at Aggregation)

Natural Evolution of the vPC Design

• Simplest design option :: traditional Aggregation / Access designs• Simplified configuration• Removal of STP• Traffic distribution over all uplinks without vPC port-channels

• Active / Active gateways (via vPC+ or Anycast HSRP)• VLAN anywhere (no trunk ports)• Option for vPC+ for legacy access switches and computer connectivity• Easily deploy L4-7 services


FabricPath ConfigurationFabricPath Designs – 4 SPINE (Routing at Aggregation w/ Anycast HSRP)

• Scale out; n-way Active HSRP in FabricPath (up to 4 today)• No longer need vPC+ at SPINE for active/active HSRP

• No peer-link or peer-keepalive link required • Leaf software needs to understand Anycast HSRP in FabricPath


FabricPath ConfigurationFabricPath Designs – Dedicated SPINE (Centralized Routing)


FabricPath ConfigurationFabricPath Designs – Dedicated SPINE (Centralized Routing) :: Alternative View

FabricPath Deployment in Preparation For Dynamic Fabric Automation (DFA)

• Paradigm shift with respect to typical designs (CLOS Fabric topology)• Simplifies SPINE design• Traditional “Aggregation” layer becomes pure FabricPath SPINE• Design helps ensure that any application node are at most only two hops away• FabricPath LEAF switches provide server connectivity like traditional designs• FabricPath LEAF switches also provide L2/L3 boundary, inter-VLAN routing, North South routing


FabricPath ConfigurationFabricPath Designs – Multi POD (with FP Multi-Topology)

NX-OS 6.2

• Provides DC wide vs. POD local VLAN segmentation / isolationCan support VLAN ID reuse in multiple PODs

• Define FabricPath VLANs :: map VLANs to topology :: map topology to FabricPath core ports

• Optional design for “disconnected” PODsEach POD can use same non-default FP topology; don’t need FabricPath Core since each POD is on its own island

• Where to place DC wide L2/L3 boundary (vPC+ or Anycast HSRP)FabricPath CorePick a any Aggregation PODRouted Sub-interfaces on Routed Core / WAN Edge via CE edge ports

• Default topology always includes all FabricPath core portsMap DC Wide VLANs to default topology

• POD local core ports also mapped to POD local topology Map POD local VLANs to POD local topology


FabricPath ConfigurationFabricPath Terminology


FabricPath ConfigurationFabricPath Encapsulation


FabricPath ConfigurationFabricPath Benefits Overview

FabricPath is a next generation Layer 2 technology from Cisco that provides multi-path Ethernet capabilities in L2 switching networks. FabricPath combines the benefits of L2 switching such as easy configuration and workload flexibility with greater scalability and availability. Specifically, FabricPath adds to L2 switching some routing type capabilities such as all active links, fast convergence, and loop avoidance mechanisms in the data plane. It allows Layer 2 networking without Spanning Tree Protocol.

FabricPath provides the following benefits:• Eliminates Spanning Tree Protocol (STP) with built-in loop prevention and mitigation (TTL & RPF)• Single control plane for unknown unicast, unicast, broadcast, and multicast traffic• VLAN anywhere• FP is transparent to L3 protocols• Easy to configure• Easy to manage• Flexibility

• Create arbitrary any topology • Multiple designs to integrate L2/L3 boundaries • Start small and expand as needed (bandwidth growth)

• Efficient and Scalable • Layer 3 availability similar features • Leverage parallel paths• Expanding available bandwidth at L2/L3 Default Gateway level• MAC address table scale (conversational learning) :: all FabricPath VLANs use conversation MAC address learning• Fast Convergence and low latency

• Enhances mobility and virtualization in the FabricPath network• Capable of running vPC (called vPC+) to connect devices to the edge in a port channel• Multi-tenant support, traffic engineering, meet security separation requirements via FabricPath topologies


FabricPath ConfigurationFabricPath Feature Configuration

Feature Benefit Overviewfabricpath VLAN mode & switchport mode

Eliminate STP protocol from the infrastructure fabric

The FabricPath ports carry traffic only for those VLANs configured as FabricPath VLANs. It is mandatory to enable the same FP mode VLAN EVERYWHERE on all switches in the FP fabric (otherwise, FP multidestination trees will be incorrectly built). VLAN pruning is performed automatically on FP core ports for FP traffic only.

fabricpath forwarding tables

Service Continuity

FabricPath uses 3 HW forwarding tables to switch frames (1) MAC address table, (2) Switch-ID table, (3) Multidestination tree table

fabricpath switch IDs

Service Continuity

Each switch in the FP fabric is allocated with a global switch ID value; this is allocated automatically or manually set (recommended). The switch ID information will be used in the MAC address-table for L2 forwarding. The vPC+ system also uses an emulated switch ID; which you assign on both peer devices.

fabricpah IS-IS link metric

Increase High-Availability

FP will always take the path with the lowest metric. Its recommended to use the default reference bandwidth.

fabricpath timers Improve Convergence Time

On a case by case basis, if convergence time needs to be improved upon switch reload, modify lsp-gen-interval and spf-interval timers.

fabricpath root priority

Service Continuity

FP uses two Multi destination Trees, Tree 1 (ftag 1) for broadcast, unknown unicast, multicast & Tree 2 (ftag 2) –multicast. Recommend to use on SPINE switches for primary and secondary root.

STP for Classical Ethernet (CE)

Service Continuity

The FP fabric must be the root of the L2 domain when connected to other legacy L2 domains / switches. Make sure STP priority is the lowest for the entire FP fabric.

vPC+ Increase High-Availability

FabricPath & vPC+ combined provides two main purposes, (1) dual attach a host to the FP fabric, (2) Leverage Active/Active HSRP capability


FabricPath ConfigurationFabricPath Feature Configuration

Feature Benefit OverviewOverload Bit Improve

Convergence Time

RFC 3277 based Overload bit is advertised in updates to prevent a corner case when a single switch restarts causing temporary loops or traffic black holing. This feature also prevents neighbors from using a switch as a transit during initial convergence as well as lowering impact insertion or removal of a switch to the FP domain.

Multiple Topologies

Design Separation

With multiple topologies, we can create up to 16 topologies where a subset of VLANs are mapped to a particular topology; allowing more design possibilities.

Anycast HSRP Increase High-Availability

Provides up to 4 active Default Gateways for the network which lowers the risk of disruption for routed and Inter-VLAN traffic and provides bandwidth capacity at L2/L3 boundaries. The Anycast HSRP feature removes the reliance on vPC+ to provide the Active/Active HSRP feature at the L2/L3 boundary.

fabricpath static routes

Traffic Engineering

The static route feature gives users capabilities to enter routes directly in the forwarding tables, ensuring predictable operation of the network.Certain uses cases where users want to override the routes computed by IS-IS. Users might want to route traffic to a particular switch using a particular link, better load balancing or routing traffic through a firewall (policing) in the network.


FabricPath ConfigurationInitial Baseline (Only 4 Commands !!)

feature lacpinstall feature-set fabricpathfeature-set fabricpath

vlan 1 – 200 mode fabricpath

interface po2 switchport mode fabricpath

interface e3/1, e4/1 channel-group 2 mode active

interface e5/1, e5/2 switchport mode fabricpath






1/11/2

1/11/2

3/14/1

3/14/1

Po2

Step 1 :: install | validate Enhanced L2 LicenseStep 2 :: install FabricPathStep 3 :: enable FabricPathStep 4 :: configure FabricPath VLANsStep 5 :: configure FabricPath core ports

7K-1 7K-2

5K-1 5K-2Po2

Install license bootflash:///enchanced_layer2_pkg.lic

show license usage

Default / Admin VDC Only

5/1 5/2 5/2 5/1

1/3 1/4 1/4 1/3

Default / Admin VDC Only










interface e1/3, e1/4 switchport mode fabricpathBeginning with the Cisco NX-OS Release 5.1 and when you

use an F Series modules and NX-OS Release 5.1(3) N1(1) with 5500 you can use the FabricPath feature


FabricPath ConfigurationManually set the FabricPath Switch-ID & Root

fabricpath switch-id 10

fabricpath domain default root-priority 255

fabricpath switch-id 11

fabricpath domain default root-priority 254

1/11/2

1/11/2

3/14/1

3/14/1

Po2

Step 1 :: set the FP Switch-IDStep 2 :: set the FP Root

7K-1 7K-2

5K-1 5K-2Po2

5/1 5/2 5/2 5/1

1/3 1/4 1/4 1/3

fabricpath switch-id 100 fabricpath switch-id 101

Root for FTAG 1 Root for FTAG 2

Each peer devices will have a unique global switch ID value – make the FP network more deterministic

Suggested switch ID scheme:SPINE :: 2 digit IDLEAF :: 3 digit IDEmulated Switch (vPC+) :: 4 digit ID

Multi destination Tree 1 (ftag 1) – broadcast, unknown unicast, multicastMulti destination Tree 2 (ftag 2) –multicast

Recommend to use on SPINE switches

Higher Number the better !!

(start at 255 and go backwards)-or-(start at 200 in case you need to introduce another MDT at a later time; ie expanded SPINE x 4)

F2/F2E uses both trees for UU/Bcast/Mcast

F1 uses MDT 2 for Mcast only

SW 10 SW 11

SW 100 SW 101


FabricPath ConfigurationManually set Spanning-Tree (one virtual root bridge)


spanning-tree pseudo-information vlan 1 – 200 root priority 0



1/11/2

1/11/2

3/14/1

3/14/1

Po2

Step 1 :: set FP domain to be root bridge

7K-1 7K-2

5K-1 5K-2Po2

5/1 5/2 5/2 5/1

1/3 1/4 1/4 1/3





The entire FabricPath domain will look like one virtual bridge to the CE domain – set best (lowest) STP root priority on the vPC+ peers (recommended at least at the access edge leaf switches); just make sure the priority is lower than anything else in the network (classical Ethernet)

FP will use the same bridge ID c84c.75fa.6000

optional optional

vlan 20, 40

spanning-tree vlan 20, 40 priority 8192

Note that the spanning-tree priority command would also work; however, it would change the priority for the spanning tree regardless of whether the switch were sending regular BPDUs (when Cisco FabricPath is not running) or sending BPDUs with the pseudo-information (when Cisco FabricPath is operational on the switch). In some scenarios, this change can have undesirable side effects.

The root and sender bridge MAC addresses of this pseudo-information are the same on every switch in the Cisco FabricPath domain

All ports at the edge of a Cisco FabricPath network are configured with the equivalent of root guard (don’t need to configure this feature), a feature that would block a port should it receive superior Spanning Tree Protocol BPDUs


FabricPath ConfigurationTune Timers for Fast Convergence

fabricpath domain default spf-interval 50 50 50 lsp-gen-interval 50 50 50

fabricpath timers linkup-delay 60



1/11/2

1/11/2

3/14/1

3/14/1

Po2

Step 1 :: tune the IS-IS timers in FabricPathStep 2 :: (optional) tune the FabricPath linkup-delay

7K-1 7K-2

5K-1 5K-2Po2

5/1 5/2 5/2 5/1

1/3 1/4 1/4 1/3





To achieve fast convergence during node failures and recovery scenarios, it is recommended to tune the IS-IS timers in Cisco FabricPath. This tuning is particularly important when a switch is inserted in the topology.

This configuration is recommended for all switches in the network

Note: Future enhancements such as Layer 2 IS-IS overload bit support in 6.2 will help to improve unicast and multicast convergence during FabricPath node failure scenarios when default IS-IS timers are used.

Problem Set: The IS-IS adjacency is established and the access-edge started sending traffic to aggregation-edge, but the control plane was not ready to forward the traffic to the next hop. The default spf and lsp-gen intervals are 8sec (default) and it attributes to the long convergence. To address this issue, the default spf and lsp-gen intervals of {max-wait, initial-wait, second-wait} are brought down to 50msec, with this configuration, the aggregation-edge restoration will yield sub-second convergence for Layer 2 traffic

Optional, to provide better network convergence upon a Cisco FabricPath switch restart, you should set a Cisco FabricPath linkup-delay timer to 60


FabricPath ConfigurationEnable vPC+ (dual attachment functionality & active/active HSRP)

1/11/2

1/11/2

3/14/1

3/14/1

Po2

Step 1 :: enable vPC+Step 2 :: set the emulated switch-idStep 3 :: enable dual-active exclude for vPC SVIs

7K-1 7K-2

5K-1 5K-2Po2

1/3 1/4 1/4 1/3

vPC+ is an extension of vPC for FabricPath. It allows dual-homed connections from Classical Ethernet (CE) switches and hosts capable of port channels. It also provides for active-active HSRP.

The configuration of peer-link and peer-keepalive links are required – as traditional vPC

Enable IP ARP Synchronization of ARP entries between vPC Peers improves convergence for North-South and East-West Layer 3 traffic when one of the vPC+ peers is brought back up

In a vPC environment, the Secondary vPC switch will bring down the SVIs by default when the peer-link is brought down. This behavior is fine in CE environment as the vPC legs are also brought down on the secondary vPC switch. However in the vPC+ environment, the down links to the Access-Edge switches are FabricPath Core ports; in the absence of the vPC+ peer-link, the SVIs can still communicate through the FabricPath core ports.

The vPC dual-active exclude vlan command helps to configure a VLAN list such that the SVI can continue to stay up on the secondary vPC switch even if the vPC+ peer-link is down.

Note: Since FabricPath does not rely on Spanning Tree Protocol, and the vPC+ peer link is a FabricPath Core port, so the peer-switch command is not needed under the vpc domain [x] configuration

With vPC+, a FabricPath switch is emulated between the CE and FabricPath domain. All packets originating behind the Emulated Switch will be marked with the source Switch ID of the emulated switch

Assign the same emulated switch ID on both vPC peers; but the emulated switch ID must be unique between different vPC domains

SW 1000SW 1000

vPC+

5/1 5/2 5/2 5/1

vpc domain 1 role priority 2 peer-keepalive destination [….] source [….] …. ip arp synchronize fabricpath switch-id 1000 dual-active exclude interface vlan 20

interface po2 switchport mode fabricpath vpc peer-link

vpc domain 1 role priority 1 peer-keepalive destination [….] source [….] …. ip arp synchronize fabricpath switch-id 1000 dual-active exclude interface vlan 20



FabricPath ConfigurationEnable vPC+ (full configuration – active/active HSRP @ SPINE)

1/11/2

1/11/2

3/14/1

3/14/1

Po2

Step 1 :: enable vPC+Step 2 :: set the emulated switch-idStep 3 :: enable dual-active exclude for vPC SVIs

7K-1 7K-2

5K-1 5K-2Po2

1/3 1/4 1/4 1/3

interface vlan 20 ip address 20.20.20.5/24 no ip redirect hsrp 20 preempt priority 110 ip 20.20.20.254

interface vlan 20 ip address 20.20.20.6/24 no ip redirect hsrp 20 preempt ip 20.20.20.254

feature interface-vlanfeature hsrpfeature lacpfeature vpc



------------------------

vpc domain 1 role priority 2 system-priority 4096 peer-keepalive destination [….] source [….] peer-gateway auto-recovery auto-recovery reload-delay delay restore 30 ip arp synchronize fabricpath switch-id 1000 dual-active exclude interface vlan 20



feature interface-vlanfeature hsrpfeature lacpfeature vpc



------------------------

vpc domain 1 role priority 1 system-priority 4096 peer-keepalive destination [….] source [….] peer-gateway auto-recovery auto-recovery reload-delay delay restore 30 ip arp synchronize fabricpath switch-id 1000 dual-active exclude interface vlan 20



5/1 5/2 5/2 5/1

SW 1000SW 1000

vPC+


FabricPath ConfigurationEnable vPC+ (dual attachment functionality @ LEAF)

1/11/2

1/11/2

3/14/1

3/14/1

Po2

Step 1 :: enable vPC+Step 2 :: set the emulated switch-idStep 3 :: add devices redundantly with vPC+

7K-1 7K-2

5K-1 5K-2Po2

5/1 5/2 5/2 5/1

1/5 1/5

vPC+

vPC 20

SW 1000SW 1000vPC+

feature lacpfeature vpc



------------------------

vpc domain 10 role priority 1 peer-keepalive destination [….] source [….] …. ip arp synchronize fabricpath switch-id 1001



interface port-channel 20 switchport switchport mode trunk switchport trunk allowed vlan 20 – 40 vpc 20 interface e1/5 channel-group 20 force mode active

feature lacpfeature vpc



------------------------

vpc domain 10 role priority 2 peer-keepalive destination [….] source [….] …. ip arp synchronize fabricpath switch-id 1001



interface port-channel 20 switchport switchport mode trunk switchport trunk allowed vlan 20 – 40 vpc 20 interface e1/5 channel-group 20 force mode active

SW 1000SW 1001 1/4 1/31/41/3

VLANs carried on vPC+ member ports must be FabricPath mode VLANs


FabricPath ConfigurationFabricPath Authentication

1/11/2

1/11/2

3/14/1

3/14/1

Po2

Step 1 :: configure the key chainStep 2 :: configure global FabricPath authenticationStep 3 :: configure FabricPath core port authentication

7K-1 7K-2

5K-1 5K-2Po2

5/1 5/2 5/2 5/1

1/3 1/4 1/4 1/3

FabricPath provides 2 levels of authentication1. Authentication at the interfaces level2. Authentication at the global level

The Key chain is used in both forms of authentication

Supported combinations:

interface port-channel2 switchport mode fabricpath fabricpath isis authentication-type md5 fabricpath isis authentication key-chain FP-KEYS

fabricpath domain default authentication-type md5 authentication key-chain FP-KEYS

key chain FP-KEYS key 0 key-string Cisc0! accept-lifetime 00:00:00 Sep 1 2012 infinite send-lifetime 00:00:00 Sep 1 2012 infinite

interface port-channel2 switchport mode fabricpath fabricpath isis authentication-type md5 fabricpath isis authentication key-chain FP-KEYS

fabricpath domain default authentication-type md5 authentication key-chain FP-KEYS

key chain FP-KEYS key 0 key-string Cisc0! accept-lifetime 00:00:00 Sep 1 2012 infinite send-lifetime 00:00:00 Sep 1 2012 infinite

global level authentication ::authenticates and controls the FP LSPs and PSNPsinterfaces level authentication ::authenticates the HELLO; the FP ISIS adjacency

You can configure the accept lifetime and send lifetime for a key. By default, accept and send lifetimes for a key are infinite, which means that the key is always valid.

accept-lifetime [local] start-time duration duration-value | infinite | end-time]send-lifetime [local] start-time duration duration-value | infinite | end-time]


FabricPath ConfigurationAnycast HSRP

1/11/2

1/11/2

3/14/1

3/14/1

Po2

Step 1 :: Step 2 :: Step 3 ::Step 4 ::Step 5 ::

7K-1 7K-2

5K-1 5K-2Po2

1/3 1/4 1/4 1/3

NX-OS 6.2

5/1 5/2 5/2 5/1


FabricPath ConfigurationOverload bit

1/11/2

1/11/2

3/14/1

3/14/1

Po2


7K-1 7K-2

5K-1 5K-2Po2

1/3 1/4 1/4 1/3

NX-OS 6.2

5/1 5/2 5/2 5/1


FabricPath ConfigurationMultiple Topologies & Multidestination Trees (MDT)

1/11/2

1/11/2

3/14/1

3/14/1

Po2


7K-1 7K-2

5K-1 5K-2Po2

1/3 1/4 1/4 1/3

NX-OS 6.2

5/1 5/2 5/2 5/1


FabricPath ConfigurationFabricPath Static Routes for Traffic Engineering Use Cases

1/11/2

1/11/2

3/14/1

3/14/1

Po2


7K-1 7K-2

5K-1 5K-2Po2

1/3 1/4 1/4 1/3

NX-OS 6.2

5/1 5/2 5/2 5/1


FabricPath vs. vPC ComparisonFabricPath is Simple !!

VPC Configuration

e1/1-2

e3/1-2

e2/1-2

e1/3 e2/3

VPC

e2/5e1/5

interface e2/5 ip address 192.168.1.2/24 vrf membership vpc-keepalivevpc domain 1 peer-keepalive destination 192.168.1.1 source 192.168.1.2 vrf vpc-keepaliveinterface port-channel 1000 switchport mode trunk vpc peer-linkinterface e2/1-2 switchport mode trunk channel-group 1000 mode activeinterface e2/3 switchport mode trunk channel-group 1 mode activeinterface port-channel1 vpc 1

interface e1/5 ip address 192.168.1.1/24 vrf membership vpc-keepalivevpc domain 1 peer-keepalive destination 192.168.1.2 source 192.168.1.1 vrf vpc-keepaliveinterface port-channel 1000 switchport mode trunk vpc peer-linkinterface e1/1-2 switchport mode trunk channel-group 1000 mode activeinterface e1/3 switchport mode trunk channel-group 1 mode activeinterface port-channel1 vpc 1

interface e3/1-2 switchport mode trunk channel-group 1 mode passive

FabricPath Configuration

FabricPath

e1/1-2

e3/1-2

e2/1-2

e1/3 e2/3

interface e3/1-2 switchport mode fabricpath



vPC FabricPath + vPC+

AdvantagesActive/active path at L2Active/active for HSRPWorks with all LC

AdvantagesActive/active path at L2Active/active for HSRPEase of configurationNo more STPExtensibility

DrawbacksNeed dedicated infrastructure (PL, PKL)Configuration on both peer devicesConsistency check to care aboutSTP still here (but runs as fail safe mechanism)

DrawbacksNeed dedicated infrastructure (PL, PKL)Need F1 (+M1) or F2


vPC to FabricPath Migration

Common Design Migration Starting Point 7k – Aggregation5k/2k – Access PodsDual Layer vPCMix F1 / M1 line cards

After Migration Completion 7k – SPINE role 5k – LEAF rolevPC converted to FabricPath core portsPeer-Link also FP core port = vPC+ (only F1/F2 support FabricPath)

Additional Reading Here :: http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c11-709336.html

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c11-709336.html




FabricPath ConfigurationStrong Recommendations and Key Notes

• FabricPath VLANs must be configured on all switches in the FP domain• It is recommended to configure the switch ID manually on all FabricPath switches• For Active-Active HSRP capability, it is recommended to configure vPC+ on the Aggregation-Edge switches even if there

are no vPC legs. Note: subject to vPC rules; so no dynamic routing over vPC to firewalls, Core layer, WAN edge• Implement Layer 3 routing backup path

• Separate L3 port channel; point-to-point links• Separate L2 port channel; use dedicated VLAN in Classical Ethernet (CE) mode as transit VLAN inside this L2

trunk• Disable IP redirects on SVIs and configure passive interface to avoid any routing adjacency over SVIs• ARP sync feature with vPC+ is recommended for improved traffic convergence during Aggregation-Edge failure and

restoration• It is recommended to configure highest and second highest MDT root priority on the Aggregation-Edge switches• Have option of choosing single links or port-channels between Aggregation-Edge and Access-Edge for ECMP. If port

channels are used, configuring IS-IS metric is preferred. With path costing, member link failure is transparent to IS-IS protocol so that the traffic would continue to use the same path

• Raise the FP IS-IS metric for VPC+ Peer-Link to prefer other FP core links• interface po2

fabricpath isis metric 200• It is recommended to have lowest path cost for the links between AGG devices so the multicast hello packets always take

the peer-link which is direct link between the AGG devices• It is recommended to tune Layer 2 IS-IS SPF and LSP generation timers to achieve better convergence during failure and

restoration scenarios. These timers should be tuned to 50 msec with 50msec initial wait and second wait. This is a requirement until the overload bit support is available with Layer 2 IS-IS

• Use default reference BW (its 400 Gbps default) • fabricpath domain default

reference-bandwidth ?• IS-IS metric cost (1Gb = cost 400, 10Gb = cost 40, 20Gb = cost 20)



• IS-IS link metric for port-channel depends on NX-OS version• Up to NX-OS 6.0: IS-IS metric for port-channel is calculated based on number of configured member ports;

meaning you may need to use LACP min-link feature to tear down port-channel if number of active member ports goes below a specific limit

• Since NX-OS 6.1: IS-IS metric for port-channel is calculated based on number of active ports• Dual-active exclude VLAN configuration is recommended so that the SVIs can continue to be active on the secondary

vPC+ peer in the event of peer-link failure. This also helps to stay with default HSRP timers there by reducing control plane load associated with aggressive HSRP timers

• Do not use dual-active exclude command for VLANs if you have vPC attached devices, for example at the access (leaf)• In typical vPC deployments it is not necessary to tune the HSRP hello timers from the defaults (3/10s). In a FabricPath

deployment it is recommended to use aggressive timers (1/3) to minimize flooding of South to North traffic from the edge switch. This allows the active HSRP virtual MAC to be learned faster at all edge switches

• hsrp 1preempt delay minimum 180timers 1 3ip ….

• In CE-FabricPath hybrid networks, it is recommended to configure the lowest Spanning-tree root priority on all FabricPath Edge switches

• The MAC timer should be consistent on all devices in the Layer 2 topology. The MAC and ARP aging timers can be left at defaults, 1800sec & 1500sec respectively

• The M1/F1 mixed VDC currently supports up to 16K MAC/ARP entries. This limitation will be lifted with the Layer 2 proxy learning feature in the upcoming NX-OS release

• The M1, M1-XL, M2 & F2E in a mixed VDC topology; meaning when F2E is placed in a chassis with M-series it will operate in Layer 2 mode only leveraging the M for Layer 3 (proxy L3 forwarding); this will enable 128K MAC/ARP scale.

• If an ASA cluster is attached to the Nexus 7000 series Aggregation-edge switches, source-dest-ip or src-dst ip-l4port is the recommended load balance algorithm if the ASA cluster is in single context mode or if the VLANs are fewer in multi-context mode. This is to prevent traffic polarization on links towards ASA cluster member



• Better use port-channel instead of individual links for the 2 following reasons• Decrease the number of direct IS-IS adjacency (1 for the whole port-channel instead of X IS-IS adjacencies if X

individual links are used between the 2 switches)• Allows to use the whole port-channel capacity for multidestination tree #1 or #2 (if multiple parallel individual links

exist between 2 switches, only 1 link will be selected for tree #1 and potentially 1 another link for tree #2)• ECMP vs. Port Channel

• Can use ECMP, port-channel, or both simultaneously• Port-channels have one main advantage over ECMP – treated as single logical link in FabricPath IS-IS. Individual

link failure invisible to upper layer protocols. Also allows more bandwidth for branches of multidestination trees• With 4 member port channel, whole interface becomes single branch of tree with 40G BW• With 4 parallel ECMP paths, only one of the 4 interfaces becomes part of the tree

• ECMP with port-channel : 2 levels of load-balancing decision :• First level : FP Core Link selection (based on L3/L4 fields by default)• Second level : Port-Channel member selection (based on src-dst ip by default)

• Do not use UDLD with FabricPath• UDLD (normal or aggressive) does not bring any benefits on single physical link and port channels with FP enabled

(for port channel, activate LACP instead of relying on UDLD to detect member port issues)• Physical link level protection and the bi-directional IS-IS hellos should take care of all (or near all) potentially link

level issue


FabricPath vPC+ ConfigurationMixed Chassis Mode Supported Topologies

Interop F2 & F2E VDC

With NX-OS 6.1 and Prior Releases ::

• Always use identical line cards on either side of the vPC+ Peer Link, vPC member ports, and FabricPath core member ports (legs to downstream device)

• The F1-series line cards can mix with M-series line cards• The F2-series line cards have to be in their own VDC; VDC type [F2] meaning

they can’t mix with F1 or the M-series in the same VDC


FabricPath vPC+ ConfigurationMixed Chassis Mode Supported Topologies

Starting in NX-OS 6.2 and Later Releases ::

• VDC type [F2, F2E, F2 F2E] must match between the 2 vPC+ peer devices when F2 & F2E are used in same VDC; meaning its ok to have F2 on vPC peer device 1 and F2E on vPC peer device 2 for the vPC Peer Link, vPC member ports, or FabricPath core member ports

• Note: in a F2 & F2E type of design; only features related to F2 apply (lowest common denominator)

• Always use identical line cards on either side of the vPC Peer Link, vPC member ports, and FabricPath core member ports when M1, M1-XL, M2 & F2E in same VDC [M-F2E] or system

• When F2E is placed in a chassis with M-series it will operate in Layer 2 mode only leveraging the M for Layer 3 (proxy L3 forwarding); this will provide 128K MAC scale


FabricPath vs. TRILL


Great External Resource

FabricPath ConfigurationFabricPath Resources & Additional Reading

External (public)http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c07-728188.pdf http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c11-605488.html

Nexus 7000/6000/5000 Configuration Guideshttp://www.cisco.com/en/US/products/ps9402/products_installation_and_configuration_guides_list.htmlhttp://www.cisco.com/en/US/products/ps9670/products_installation_and_configuration_guides_list.html http://www.cisco.com/en/US/partner/products/ps12806/products_installation_and_configuration_guides_list.html

FabricPath Scaling limits http://www.cisco.com/en/US/docs/switches/datacenter/sw/verified_scalability/b_Cisco_Nexus_7000_Series_NX-OS_Verified_Scalability_Guide.html#reference_3AD0536C32FF4B499A0936409729951D http://www.cisco.com/en/US/docs/switches/datacenter/nexus5500/sw/configuration_limits/b_N5500_Config_Limits_602N11_chapter_01.html

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c07-728188.pdf

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c07-728188.pdf



http://www.cisco.com/en/US/products/ps9402/products_installation_and_configuration_guides_list.html




http://www.cisco.com/en/US/partner/products/ps12806/products_installation_and_configuration_guides_list.html

http://www.cisco.com/en/US/partner/products/ps12806/products_installation_and_configuration_guides_list.html

http://www.cisco.com/en/US/docs/switches/datacenter/sw/verified_scalability/b_Cisco_Nexus_7000_Series_NX-OS_Verified_Scalability_Guide.html#reference_3AD0536C32FF4B499A0936409729951D




http://www.cisco.com/en/US/docs/switches/datacenter/nexus5500/sw/configuration_limits/b_N5500_Config_Limits_602N11_chapter_01.html



Documents

Quick Start Guide FabricPath