Deploying Nexus 7000 in Data Center Networksd2zmdbbm9feqrf.cloudfront.net/2010/usa/pdf/BRKDCT-2951.pdf · Cisco Nexus 7000 Switch Architecture ... Deploying Virtual Port Channel in

BRKDCT-2951

Deploying Nexus 7000 in Data Center Networks

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 2

Session Abstract

This session is targeted to network administrators and operators who have deployed or are considering the deployment of the Nexus 7000. The session starts with a brief introduction to the Nexus 7000 hardware components. Then it is followed by a brief design discussion. The focus of the presentation is on implementation and best practices. The implementation section will cover installation, layer-2 & layer-3 protocols, security features, and system management features. The session will cover NX-OS CLI but troubleshooting is not part of this presentation’s scope.

Attendee should have a basic knowledge of the Nexus 7000 hardware and software platform as well as solid knowledge of L2 and L3 protocols.


Associated Sessions/Labs

Cisco Nexus 7000 Switch Architecture - BRKARC-3470

Cisco NX-OS Software Architecture - BRKARC-3471

Deploying Virtual Port Channel in NXOS - BRKDCT-2048

Nexus 7000/NX-OS Hands On Lab - LRTDCT-2847


Agenda

Hardware Overview

Data Center Designs

Implementation and Best Practices


Hardware Overview7010 Chassis

8 I/O slots, 2 supervisor slots (5, 6)

Front-to-back air flow utilizing 2 system fan trays and 2 fabric fan trays

21 RU (2 per 42 RU rack)

Up to three power supplies

6kW AC, 7.5kW AC and 6kW DC PS

5 fabric module slots

46 Gbps per I/O module slot

fabric module is unique to chassis type

All components support Online Insertion and Removal (OIR)

Optional air filter satisfies NEBS requirements

Front View Rear View


Hardware Overview7018 Chassis

Front View Rear View

16 I/O slots, 2 supervisor slots (9, 10)

Side-to-side (right to left) air flow utilizing 2 system fan trays

25 RU

Up to four power supplies

6kW AC, 7.5kW AC and 6kW DC PS

5 fabric module slots

46 Gbps per I/O module slot

fabric module is unique to chassis type

All components support Online Insertion and Removal (OIR)


Hardware OverviewI/O Modules (Non-XL)

32 port 10GE (80G) SFP+

4:1 port-level oversubscription

Default rate-mode is shared

48 port 10/100/1000 (46G) RJ45

48 ports 1GE (46G) SFP

N7K-M132XP-12

N7K-M148GT-11

N7K-M148GS-11

Capability Size

MAC entries 128K

IPv4 / IPv6 routes (128K / 64K)

Security / QoS ACL entries 64K

Netflow 512K


Hardware OverviewI/O Modules (XL)

8 port 10GE XL (80G) X2

2 forwarding engines (up to 120Mpps)

48 ports 1GE XL (46G) SFP

N7K-M108X2-12LN7K-M148GS-11L

CapabilitySize (w/o Scalable Feature License)

Size (w/ Scalable Feature License)

MAC entries 128K 128K

IPv4 / IPv6 routes (128K / 64K) Up to (1M / 350K)*

Security / QoS ACL entries 64K 128K

Netflow 512K 512K

NX-OS 5.0 NX-OS 5.0

* Actual limit depends on prefix distribution


Agenda

Hardware Overview

Data Center Designs



Virtual Port-Channel (vPC)Design Motivations

Provides multi-chassis etherchannel capability (L2 port-channel only)

Eliminates STP blocked ports and reduce STP complexity

Does not depend on access switches for STP convergence

Uses all available uplink bandwidth

Enables dual-homed servers to operate in active-active mode

Provides fast convergence upon link/device failure

Software Version Number of vPC

Pre 4.2 release 196

4.2(1) and later 256

Double-sided vPC

vPC vPC

vPC vPC


Virtual Device Contexts (VDCs)Design Motivations

Consolidate and support multiple business units, departments, and networks

Web, App, Database

Production, OOB mgmt, Development, Test

Customer A, Customer B, Customer C

Provide network segmentation to meet security compliance requirements

Internet, Extranet, DMZ, Intranet

Non-Secured, Secured, PCI

Implement logical tier design

Core, Aggregation, Access

VDC2

Secure

VDC3Non-

Secure

VDC2Prod

VDC3Dev

VDC3Agg

VDC4Access

VDC2Core

VDC2BU1 /App 1

VDC3BU2 / App 2

VDCs provide logical separation of control-plane,

data-plane, management, resources, and system

processes within a physical switch

VDC4Test


Data Center Design Example 1

Large Data Center utilizing 3-Tier DC design

Nexus 7000 in core and aggregation

10GE/GE ToR and GE MoR access layer switches

Implement vPC / double-sided vPC to eliminate L2 loops and to support active/active server connections

VPC

L2

L3

L3

L2

L2 Channel

L3 link

L2 link

L3 Channel

Access

Core

Aggregation ……..

Core1 Core2

agg1bagg1a aggNa aggNb

......

VPC

active standbyactive active active standbyactive active

VPCVPC




Nexus 7000 in core and aggregation, Nexus 5000 and Nexus 2000 in access layer

Implement vPC / double-sided vPC to eliminate L2 loops

Two different vPC redundancy models can be utilized to support active/active or active/standby server connections

L2L3

L3

L2

L2 Channel

L3 link

L2 link

L3 Channel

Access

Core

Aggregation

……..VPCVPC

VPC VPC VPC VPC

active standby active standby Active/Active Active/Active

Core2Core1

aggNa aggNbagg1a agg1b

vPC vPC vPC vPC

vPC vPC




Nexus 7000s in Core and Aggregation

Utilize VDCs in aggregation layer to create a non-secured zone and a secured zone

10GE/GE ToR and GE MoR access layer switches


L2

L3

L3

L2

L2 Channel

L3 link

L2 link

L3 ChannelCore

Aggregation

Access

SW-2b

VDC3

SW-2a

VDC3SW-2a

VDC2SW-2b

VDC2

SW-1a

VDC3SW-1b

VDC3

SW-1a

VDC2

SW-1b

VDC2

Core2Core1

vPC vPC

active activeactive standbyactive activeactive standby

vPCvPC vPCvPC



Small Data Center with a “virtualized” 3-Tier DC design

Utilize VDCs on a single device to create a core and aggregation layer

GE and 10GE ToR access layer switches


L2

L3

L3L2 Channel

L3 link

L2 link

L3 ChannelCore

Aggregation

Access

SW-1b

VDC2SW-1a

VDC2

active standby

SW-1b

VDC3SW-1a

VDC3

vPC

L2active active

vPC


Agenda

Hardware Overview

Data Center Designs


© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 17


Installation and Maintenance


Chassis Installation

Use standard four-post, 19-inch EIA data center rack

Cabinet can be leveraged to convert 7018 to front-to-back air cooling

When installing 7018

Reserve 11” space on both sides of the rack to allow for side-to-side airflow

Route cables on front side of the rack to clear the rear side for airflow

Always perform chassis / system grounding

7010 chassis

7018 chassis


Hardware Installation

Two supervisors are recommended for high availability and ISSU

Configure redundant power redundancy-mode

Available power in redundant mode is the minimum of IS and PS redundancy

System default is PS redundant

Connect PS input sources to two different power grids

Setting maximum number of fabric modules per system allows the system to release some of the reserved power (supported in NX-OS 5.0)

By default system reserves enough power for five fabric modules

Fabric modules must be installed in the first N fabric module slots

220V

Grid 1 Grid 2

220V

Nexus7K(config)#

power redundancy-mode redundant

Nexus7K(config)#

hardware fabrics max 3


Virtual Device Contexts (VDCs)

VDC2Agg1

VDC4Test

VDC1Admin

VDC3Acc1

HA Policy =

Bring down

Production VDC

HA Policy =

Switchover

Nexus7K(config-vdc)# ha-policy dual-sup <policy> single-sup <policy>

Nexus7K(config-vdc)# limit-resource vlan minimum <#> maximum <#>

Test VDC

vlan = 50

Linecard 1 Linecard 2

64K 64K

128K 128K

FIB

TCAMACL

TCAM

VDC 2 VDC 3 VDC 4

Assign I/O modules to VDCs such that TCAM resources are shared effectively

Allocate entire I/O module per VDC if possible

All ports in the same port group on the 32 port 10GE I/O modules must be allocated to the same VDC

Customize VDC HA policy and resource configurations as necessary

Dual-sup default is switchover and single-sup default is restart

Only non-default VDCs can be suspended, resumed, reloaded, or restarted Nexus7K(config)# vdc <name> suspend Nexus7K# reload vdc <name>

Nexus7K(config-vdc)#allocate interface e2/1,e2/3,e2/5,e2/7

If 3 or less data forwarding VDCs are required, reserve the default VDC as the administrative VDC

On the default VDC, assign accounts with minimum privileges necessary to accomplish operational tasks

FIB

TCAMACL

TCAM


OOB (out-of-band) Management Network

Supports both mgmt0 and CMP ports

The IP address for default and non-default VDCs must be assigned to the same IP subnet

Assign different IP address’ for redundant CMP (same IP address for redundant mgmt0 interface)

If default VDC is reserved as the “admin” VDC, OOB mgmt network is necessary to provide access

Provides the option to assign all system management servers in “management” VRF and control access via ACL on mgmt interface

Consider the resiliency of the OOB mgmt network

Mgmt0 Mgmt0

Mgmt0 x2 Mgmt0 x2

Agg1a Agg1b

Acc1 Acc2

Core2Core1

OOB MgmtDist

L3

VDC2Agg1

VDC3Agg2

VDC1Admin

VDC2Agg1

VDC3Agg3

VDC1Admin

Mgmt0 Mgmt0

OOB MgmtNetwork

CMP x2 CMP x2

Mgmt0 x2Mgmt0 x2

Sys Mgmt server

It is recommended to implement an

OOB management network

mgmt1 mgmt2

Management VRF

Default VRF


Software Licensing

Feature License Features

Enterprise LAN OSPF, EIGRP, BGP, ISIS, PIM, MSDP, PBR, GRE

Advanced LAN CTS, VDCs

Scalable Feature XL TCAM

Transport Services OTV

Base NX-OS features do not require a license

Include basic Infrastructure, L2 switching, etc

All features are shipped with NX-OS image

Install individual licenses or enable the license grace period (120 days) to enable advanced features

License is tied to chassis serial number

License is stored in dual redundant NVRAM modules on chassis backplane

If chassis is replaced, work with Cisco TAC to re-key the license

If supervisor is replaced, license can be re-installed but not required (show license usage will indicate the license is installed but is missing)


Software Licensing (cont.)License PAK

PAK +

chassis serial #

<xml...

licA ...>

license

file

Nexu7K# install license bootflash:<license_file.lic>

………

Nexu7K# copy bootflash:<license_file.lic> tftp:….

…………

Nexu7K# show license usage

Feature Ins Lic Status Expiry Date Comments

Count---------------------------------------------------------------------------------------------------------------

LAN_ADVANCED_SERVICES_PKG Yes - In use Never -

LAN_ENTERPRISE_SERVICES_PKG Yes - Unused Never -

The required licenses can be either factory installed or manually installed

License installation is non-disruptive to features already running under the grace period

Follow the steps to manually install the licenses

Identify chassis serial number and PAK (Product Activation Key)

Nexu7K# show license host-id

License hostid: VDH=TBM########

Obtain the license key file from http://www.cisco.com/go/license

Copy licenses to bootflash, install licenses and backup the licenses

http://www.cisco.com/go/license


Software UpgradeCold Start Upgrade

Utilize cold start upgrade procedure to minimize the upgrade window for non-production devices

It is recommended to synchronize the kickstart image and the system image

Nexu7K(config)#

boot system bootflash:<system-image> sup-1 sup-2

boot kickstart bootflash:<kickstart-image> sup-1 sup-2

Nexus7K# copy run startup-config

Nexus-3# sh boot

---deleted---

Boot Variables on next reload:

sup-1

kickstart variable = bootflash:/<kickstart-image>

system variable = bootflash:/<system-image>

sup-2

kickstart variable = bootflash:/<kickstart-image>

system variable = bootflash:/<system-image>

No module boot variable set

Nexus7K# reload


In-Service Software Update (ISSU)

Utilize ISSU to upgrade devices with zero disruption

Upgrade the system with a single “install all” command

Reload the CMP modules to complete the CMP upgrade

It is recommended to synchronize the kickstart image and the system image

Show commands can be used ahead of time to determine any potential impact prior to performing ISSU

Issue “show install all impact” to determine upgrade impact

When downgrading software, use “show incompatibility-all” to determine if any features need to be disabled

Nexus7K# show install all impact kickstart bootflash:<kickstart> system bootflash:<system>

Nexus7K# show incompatibility-all system bootflash:<system-image>

The following configurations on active are incompatible with the system image

1) Service : vpc , Capability : CAP_FEATURE_VPC_RELOAD_RESTORE

---deleted---

Nexus7K# install all kickstart bootflash:<kickstart-image> system bootflash:<system-image>

Nexus7K# show install all status


ISSU (cont.)

ISSU upgrade performs the following actions

Verify and validate the image, check image compatibility, provide descriptive upgrade information and option to cancel, sync images to standby sup, upgrade and switchover standby sup, upgrade previous active sup and I/O modules, load new image to CMP

Avoid disruption to the system during upgrade

STP topology change, module removal, power interruption, etc

Understand configuration conditions that cause ISSU failure

Active config sessions, suspended VDCs, disabling OSPF/ EIGRP / BGP / ISIS graceful restart, BGP hold timer tuned to less than switchover time (15 sec)

Pre-Upgrade check failed. Return code 0x80960002 (No such file or directory)

Nexus7K# show install all failure-reason

Service "session-mgr" returned error: Session-Manager active sessions present,

Nexus7K# show install all failure-reason

Service "stp" in vdc: 1 returned error: STP topology change in progress which can impact

ISSU. As a precaution ISSU is rejected. (x40DD0033)

Nexus7K# sh install all status

This is the log of last installation.


EPLD Upgrade

EPLDs (Electronic Programmable Logical Devices) upgrade is used to enhance hardware functionality or to resolve known issues

Performed on all the field replaceable modules (fan trays, fabric modules, I/O modules, and supervisor)

It is recommended to upgrade to the latest EPLD image for non-production devices

EPLD upgrade is a separate and independent process from ISSU and is typically not required

Check EPLD release notes or issue “show install all impact epld” to determine if EPLD upgrade is required

Nexus7K# show install all impact epld bootflash:<EPLD_image_name>

Nexus7K# install all epld bootflash:<EPLD_image_name>

Nexus7K# show version <type> <mod #> epld


EPLD Upgrade (cont.)

When perform EPLD upgrade for a dual supervisor system, upgrade the standby first, then switchover and upgrade previous active supervisor

In a redundant system, only EPLD upgrade for an I/O module can disrupt traffic since the module need to be power-cycled

When performing EPLD upgrade for mission critical systems, upgrade I/O module individually instead of all installed modules

Nexus7K# install module <module> epld bootflash:<EPLD_image_name>


Command Line Interface (CLI)

Leverage CLI alias to replace frequently used commands / actions

Nexu7K(config)#

cli alias name wri copy run start

cli alias name vpcpreempt conf t ; vpc dom 1 ; role pri 16384 ; int po 1 ; sh ; no sh

Nexus7K# show cli var

VSH Variable List

-----------------

SWITCHNAME="Nexus7K"

TIMESTAMP="2010-05-06-20.49.24"

Nexus7K# copy run bootflash:/$(TIMESTAMP)-$(SWITCHNAME)-cfg

Nexus7K# dir bootflash:

4265 May 06 20:22:24 2010 2010-05-06-20.50.24-Nexus7K-cfg

Nexus7K# show cli syntax | i spanning-tree

(788)[ no ] debug spanning-tree all

---deleted---

Nexu7K(config)# show cli syntax | i spanning-tree

(125) spanning-tree mode <stp-mode> | no spanning-tree mode [ <stp-mode> ]

Nexu7K(config-if)# show cli syntax | i spanning-tree

(58)[ no ] spanning-tree [ vlan <vlan-id> ] cost auto

Nexus7K# sh cli list arp

Reference CLI variables in scripts and CLI commands

Utilize CLI syntax / CLI list to identify available commands

Utilize CLI history to identify the command history


Nexu7K# checkpoint checkpt1

Processing the Request... Please Wait

Nexus7K# show diff rollback-patch running-config checkpoint checkpt1

Processing the Request... Please Wait

Nexus7K# config t

Enter configuration commands, one per line. End with CNTL/Z.

Nexus7K# rollback running-config checkpoint checkpt1……..

Configuration Rollback

The rollback feature allows users to take a configuration snapshot and reapply the configuration at any point

Create up to 10 checkpoints per VDC

Nexus7K(config)# no feature ospf

Nexus7K(config)# sh checkpoint all

----------------------------------------------

Name: system-fm-__inst_1__ospf

The rollback changes can be viewed before committing to the rollback operation

Auto-checkpoint protects against unintended loss of configuration (invoked with feature removal and license expiration)


Other Installation Considerations

Configure complete boot-up diagnostic level (default)

Before bringing staged devices to production, power-cycle again to perform boot-up diagnostic

Utilize “show hardware capacity” to determine system capacity and capacity planning

Nexus7K(config)#

diagnostic bootup level complete

Nexus7K# show diagnostic result module all

Nexus7K# show hardware capacity



Layer-2 Features


VLAN Trunking Protocol (VTP)

VTP “OFF” mode is recommended

Switches do not participate in VTP and all VTP advertisements are not forwarded

Utilize VTP transparent mode if VTP domain needs to extend across Nexus 7000 switches

Must ensure VLAN1 is allowed on trunks when operating in VTP transparent mode

VTP client / server mode is not supported

DCNM (Data Center Network Manager) can be leveraged to replicate VLAN database

Internal VLANs (3968 - 4047, 4094) are reserved and can’t be re-allocated

The reserved VLANs for Cat6Ks are 1002 –1005. Additionally, users can configure the internal VLAN allocation policy (ascending from 1006 or descending from 4094)

Nexus7K(config)# feature vtpNexus7K(config)# vtp domain <name>

Nexus7K(config)# no feature vtp

VTP server VTP client

Transparent

VTP

packets

Acc2

agg1bagg1a

Acc1

Transparent

Off

Acc2

agg1bagg1a

Acc1

Must allow

VLAN1


General Layer-2 Features

Enable UDLD feature to configure UDLD normal mode on all fiber interfaces

Enabling UDLD feature is equivalent to configuring UDLD normal mode globally

All fiber interfaces will inherit the global UDLD setting

Nexus7K(config)# feature udld

Nexus7K(config-if-range)# udld aggressive

Nexus7K(config-if)#

mtu <mtu>

Nexus7K(config)#

system jumbomtu <MTU>

It is recommended to configure UDLD aggressive on port-channel member ports

Interface configuration supersedes the global UDLD setting

The default CAM aging timer is 1800s and ARP timeout is 1500s

The default timers limits unicast flooding associated with asymmetric forwarding by synchronizing the CAM aging with ARP timeout

When implementing jumbo frame, L2 MTU must match the system jumbo MTU

The default system jumbo MTU is 9216

Match L3 MTU if L3 forwarding is required


General Layer-2 Features (cont.)

Implement one of the following methods to prevent double encapsulation 802.1Q attack

Assign unused VLAN as native VLAN (consistent across the same L2)

Clear native VLAN from the trunk

Configure to tag the native VLAN on all trunksNexus7K(config)# vlan dot1Q tag native

Nexus7K# show interface status err-disabled

Nexus7K(config)# errdisable recovery cause <cause>

Nexus7K(config)# errdisable recovery interval <time>

It is recommended to manually bring up error-disabled interface after the cause is identified

Errdisable recovery is disabled by default

Implement storm-control on L2 host ports and access layer to prevent disruptions caused by broadcast and multicast storm

Unsupported Layer-2 features

DTP, ISL Trunk, Flexlink, Link-State Tracking


EtherChannel

Active Utilize LACP to negotiate both L2 and L3 port-channels N7010-2

N7010-1 BPDUs

ONRoot

Active

N7010-2N7010-1

ON L3 Po is

down !

L3 Po

is up

BPDUs

Nexus7K(config)# feature lacp

Nexus7K(config)# int e<mod>/<port>

Nexus7K(config-if)# channel-group <#> mode activeDispute!

Mismatch Conditions

Nexus7K(config)#

Shut

no lacp graceful-convergence

no shut

Nexus7K(config)#

Shut

no lacp suspend-individual

no shut

Disable LACP suspend-individual only on “edge” port-channel

Nexus7K(config-if-range)# lacp rate normal

Disable LACP “graceful-convergence” on port-channel if “graceful-convergence” interoperability is an issue

If required, disable LACP “suspend-individual” on port-channel interface to allow the individual member ports to operate as “individual”

Implement port channels with 2, 4 or 8 members for optimal traffic distribution

Understand LACP compatibility enhancements

Implement normal LACP timer in a dual supervisor system (default)


EtherChannel (Cont.)

Understand port-channel failure behaviors

BW and IGP cost for L3 channel are recalculated when physical member fails

STP cost for L2 channels does not recalculate when physical member fails

OSPF Cost

50 100

50

Nexus7K#sh port-channel load-balance forwarding-path interface port-channel 1 src-ip 1.1.1.1 dst-ip 2.2.2.2 vlan 2 mod 3

Missing params will be substituted by 0's.

Module 3: Load-balance Algorithm: source-dest-ip-vlan

RBH: 0x7 Outgoing port id: Ethernet3/3

Access

Aggr1a Aggr1b

Core1-1 Core-2

Statically configure IGP cost on L3 channel if the default behavior is not desired

Modify port-channel load-balancing to match needs

Configure on default VDC and the default is Source-Destination-IP-VLANNexus7K(config)# port-channel load-balance ethernet <lb-method>

Nexus7K(config)# port-channel load-balance ethernet <lb-method> module <mod>

Unsupported etherchannel features

PAgP, LACP min-link

Nexus7K# sh port-channel load-balance


Spanning-tree (STP)

Implement consistent STP mode in the same L2 domain

RPVST+ is the default and is backward compatible with PVST

Nexus7K# sh spanning-tree active | i PeerPo11 Desg FWD 12 128.4106 P2p Peer(STP)

Nexus7K# show spanning-tree summary total

----deleted----

Name Blocking Listening Learning Forwarding STP Active

---------- -------- --------- -------- --------- ---------

9 vlans 0 0 0 18 18

Total number of

logical ports

Nexus7K# sh spanning-tree active | i Bound

Po11 Desg FWD 100000 128.4106 P2p Bound(PVST)

Utilize MST to scales large L2 network

MST supports 75K logical ports (90K in NX-OS 5.0) and RPVST+ supports 16K logical ports

MST introduces some complexity and requires proper planning

MST interoperates with both RPVST+ and PVST+ by utilizing PVST+ simulation

3 msts 2 0 0 8 10 MST ports

Configure the allowed VLANs on trunk interfaces


Spanning-tree (Cont.)

Configure aggregation switches as the STP primary and secondary root

Enable Bridge Assurance (BA) if supported on both local and remote switches

BA is enabled globally (default) and active only on interfaces configured as port type “network”

BPDUs are sent on all active BA ports

BPDU-

guard

Bridge Assurance

Loop-Guard

Port Type Edge / Edge Trunk

Pri / Sec

root

agg1a agg1b

Access2

Access1

L3 FWL3 FW

Enable loopguard globally If BA is not supported on access switches

Configure host ports as port type “edge” or port type “edge trunk”

Enable STP BPDU-guard globally

Dispute mechanism is integrated by default

Nexus7K(config-if)#spanning-tree port type edge trunk

Nexus7K(config)#spanning-tree port type edge bpduguard default

Nexus7K(config-if)# spanning-tree port type network

Nexus7K(config)# spanning-tree vlan <vlan> pri <pri>


Spanning-tree (Cont.)

Implementing STP long path-cost method

RSTP default is short and MST default is long

Utilize port-profiles to enforce consistent configuration

Nexus7K(config)#port-profile type ethernet host-port

state enable

switchport

switchport mode access

spanning-tree port type edge

spanning-tree bpduguard enable

no shut

Nexus7K(config-if)#

switchportinherit port-profile host-port

switchport access vlan <vlan>

Nexus7K(config)#port-profile type ethernet trunk-port

state enable

switchport

switchport mode trunk

switchport trunk native vlan <vlan>

spanning-tree port type network

no shut

Nexus7K(config-if)#

switchportinherit port-profile trunk-port

switchport trunk allow vlan <vlans>

Note: Port-Profiles are live profiles (modify or

delete port-profiles will be reflected on the

assigned interfaces)

Nexus7K(config)#

spanning-tree pathcost method long

Unsupported STP features

PVST+

Nexus7K# sh run int e<mod/port> expand-port-profile


Multiple Spanning-tree (MST)

Determine the maximum number of MST instances

Develop the VLAN plan

Map the entire ranges of VLANs to pre-determined MST instances

agg1a agg1b

Acc1

VLANs Description

IST 0 3968-4047,

4094

Internal VLANs

IST 1 1-299 Production 1

IST 2 300 -599 Production 2

IST 3 600 -699 Service

State / Keepalive

IST 4 700-3967,

4048-4093

Reserved for

future

Primary

root

IST 0,1,3

Primary

root

IST 2,4

IST1

IST2

IST2

IST1

IST3

Acc2

Nexus7K#

spanning-tree mst configuration

instance 1 vlan 1-299



instance 4 vlan 700-3967,4048-4093

name <name>

revision <rev>

!

spanning-tree mode mst

VLAN numbers are provided as an example

Plan ahead to avoid future MST

configuration changes


MST (cont.)

Configure aggregation switches as STP MST primary root and STP MST secondary root

Disable PVST+ simulation for tighter administration control

PVST+ simulation can be disabled per interface or globally

Nexus7K(config)#

no spanning-tree mst simulate pvst global

Nexus7K(config-if)#

spanning-tree mst simulate pvst disable

Nexus7K# sh spanning-tree active | i Bound

Po1 Root FWD 1000 128.4096 P2p Bound(RSTP)

Nexus7K-1a(config)#

spanning-tree mst 0,1,3 priority 8192

spanning-tree mst 2,4 priority 16384

Nexus7K-1b(config)#

spanning-tree mst 2,4 priority 8192

spanning-tree mst 0,1,3 priority 16384



Virtual Port-Channel (vPC)


Virtual Port-Channel (vPC)vPC Terminology

vPC peer - a vPC switch, one of a pair

vPC member port - one of a set of ports (port channels) that form a vPC

vPC - the combined port channel between the vPC peers and the downstream device

vPC peer-link (vPC_PL) - synchronize state between vPC peer devices (must be 10GE port-channel)

vPC peer-keepalive link (vPC_PKL) - detect the status of vPC peer devices

CFS - Cisco Fabric Services protocol, used for state synchronization and configuration validation between vPC peer devices

vPC VLANs - VLANs carried over the peer-link

Non-vPC VLANs - VLANs not carried over the peer-link

vPC orphan-ports - non vPC ports that are mapped to the vPC VLANs

vPC_PL

vPC_PKL

vPC

orphan portvPC member

port

agg1a

Access1 Access2

agg1b

Core1Core2

CFS Protocol

agg1bagg1a


vPC (cont.) vPC failure convergence summary

No impact with vPC peer-keepalive link failure

No impact with supervisor failover or ISSU

When vPC channel member fails, traffic is re-hashed over existing member

When half of the vPC channel fails, traffic is re-hashed / re-routed to vPC peer

Some traffic will traverse across vPC peer-link

When vPC primary switch fails, traffic is re-hashed / re-routed to vPC secondary switch

vPC peer-switch can be implemented to eliminate STP convergence

When vPC secondary switch fails, traffic is re-hashed / re-routed to vPC primary switch

vPC_PL

vPC_PKL agg1bagg1a

Acc2Acc1

Core

vPC_PL

vPC_PKL agg1bagg1a

Acc2Acc1

Core

vPC SecondaryvPC Primary

Half or vPC channel fails

vPC primary device fails

New

root

rootOld

root

convergenceconvergence



Operation

secondary

vPC (cont.) vPC failure convergence summary

If vPC peer-link fails, the secondary vPC peer suspends local vPCs and shuts down SVIs of vPC VLANs

Single-homed devices connected to vPC secondary devices will be isolated

Dual-active does not occur if peer-keepalive link fails after vPC peer-link failure

If vPC peer-keepalive link fails first and vPC peer-link fails later (or both fail together), both vPC peers become active

Need to bring up both vPC_PKL and vPC_PL to recover from dual active state

After recovery the “configured” vPC secondary peer will remain as the operational primary

A vPC role change requires bring down/up peer-link

Auto vPC preempt is not supported since role change is disruptive

agg1bagg1a

vPC_PL

vPC_PKL

vPC dual

active!

Isolated!!

Acc2

agg1bagg1a

Acc2

vPC Primary

Acc1

Acc1

Core


vPC_PL

vPC_PKL

vPC peer-link fails

vPC Secondary

Operation

primary

Both vPC_PL and vPC_PKL fail

Shut SVIs

Route all traffic

to agg1a


Core2

vPC (cont.) Implementation Best Practices

Utilize diverse 10GE modules to form vPC peer-link

Support a mix of 8 and 32 port 10GE modules

Single 10GE module implementation is supported but not recommended

Implement physical vPC peer-link ports in dedicated rate-mode

Shared rate-mode is supported but not recommended

Use a dedicated link for vPC peer keepalive link and assign it to a separate VRF

If mgmt0 interface is used as vPC keepalive link, it should be connected to an OOB mgmt network

Back-to-back mgmt0 connection should only be used in single supervisor implementation

Do not use SVI interface between vPC peer-link as vPC keepalive link

routing peer agg1b

vPC Secondary

role pri 16384

domain 1

agg1a

vPC Primary

role pri 8192

domain 1

Acc1a

vPC domain 2

Acc1b

Core1

Acc2

vPC_PL

vPC_PKL

agg1bagg1avPC_PL

vPC_PKLMgmt0

X 2Mgmt0

X 2

Mgmt

Network


Core2

vPC (cont.) Implementation Best Practices

routing peer agg1b

vPC Secondary

role pri 16384

domain 1

agg1a

vPC Primary

role pri 8192

domain 1

Acc1a

vPC domain 2

Acc1b

Core1

Assign unique vPC domain-ID for each pair of vPC peer devices in the same “L2 domain”

Acc2

vPC_PL

vPC_PKL

Nexus7K# show vpc orphan-ports

Define vPC primary peer role with lower role priority

Do not configure HSRP tracking, implement IGP routing over vPC peer-link to re-route traffic in case of complete uplink failure

Enable vPC delay restore (supported and enabled by default in NX-OS 4.2)

Match vPC with port-channel number

Nexus7K# sh vpc role | i "vPC system-mac”

vPC system-mac : 00:23:04:ee:be:01

Dual home all devices to vPC domain using vPC

If required, connect single attached devices to vPC primary peer and leverage “dual-active exclude interface-vlan”

Nexus7K(config-vpc-domain)# dual-active exclude interface-vlan 11


Nexus7K-Agg1a#

feature vpc

feature lacp

feature ospf

feature interface-vlan!

vlan 98,99,<vPC vlans>!

vrf context vpc-keepalive!

int e3/48

vrf member vpc-keepalive

ip address 10.1.1.1/30

no shut!

vpc domain 1

role priority 8192

peer-keepalive destination 10.1.1.2 source

10.1.1.1 vrf vpc-keepalive !

int e1/1,e2/1

rate-mode dedicated

switchport


channel-group 1 mode active

no shut!

int port-channel 1

switchport


vpc peer-link

spanning-tree port type network!

int e3/1-2

switchport



no shut!

int port-channel 11

switchport


switchport trunk allowed vlan remove 98-99

vpc 11!

router ospf 1!

interface vlan 98

ip address 10.1.98.1/30

ip router ospf 1 area 0

ip ospf network point-to-point

no shut!

interface vlan 99

ip address 10.1.99.1/30



no shut

vPC (cont.) Sample vPC Configuration

routing peer

VPC 11

Po11 Po11

e1/1,e2/1

e3/1-2

e1/1,e2/1

e3/1-2

e3/48 e3/48

V98,V99

agg1bagg1a

Access1 Access2

vPC_PL

vPC_PKL


Nexus7K-Agg1b#

feature vpc

feature lacp

feature ospf


vlan 98,99,<vPC vlans>!

vrf context vpc-keepalive!

int e3/48

vrf member vpc-keepalive


no shut!

vpc domain 1

role priority 16384

peer-keepalive destination 10.1.1.1 source

10.1.1.2 vrf vpc-keepalive !

int e1/1,e2/1

rate-mode dedicated

Switchport



no shut!

int port-channel 1

switchport


vpc peer-link

spanning-tree port type network!

int e3/1-2

Switchport



no shut!

int port-channel 11

switchport


switchport trunk allowed vlan remove 98-99

vpc 11!

router ospf 1!

interface vlan 98

ip address 10.1.98.2/30



no shut!

interface vlan 99

ip address 10.1.99.2/30



no shut

vPC (cont.) Sample vPC Configuration

routing peer

VPC 11

Po11 Po11

e1/1,e2/1

e3/1-2

e1/1,e2/1

e3/1-2

e3/48 e3/48

V98,V99

agg1bagg1a

Access1 Access2

vPC_PL

vPC_PKL


vPC (cont.) STP Best Practices

Do not disable STP !!

Configure aggregation vPC peers as root and secondary root

If vPC peer-switch is implemented, both vPC peers will behave a single STP root

Align STP primary root, HSRP active router and PIM DR with vPC primary peer

BA is enabled by default on vPC peer-link

Do not enable Loopguard and BA on vPC (disabled by default)

Enable STP port type “edge” and port type “edge trunk” on host ports

Enable STP BPDU-guard globally

Disable STP channel-misconfig guard if supported by access switches

BPDU-guardBPDU-guard

Port Type

Edge / Edge Trunk

routing peeragg1bagg1a

Acc1 Acc2

Disable STP channel-

misconfig guard

vPC primary

VLAN 1-4094 root

MST 0 – 3 root

HSRP Active

PIM DR

vPC primary STP

VLAN 1-4094 sec root

MST 0 – 3 sec root

HSRP Standby

vPC_PL

vPC_PKL


vPC (cont.) STP Best Practices

BPDU-guardBPDU-guard

Port Type

Edge / Edge Trunk

routing peeragg1bagg1a

Acc1 Acc2

Implement consistent STP mode in the same L2 domain

Configure the allowed VLANs on trunk interfaces

Utilize MST to scale L2 domain

Logical port limitation is applicable with vPC implementation

Plan ahead to avoid future configuration changes that can trigger vPC type-1 consistency failure

Sample global type-1 parameters include MST region configuration, STP mode, STP global configuration, STP state, etc

Sample of the interface type-1 parameters include port-channel mode, trunk configuration on vPC channel, link speed, etc

Disable STP channel-

misconfig guard

vPC primary

VLAN 1-4094 root

MST 0 – 3 root

HSRP Active

PIM DR

vPC primary STP

VLAN 1-4094 sec root

MST 0 – 3 sec root

HSRP Standby

vPC_PL

vPC_PKL


vPC (cont.) Special Considerations

It is recommended to configure “peer-gateway” to enable vPC peer devices to act as the gateway for packets destined to the vPC peer device's MAC address (supported in NX-OS 4.2)

The feature is necessary to support NAS devices, load-balancers, and other devices which reply to sender’s mac-address instead of HSRP virtual mac-address

Disable IP redirects on all SVIs of the vPC VLANs to avoid generating IP redirect messages if “peer-gateway” is configuredNexus7K(config)# vpc domain <domain-id>

Nexus7K(config-vpc-domain)# peer-gateway

Note: Disable IP redirects on all interface-vlans of this vPC domain for correct operation of this feature!

interface vlan <vlan x>, vlan <vlan y>

no ip redirects

Upgrade to NX-OS 4.2 to provides interoperability support for appliances which use unicast ARP requests to monitor gateway reachability (enabled by default)

Unicast ARP requests received by the HSRP standby router will be forwarded via the vPC peer-link to HSRP active router


vPC (Cont.)Connect layer-3 routing device

When connect layer-3 routing devices to a vPC domain, do not form routing adjacency with vPC peer devices over the vPC peer-link (unsupported design)

routing

peer

routing

peervPC

vPC

agg1bagg1a

Acc1a

vPC

agg1bagg1a

Acc1arouting

peer

routing

peer

routing

peerrouting

peer

agg1bagg1aL3 FW L3 FW

VPC

VLANs

VPC

VLANs

IBM OSA

vPC_PL

vPC_PKL

vPC_PL

vPC_PKL

vPC_PKL

vPC_PL

L3 link

L2 link

L3 link

L2 link

L3 link

L2 link


vPC (Cont.)Connect layer-3 routing device

If dynamic routing is required to a vPC domain, L3 routed interfaces should be utilized

If L3 routed interfaces can not be used, connect L3 routing devices to a vPC domain using vPC and implement static routing to FHRP address

routing

peer

routing

peer

agg1bagg1a

Static route to FHRP

vPC

agg1bagg1a

routing

peer

routing

peer

IBM OSA

L3 FW

vPC

vPC_PKL

vPC_PL

vPC_PKL

vPC_PL

L3 link

L2 link

L3 link

L2 link

Dynamic routing

L3L3

L3 L3


vPC (Cont.)Service appliances

Dedicate a L2 port-channel for the service appliances’ state and keepalive VLANs

Connect service appliances to vPC domain via vPC and configure static routes to HSRP address

If port-channel is not supported, this can create orphan ports

Implementing a separate L2 port channel for non-vPC VLANs can be used to

Support single attached devices without creating orphan ports by mapping interfaces to non-vPC VLANs and assign them to different VRFs

Support both routed and bridged traffics

routing

peer

Static

FHRP

agg1bagg1aL3 FW L3 FW

VPC

VLANsVPC

VLANs State/Keepalive

Static

FHRP

agg1bagg1a

L3 FW L3 FWVPC

VLANs

VPC

VLANs

State/Keepalive

Static

VIP

Static

VIP

Static

FHRP

Static

FHRP

vPC orphan ports

vPC_PL

agg1bagg1a

L3 FWL3 FW

Non-vPC

VLANs

State/Keepalive

Non-VPC VLANs Non-vPC

VLANs

routing

peer

vPC_PKL

vPC_PL

vPC_PKL

vPC_PL

vPC_PKL

VRF1agg1bagg1a

L3 FW L3 FW

State/Keepalive

vPC_PKL

Non-VPC VLANs

VRF2 vPC_PL

VRF1

VRF2

Non-vPC

VLANs

Non-vPC

VLANs


vPC (cont.) Single 10GE LC Implementation

If uplinks to core switches and vPC peer-link are implemented with a single 10GE LC, enable object tracking to prevent traffic black-hole (supported in NX-OS 4.2)

Since either device can be operational primary, enable object tracking on both vPC switches

routing peer agg1bagg1a

Acc1 Acc2

Core

Nexus7K-1a(config)#

track 1 interface port-channel1 line-protocol

track 2 interface ethernet1/25 line-protocol

track 3 interface ethernet1/26 line-protocol!

track 10 list boolean or

object 1

object 2

object 3!

vpc domain 1

track 10

e1/25-26 e1/25-26

po1 po1

vPC 11 vPC 12

routing peer agg1bagg1a

Acc1 Acc2

Core

e1/25-26 e1/25-26

po1 po1

vPC 11 vPC 12

w/o object

tracking!


Nexus7K-1a# show int po 11

port-channel11 is down (suspended by vpc)

Nexus7K-1a# show int vlan 11

Vlan11 is down, line protocol is down

Nexus7K-1a# show track 10

Track 10

List Boolean or

Boolean or is DOWN

6 changes, last change 00:11:12

Track List Members:

object 3 DOWN

object 2 DOWN

object 1 DOWN

Tracked by:

vPCM

vPC_PKL

vPC_PL

vPC_PKL

vPC_PL

Shut SVIs

No routes!

vPC PrimaryvPC Primary

Operation

Secondary

vPC SecondaryvPC Secondary

Operation

Primary

with object

tracking!


vPC (cont.) Multi-layer vPC

If utilizing a single HSRP group for the inter-DC VLANs, configure active/standby router in one DC and configure listen/listen router in the other DC (supported in NX-OS 4.2)

Implement BPDU-filter to segment the STP domain between data centers

L2

L3

L3

Core

Aggregation

AccessDCi1b

DCi1a

DCi2b

DCi2a

VPC VPC

DC1

Agg1a

DC1

Agg1bDC2

Agg1a

DC2

Agg1b

DC1

Core1a

DC1

Core1b

DC2

Core1a

DC2

Core1b

vPC

Data

Center 1

L2

L3

L3

Core

Aggregation

Access

BPDU-filter

HSRP

Active

HSRP

Standby

HSRP

Listen HSRP

Listen

Domain 1

Domain 2 Domain 3

vPC

Domain 4

Nexus7K-DCi(config)#

int po 100

vpc 100

spanning-tree bpdufilter enable

Po 100 Po 100

Po 100 Po 100

vPC

100

Data

Center 2

Nexus7K-Agg1a(config)#

int vlan <vlan>

hsrp <group>

priority 130

Nexus7K-Agg1b(config)#

int vlan <vlan>

hsrp <group>

priority 120

Nexus7K-Agg2a(config)#

int vlan <vlan>

hsrp <group>

priority 110

Nexus7K-Agg2b(config)#

int vlan <vlan>

hsrp <group>

priority 100


vPC (cont.) vPC peer-switch

vPC peer-switch feature allows a pair of vPC peer devices to behave as a single STP device and send BPDUs from both vPC devices

Improves vPC convergence during vPC primary switch failure

Simplifies STP configuration by configuring both vPC with the same STP priority

Supports a hybrid topology of vPC and non-vPC connections by using the spanning-tree pseudo-information

Nexus7K-1a(config-vpc-domain)#

peer-switch

Nexus7K-1a(config)#

Spanning-tree vlan 1-4094 pri 8192

vPC

agg1bagg1a

Acc1a

STP root pri 8192 Bridge ID = vPC system ID

STP root

VLAN 1 - 4094 Priority 8192

MST 0 – 4 Priority 8192

Nexus7K-1a# show spanning-tree summary | i peer

vPC peer-switch is enabled (operational)

Nexus7K-1b# show spanning-tree summary | i peer

vPC peer-switch is enabled (operational)

Nexus7K-1a# sh spanning vlan 1

---deleted---

Root ID Priority 8193

Address 0023.04ee.be01

This bridge is the root

---deleted---

Po1 Desg FWD 1 128.4096(vPC peer-link) Network P2p

Nexus7K-1a# sh spanning vlan 1

---deleted---

Root ID Priority 8193

Address 0023.04ee.be01

This bridge is the root

---deleted---

Po1 Root FWD 1 128.4096 (vPC peer-link) Network P2p

Nexus7K-1b(config-vpc-domain)#

peer-switch

Nexus7K-1b(config)#

Spanning-tree vlan 1-4094 pri 8192

vPC_PKL

vPC_PL

New in NX-OS 5.0 !!

BPDUs BPDUs

No convergence


vPC (cont.) vPC restore on reload

If both vPC switches reload, by default all vPCs are suspended until peer adjacency is reestablished between vPC devices

If only one vPC device becomes operational, the local vPC ports will remain suspended

Nexus7K(config-vpc-domain)#reload restore delay <delay>

Warning:

Enables restoring of vPCs in a peer-detached state after reload, will wait for 240 seconds (by default) to determine if peer is un-reachable

vPC

agg1bagg1a

Acc1a

vPC

agg1a

Acc1a

Both vPC switches reloaded !!

w/o vPC

Restore

on Reload

Agg1b

is

down

vPC_PKL

vPC_PL

vPC_PKL

vPC_PL

vPC

agg1a

Acc1a

with vPC

Restore

on

Reload

Agg1b

is

down

vPC_PKL

vPC_PL

vPC Primary

STP root

Wait until all LCs ups12 Start timer

3 Timer expires

Bring up local

vPC ports

vPC restore on reload allows the one vPC device to assume STP / vPC primary role and bring up all local vPCs after delay timer expiration

Both vPC switches needs to be configured

The default and minimum delay timer is 240s

4

5

Nexus7K# sh vpc 11

---deleted---

11 Po11 up success Type checks were bypassed 1,10-15

for the vPC

New in NX-OS 5.0 !!



Layer-3 Features


Interior Gateway Protocol (IGP)

Enable NSF/Graceful Restart (default)

Configure IETF graceful OSPF restart on neighboring devices as Nexus 7000s only supports standard NSF

Use default IGP timers in a dual supervisor system to avoid unnecessary convergence w/ supervisor failover

Reduced IGP timers can be leverage over L2 cloud or in a single supervisor system

Bidirectional Forwarding Detection is supported in NX-OS 5.0

BFD is performed by the I/O modules

Nexus7K(config)# feature bfd

Please disable the ICMP redirects on all interfaces

running BFD sessions using the command below

'no ip redirects'

Nexus7K(config)# feature bfd

BFD Feature could not be enabled.

Please disable the address-identical IDS check for BFD Echo to be operational using the configuration command given below in the default VDC.

'no hardware ip verify address identical'

Nexus7K(config)# router eigrp 1

Nexus7K(config-router)# bfd

Nexus7K(config)# router ospf 1

Nexus7K(config-router)# bfd

Nexus7K# show bfd neighbors details

IOS(config-router)# nsf ietf


OSPFGeneral OSPF best practices

General OSPF Best Practices Notes

Enable NSF/Graceful Restart Default (IETF only)

Implement consistent auto-cost reference bandwidth default is 40G

Configure OSPF point-to-point network on point-to-point interfaces

Configure passive-interface on server VLANs

Implement routing protocol authentication

Implement OSPF route summarization

Configure deterministic router-id (loopback0)

Enable routing process on the router-id interface

Utilize OSPF stub/NSSA or totally stub/NSSA area for server VLANs

Utilize OSPF stub for IBM Open System Adaptor (OSA)

Configure intra-area transit link between the ABRs

Configure OSPF log adjacency changes disabled by default

Utilize route-map when redistributing routes default


Nexus7K(config)#

feature ospf

feature interface-vlan !

int loopback0

ip address <address>/32

ip router ospf <process> area 0

!

vlan <vlan#>!

ip prefix-list <name> seq 5 permit <net1>/<mask>

ip prefix-list <name> seq 10 permit <net2>/<mask>!

route-map <name>

match ip address prefix-list <name>

set metric <metric>!

int e<mod>/<port>

no shut

rate-mode dedicated

no ip redirects

ip address <ip address>/<mask>

ip ospf authentication message-digest

ip ospf message-digest-key <id> md5 <pw>


ip ospf network point-to-point!

interface vlan <area 0 vlan>




no shut!

interface vlan <area x vlan>


ip router ospf <process> area <area-x>


no shut!

interface vlan <server vlan>

no ip redirects


ip router ospf <process> area <area-x>

ip ospf passive-interface

no shut!

router ospf <process>

router-id <loopback>

log-adjacency-changes

redistribute static route-map <name>

auto-cost reference-bandwidth 100000

area <area-x> nssa no-summary

area <area-x> range <network/mask>

OSPF (cont.)Sample OSPF Configuration

agg1a agg1b

NSSA

Area X

Core1 Core2

agg2a agg2b

NSSA

Area Y

Backbone Area

Area 0

Summarize

routes

Summarize

routes

Summarize

routes

e1/25

Area 0 Area 0

Area xArea x


EIGRPGeneral EIGRP best practices

General EIGRP Best Practices Notes

Enable NSF/Graceful Restart default

Configure passive-interface on server VLANs

Implement routing protocol authentication

Implement EIGRP route summarization

Configure deterministic router-id (loopback0)

Enable routing process on the router-id interface

Configure EIGRP log adjacency changes default

Utilize route-map when redistributing routes default


EIGRP (cont.)Sample EIGRP Configuration

Nexus7K(config)#

feature eigrp

feature interface-vlan

!

int loopback0

ip address <address>/32

ip router eigrp 1

!

vlan <vlan#>

!



!

route-map <name>

match ip address prefix-list <name>

!

key chain <name>

key 1

key-string <string>

!

int e1/25

no shut

no ip redirects


ip router eigrp 1

ip authentication mode eigrp 1 md5

ip authentication key-chain eigrp 1 <name>

ip summary-address eigrp 1 <network>/<mask>

!

interface vlan <inter-switch vlan>


ip router eigrp 1

ip authentication mode eigrp 1 md5

ip authentication key-chain eigrp 1 <name>

ip summary-address eigrp 1 <network>/<mask>

no shut

!

interface vlan <server vlan>

no ip redirects


ip router eigrp 1

ip passive-interface eigrp 1

no shut

!

router eigrp 1

router-id <loopback>

redistribute static route-map <name>

agg1a agg1b

Core1 Core2

agg2a agg2b

Summarize

routes

Summarize

routes

Summarize

routes

e1/25

routing

peer

routing

peer


Border Gateway Protocol (BGP)

Enable NSF/Graceful Restart (default)

If full internet routing table is required, XL I/O modules should be utilized

If non-XL I/O modules are used, limit BGP table size by configuring maximum BGP AS paths and maximum BGP prefixes

Dynamic FIB TCAM allocation allows non-XL I/O modules to support up to 112k IPv4 unicast routes (supported in NX-OS 4.2 and enabled by default)

Without Dynamic FIB TCAM allocation, non-XL I/O modules support up to 56k IPv4 unicast routes

BFD is supported in NX-OS 5.0

Nexus7K(config-router)#

maxas-limit <# of AS paths>

Nexus7K(config-router-neighbor-af)#

maximum-prefix <# of prefix>

Nexus7K# hardware forwarding dynamic-allocation enable

Nexus7K# show hardware internal forwarding table utilization mod <mod>

Nexus7K# show hardware capacity forwarding | b Unicast


General Layer-3 Features

Configure extended hold timers for HSRP to support NSF during ISSU and supervisor switchovers

Not applied with sub-second timers

Configure on all HSRP routers with the same timer (default /minimum is 10s)

Sub-second FHRP timers are not recommended for a dual-sup system

Nexus7K(config)#

feature hsrp

feature interface-vlan

!

vlan <vlan>

!

hsrp timers extended-hold <time>

!

interface vlan <vlan>

description <description>

no shutdown

no ip redirects

ip address <address>/<mask>

hsrp <group>

authentication <text>

preempt delay minimum 180

priority 110

timers 1 3

ip <hsrp address>

General L3 Best Practices

Utilize per flow load-balancing (default)

Disable IP redirects

Configure HSRP preemption delay and authentication

Re-use HSRP group for all VLANs or enable HSRP v2

so HSRP group can match with the VLAN number

1s hello and 3s hold timer is recommended

If vPC is implemented, aggressive timers are not necessary

BFD is supported in NX-OS 5.0

Unsupported Layer-3 features

MPLS, NAT, Interface IP dampening, IP SLA



Security Features


Access Control List (ACL)

Utilize config session manager with atomic ACL updates for non-disruptive ACL updates

Atomic Update is enabled by default

ACL management can be simplified by utilizing object groups

Nexus7K# config session test1ip access-list vlan11-acl

no 20 …….

32 permit ….

….

verify

commit

Nexus7K(config)# no hardware access-list update atomic

Nexus7K(config)# hardware access-list update default-result permit

Nexus7K(config)# hardware access-list resource pooling mod <mod>

Nexus7K(config)#

object-group ip address <name>

10.10.1.0/24

10.10.2.0/24

……

ip access-list acl1

deny ip addrgroup <name> any

XL I/O modules support 128K ACL entries per module

Enable TCAM bank-chaining on to support large ACL

XL I/O modules supports 32K entries in each TCAM bank

Non-XL I/O modules supports 16K entries in each TCAM bank

If I/O module lacks the TCAM resources, disable atomic ACL update and optionally permit all traffic during non-atomic update

Nexus7K#

int vlan 11ip access-group vlan11-acl in


Nexus7K(config)#

no feature telnet

!

vrf context management

ip route 0.0.0.0/0 <IP address>

!

ip access-list <ACL-name>

10 remark allow specific ssh

11 permit tcp <addr>/24 any eq 22

12 permit tcp any eq 22 <addr>/24

13 deny tcp any any eq 22

14 deny tcp any eq 22 any

20 remark allow specific snmp

21 permit udp <addr>/24 any eq snmp

………..

50 permit ip any any

!

interface mgmt0


ip access-group <ACL-name> in

!

line vty

exec-timeout <time>

session-limit <session#>

line console

exec-timeout <time>

!

int cmp-mgmt module <module>

ip address <addr>/<mask>

ip default-gateway <IP addr>

Network Access

Allow only SSH remote access (default)

If telnet access is required, “feature telnet” needs to be configured

If telnet access to CMP is required, “telnet server enable” need to be configured on the CMP

Secure interface mgmt0 with ACL

CoPP does not protect interface mgmt0

ACL with the logging option is supported in NX-OS 5.0

ACL is not supported on VTY

CoPP can be leveraged to secure VTYaccess

Configure exec-timeout for VTY and console access

Nexus7K-cmp10(config)#

telnet server enable


Control Plane Policing

Implement strict control plane policing (default)

If default policy is used, run “setup” command to reapply the default policy after software upgrade between major releases

Any non-default CoPP policies need to be reapplied after setup

Future software release will generate syslog on CoPP policy changes

Tune default CoPP policy according to needs

The configured setting is per line card and not per system. If high number of I/O modules are installed, the conform rate may need to be tuned down

Future enhancement to generate syslog messages if drops exceed user configured threshold

Critical

39600

kbps

Important

1060 kbps

Mgmt

10000

kbps

Normal

680

kbps

Redirect

280 kbps

Monitoring

130 kbps

Exception

360 kbps

Undesirable

32 kbps

conform drop

Default

100

kbps

Strict (bc ) 250 ms 1000 ms 250 ms 250 ms 250 ms 1000 ms 250 ms 250 ms 250 ms

Moderate (bc) 310 ms 1250 ms 310 ms 310 ms 310 ms 1250 ms 310 ms 310 ms 310 ms

Lenient (bc) 375 ms 1500 ms 375 ms 375 ms 375 ms 1500 ms 375 ms 375 ms 375 ms

Nexus7K# show policy-map interface control-plane | inc violated

violated 59 bytes; action: drop

Nexus7K# setup

----deleted----

Configure best practices CoPP profile (strict/moderate/lenient/none) [strict]:

per module statistics


Control Plane Policing (cont.)Tuning Example

Example: Customer utilizes ICMP to monitor the network. The ICMP packet rate exceeds the default setting for the monitoring class. Increase the CIR to allow the monitoring tools to function properly.

Nexus7K(config)# policy-map type control-plane copp-system-policy

Nexus7K(config-pmap)# class copp-system-class-monitoring

Nexus7K(config-pmap-c)# police cir 200 kbps bc 1000 ms conform transmit violate drop

Nexus7K# sh policy-map int control-plane | b monitor

class-map copp-system-class-monitoring (match-any)

match access-grp name copp-system-acl-icmp

match access-grp name copp-system-acl-icmp6

match access-grp name copp-system-acl-traceroute

police cir 200 kbps , bc 1000 ms

Nexus7K(config)# policy-map type control-plane copp-system-policy

Nexus7K(config-pmap)# class copp-system-class-normal

Nexus7K(config-pmap-c)# police cir 680 kbps bc 400 ms conform transmit violate drop

Nexus7K# sh policy-map interface control-plane | b normal

class-map copp-system-class-normal (match-any)

match access-grp name copp-system-acl-dhcp

match redirect dhcp-snoop

match protocol arp

police cir 680 kbps , bc 400 ms

Example: The newly active LB appliance sends out large amount of gratuitous ARPs after a failover and exceed the default setting for the normal class. Increase the burst interval allow to interoperate with the LB appliances


Control Plane Policing (cont.)Tuning Example

This is a sample CoPP configuration to limit SSH access to VTY. Only SSH traffic to and from the management network is allowed to access the Nexus 7000

Nexus7K(config)#

ip access-list copp-system-acl-allow

10 permit tcp <IP network>/24 any eq 22

20 permit tcp any eq 22 <IP network>/24!

ip access-list copp-system-acl-deny

1 remark ### catch-all for modified mgmt traffic ###

10 permit tcp any any eq 22

20 permit tcp any eq 22 any!

class-map type control-plane match-any copp-system-class-management

no match access-group name copp-system-acl-ssh!

class-map type control-plane match-any copp-system-class-management-allow

match access-group name copp-system-acl-allow

class-map type control-plane match-any copp-system-class-management-deny

match access-group name copp-system-acl-deny !

policy-map type control-plane copp-system-policy

class copp-system-class-management-allow insert-before copp-system-class-normal

police cir 3000 kbps bc 250 ms conform transmit violate drop

class copp-system-class-management-deny insert-before copp-system-class-normal

police cir 3000 kbps bc 250 ms conform drop violate drop


Hardware Rate-Limiter

Hardware-limiters complement CoPP to protect the CPU (enabled by default)

Rate limit supervisor-bound egress exception and egress redirected traffic

Configure on the default VDC and apply to all VDCs

The configured setting is per line card

Modify and enable hardware rate-limiters according to needs

Rate Limiter Class Default

(pps)

Layer-3 MTU 500

Layer-3 TTL 500

Layer-3 control 10,000

Layer-3 glean 100

Layer-3 multicast directly-

connected

3,000

Layer-3 multicast local-

groups

3,000

Layer-3 multicast rpf-leak 500

Layer-2 storm-control Disabled

Access-list-log 100

Copy 30,000

Receive 30,000

Layer-2 port-security Disabled

Layer-2 mcast-snooping 10,000

Layer-2 vpc-low 4,000

Nexus7K# sh hardware rate-limiter

Units for Config: packets per second

Allowed, Dropped & Total: aggregated since last clear counters

Rate Limiter Class Parameters

-------------------------------------------------

layer-3 mtu Config : 500

Allowed : 0

Dropped : 0

---deleted---

Nexus7K(config)#

hardware rate-limiter layer-2 <class> <packets/s>


Packet Sanity Checks

The IDS check performs sanity checks on the IP headers to protect the network and the system (enabled by default)

In NX-OS 5.0, the system generates syslogs on IDS drops (max is one every 30 min)

It is recommended to disable fragment IDS check since some applications sends IP fragment Packets with DNF bit

Fragment IDS check is disabled by default in NX-OS 5.0

Disable individual IDS checks as required

Ex. If BFD is configured, disable “address identical” IDS check

Nexus7K# show hardware forwarding ip verify

IPv4 and v6 IDS Checks Status Packets Failed

-----------------------------+---------+------------------

---deleted---

address identical Enabled 0

---deleted---

fragment Enabled 0

---deleted---

Nexus7K(config)# no hardware ip verify fragment

Nexus7K(config)# no hardware ip verify address identical


Nexus7K(config)#

feature ospf


vrf context inside!

interface vlan 10


ip router ospf 1 area 0!

interface vlan 20

mac-address <mac-address>

vrf member inside



!

router ospf 1

vrf inside

Virtual Routing & Forwarding (VRFs)

VRFs can be utilized to provide network segmentation within VDC

VRF Import/Export is not supported

External devices or connections can be used to interconnect multiple VRFs

Policy based routing (PBR) provides the option to interconnect multiple VRFs without utilizing external connections

When forming routing peer between VRFs within the same VDC, static router mac-address must be configured to avoid address conflict

The mac-address conflict problem is masked on releases prior to 4.2(4)

Default

VRF

Inside

VRF

VLAN 10

VLAN 20

L2 FW

Default

VRF

Inside

VRF

VLAN 10

VLAN 20

L3 FW


Nexus7K(config)#

feature pbr


vlan 10,20!

route-map VRF-A_to_VRF-B permit 10

match ip address VRF-A_to_VRF-B

set vrf VRF-B

!

route-map VRF-B_to_VRF-A permit 10

match ip address VRF-B_to_VRF-A

set vrf VRF-A!

vrf context VRF-A

vrf context VRF-B!

interface vlan 10

vrf member VRF-A


ip policy route-map VRF-A_to_VRF-B

no shutdown!

interface vlan 20

vrf member VRF-B


ip policy route-map VRF-B_to_VRF-A

no shutdown!

ip access-list VRF-A_to_VRF-B

10 permit ip 10.1.1.0/24 10.1.2.0/24

!

ip access-list VRF-B_to_VRF-A

10 permit ip 10.1.2.0/24 10.1.1.0/24

VRFs (cont.)Sample PBR configuration

VRF-AVRF-B

VLAN 10

10.1.1.0/24VLAN 20

10.1.2.0/24



Quality of Service (QoS)


Quality of ServiceConfiguration Example 1

Applications COS 1P3Q4T (GE) 1P7Q4T (10GE)

OSPF, BGP, HSRP, 6, 7 Q3T3 Q7T3

Voice over IP 5 PQ PQ

HD Video Conference 4 PQ PQ

SD Video Conference 4 PQ PQ

Voice/Video Signaling 3 Q3T2 Q7T2

SSH, Telnet 3 Q3T2 Q7T2

DLSW, TACACs 2 Q2T3 Q2T3

Oracle, Citrix 2 Q2T3 Q2T3

TFTP, FTP 1 Q1T2 Q1T2

Default 0 Q1T3 Q1T3

By default, traffic with COS 5 – 7 are mapped to priority queue and traffic with

COS 0 – 4 are mapped to default queue. The default queue is assigned with

82% of the queue-limit and 25% of remaining bandwidth. A single tail drop

threshold will be used by the default queue.

This example modifies the default egress queuing based on the following table


Nexus7K(config)#

class-map type queuing match-any 1p3q4t-out-pq1

match cos 4, 5

class-map type queuing match-any 1p3q4t-out-q3

match cos 3,6-7

class-map type queuing match-any 1p3q4t-out-q2

match cos 2

class-map type queuing match-any 1p3q4t-out-q-default

match cos 0-1!----------------------------------------------------------------------------------------------------------------------------

policy-map type queuing GE-Outbound

class type queuing 1p3q4t-out-pq1

priority level 1

queue-limit percent 15

class type queuing 1p3q4t-out-q3

bandwidth remaining percent 25


queue-limit cos 7 percent 100



class type queuing 1p3q4t-out-q2



class type queuing 1p3q4t-out-q-default



random-detect cos-based

random-detect cos 1 minimum-threshold percent 60 maximum-threshold percent 100

random-detect cos 0 minimum-threshold percent 80 maximum-threshold percent 100!

interface e<mod>/1 – 48, e<mod>/1 – 48

service-policy type queuing output GE-Outbound

Quality of Service (cont.)Configuration Example 1

Configure on the default VDC

Configure on the specific VDC

Apply service-policy on all GE interfaces

System Queuing Class Maps


Quality of Service (cont.)Configuration Example 2

Nexus7K(config)#

policy-map type queuing 10ge-reset-cos

class type queuing 8q2t-in-q-default

set cos 0

bandwidth percent 100!

policy-map type queuing ge-reset-cos

class type queuing 2q4t-in-q-default

set cos 0

bandwidth percent 100!

ip access-list for-untrusted

10 permit ip any any

class-map type qos match-all reset-dscp

match access-group name for-untrusted

policy-map type qos reset-dscp

class reset-dscp

set dscp 0!

interface e<mod/port>

service-policy type qos input reset-dscp

service-policy type queuing input 10ge-reset-cos

By default, the port is set to trust DSCP and COS. This example configures a port as untrusted port by setting DSCPand COS to 0.

Configure on the specific VDC



System Management Features


Simple Network Management Protocol

Apply ACL to control SNMP access

SNMP ACL is supported in NX-OS 4.2

Use CoPP to limit access prior to NX-OS 4.2

Configure SNMP traps notification

Enable SNMP traps and specify host receivers for SNMP traps

Specify the correct VRF (Management VRF is the default)

Nexus7K(config)#ip access-list <ACL-name>

1 remark <remark>

10 permit ip <network 1> <mask> any

20 permit ip <network 2> <mask> any!

snmp-server contact <contact>

snmp-server location <location>

snmp-server host <address> traps version <ver> <commu>

snmp-server source-interface traps <interface>

snmp-server host use-vrf <vrf-name>

snmp-server enable traps

snmp-server community <RO-string> rosnmp-server community <RO-string> group use-acl <ACL-name>

snmp-server community <RW-string> rwsnmp-server community <RW>-string group use-acl <ACL-name>


Nexus7K(config)#ntp server <address1> prefer use-vrf <vrf>

ntp server <address2> use-vrf management

ntp source-interface <interface>

clock timezone <zone> <hour offset> <min offset>

clock summer-time <zone> ….

Network Time Protocol (NTP)

NTP is only configured on the default VDC

Synchronize time with NTP servers

Prefer primary server with “prefer” key word

NTP server mode is not supported

Specify the source interface

Specify the correct VRF (default VRF is the default)

Nexus7K-1(config)#cfs ipv4 distribute

ntp distribute

ntp server ....

ntp commit

Nexus7K-2(config)#cfs ipv4 distribute

ntp distribute

CFS can be utilize to distribute NTP configuration

NTP enhancements in NX-OS 5.0

NTP Authentication, NTP ACL, NTP logging


Nexus7K(config)#

vlan 100

remote-span!

monitor session 1

description <description>

source interface <interface>

filter vlan <vlan range>

destination interface <int range>

no shut!

monitor session 2

source vlan 100

….!

monitor session 3

source interface sup-eth 0

destination interface <interface>

shut!

int <interface>

switchport

switchport monitor

no shut!

int <interface>

switchport

switchport monitor

Switchport mode trunk

Switchport trunk allowed vlan <vlan>

no shut

Switched Port Analyzer (SPAN)

Configure up to 18 SPAN session templates to simplify operation

2 active bidirectional SPAN sessions are supported per system

Virtual SPAN (VSPAN) can be used to scale the SPAN session limitation

SPAN source can be a combination of interfaces and VLANs

Configure SPAN destination port as a “monitor” port

Monitor supervisor inband port to troubleshoot control plane issues

RSPAN VLANs can be used as source VLANs or extended to another switch

Unsupported SPAN related features

VACL capture, ERSPAN, RSPAN


Nexus7K(config)#logging server <addr1> <lvl> prefer use-vrf <vrf>

Nexus7K(config)#logging server <addr2> <lvl> use-vrf <vrf>

Logging

Customize logging level for individual features as necessary

Nexus7K# show logging onboard status

Nexus7K# show logging onboard mod 1 ?

Nexus7K(config)#no hw-module logging onboard <parameter>

Parameter Default

Console Enabled, level 2

Monitor Enabled, level 5

Log file Enabled, level 5

Name=Messages

Size = 1G

Module Enabled, level 5

Time-stamp Seconds

Syslog Disabled

Nexus7K(config)#logging level spanning-tree <level>

Nexus7K(config)#logging timestamp milliseconds

Set the logging time-stamp units to millisecond

Debug logging is always time-stamped to microsecond

Enable logging to syslog servers

Standard level recommendation is notification

Support up to 3 syslog servers

Specify the correct VRF(default VRF is the default”)

Enable link-status and trunk-status logging globally (default)

Onboard Failure Logging (OBFL) is enabled by default


Nexus7K(config)#

feature tacacs+

!

username admin password <password> role network-admin

username netop password <password> role network-operator

username <name> password <password> role vdc-admin

username <name> password <password> role vdc-operator

!

tacacs-server key <unencrypted key>

ip tacacs source-interface loopback0

tacacs-server host <IP address1>

tacacs-server host <IP address2>

tacacs-server directed-request!

aaa group server tacacs+ <group-name>

server <IP address tacacs-1>

server <IP address tacacs-2>

use-vrf <vrf-name>

!

aaa authentication login default group <group-name>

aaa authentication login console local

aaa authorization config-commands default group <group-name> local

aaa authorization commands default group <group-name> local

aaa accounting default group <group-name>

aaa authentication login error-enable

AAASample Configuration


Other System Management Features

Ethanalyzer can be used to capture packets to or from the supervisor

Capture traffic on inband or management interface

Nexus7K# ethanalyzer local interface ?

inband Inband/Outband interface

mgmt Management interface

Leverage Netflow feature to provide statistics for accounting, network monitoring, and network planning

It is recommended to use Netflow in sampled mode on high BW interfaces to reduce CPU

Netflow supports version 5 and 9 export format

Version 9 to support variable field specification format, IPv6, Layer 2, MPLS field and more efficient network utilization


Nexus7K(config)#

feature netflow!flow record Netflow-Record-1description <description>match ipv4 source addressmatch ipv4 destination addressmatch transport destination-portcollect counter bytescollect counter packets!flow exporter Netflow-Exporter-1description <description>destination <destination IP>source <source interface>version 9!flow monitor Netflow-Monitor-1description <description> record Netflow-Record-1exporter Netflow-Exporter-1!interface ethernet <mod>/<port>ip flow monitor Netflow-Monitor-1 input

Other System Management Features (cont.)Sample Netflow Configuration

Nexus7K(config)#

sampler Netflow-Sampler-1

mode 1 out-of <number>

!

interface ethernet <mod>/<port>

no ip flow monitor Netflow-Monitor-1 input

ip flow monitor Netflow-Monitor-1 input sampler Netflow-Sampler-1


Q & A


Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily.

Receive 20 Cisco Preferred Access points for each session evaluation you complete.

Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.

Don’t forget to activate your

Cisco Live and Networkers Virtual

account for access to all session

materials, communities, and on-demand

and live activities throughout the year.

Activate your account at any internet

station or visit www.ciscolivevirtual.com.

http://www.ciscolivevirtual.com/


Check the Recommended Reading brochure for

suggested products available at the Cisco Store

Enter to Win a 12-Book Libraryof Your Choice from Cisco Press

Visit the Cisco Store in the World of Solutions, where you will be asked to enter this Session ID code



Hardware OverviewSupported Transceivers

As of NX-OS release 5.0

Product Number Min SW

SFP-10G-SR 4.0(1)

SFP-10G-LR 4.0(3)

SFP-10G-ER 5.0(2)

X2-10GB-LRM 5.0(2)

X2-10GB-SR 5.0(2)

X2-10GB-LR 5.0(2)

X2-10GB-ER 5.0(2)

DWDM-X2-<> 5.0(2)

Product Number Min SW

SFP-GE-S / GLC-SX-MM 4.1(2)

SFP-GE-L / GLC-LH-SM 4.1(2)

SFP-GE-Z / GLC-ZX-SM 4.1(2)

SFP-GE-T / GLC-T 4.2(1)

CWDM-SFP- 4.2(1)

DWDM-SFP- 4.2(1)

Documents

Deploying Nexus 7000 in Data Center Networksd2zmdbbm9feqrf.cloudfront.net/2010/usa/pdf/BRKDCT-2951.pdf · Cisco Nexus 7000 Switch Architecture ... Deploying Virtual Port Channel in