Nexus 5000 Architecture

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 1

Nexus 5000 ArchitectureNexus 5000 Architecture


Agenda

System Hardware Overview

Internal Architecture

Fabric Data Path

Lossless data path

Forwarding and Policy Enforcement


Nexus 5000 Solution Components

• VM-Optimized network services

• End Port Virtualization

• Unified I/O with 10Gb and FCoE

• Delivers increased, security operational agility, and better utilization of network assets

VirtualizationNetworking

VirtualizationNetworking

MACB

MACA

MACC

A & B C

End nodes

LAN

MACB

MACA

MACC

A & B C

End nodes

LAN

• L2 Multi-pathing for increased bandwidth, scalable L2 domains

• Priority Flow Control –based Lossless fabric

• Congestion Management

• DCE auto-negotiation

Lossless fabric & Greater Data Center Scale

Data Center Ethernet (DCE)

Data Center Ethernet (DCE)

LAN

N5000

Active-Active

MACB

MACA

• Unified fabric for LAN, SAN, HPC/IPC

• Enables FC connectivity across more servers

• Fewer switches, fewer points of management

• Significant cable and adapter reduction

• Lower cost, power, cooling

FCoEFCoE

LAN SAN BSAN ALAN SAN BSAN A

N5000

StandardsStandards

Eco System PartnersEco System Partners

• 10GE L2 non-blocking, wire-speed switch

• Competitive price-performance

• Low latency < 3.2s

• Lossless

• Unified fabric

• Future proof

Wire speed 10GE Access Switch

Wire speed 10GE Access Switch

LAN

Access LayerN5000


System Hardware Overview


Nexus 5000 Product PortfolioIndustry’s First I/O Consolidation Virtualization Fabric for

Enterprise Data CenterIndustry’s First I/O Consolidation Virtualization Fabric for

Enterprise Data Center

OS

Cisco Fabric Manager and Cisco Data Center Network Manager

Cisco NX-OS

FC + Ethernet • 4 Ports 10GbE/FCoE • 4 Ports 1/2/4G FC

Mgmt

ExpansionModules

Ethernet • 6 Ports

10GE/FCoE

Eco System PartnersEco System Partners

Fibre Channel • 8 Ports 1/2/4G FC

28-Port 1RU Switch 56-Port 2RU Switch


Rear PanelsNX5020NX5020

Expansion Module(s)

Cables connect in the rear for ease of server wiringCables connect in the rear for ease of server wiring

Power Entry

Base 10GE 10/100/1000

Out of Band Mgmt Console

All 10GE ports are FCoE capable!

Support for 1 GE Support for crypto


Front PanelsNX5020NX5020

N+1 redundant fans

Replaceable components on the front for easy accessReplaceable components on the front for easy access

Dual redundant power supplies


Power

NX5020Power Supply

NX5020Power Supply

Fully redundant, load sharing and hot swappableFully redundant, load sharing and hot swappable

Maximum Power – 750WTypical Operating Power – 480WAC Input - 110/208 VoltsEfficiency - 82-88%Protection - 110%-150% max load


Expansion Modules

Ethernet Expansion Module

Ethernet Expansion Module

CombinationExpansion Module

CombinationExpansion Module

Six 10G Ethernet

Four 10G Ethernet

Four 1/2/4GFibre Channel

Fibre ChannelExpansion Module

Fibre ChannelExpansion Module Eight 1/2/4G

Fibre Channel


Cooling

NX5020Cooling Module

NX5020Cooling Module

Max RPM - 12KFailover - N+1Op Temp - 0 to 40 CHumidity - 95% non-condensingElevation - 10K feet


SFP+ Transmission Media

SFP+ CuSFP+ Cu

CableTransceiver

Latency (link)Power

(each side)DistanceTechnology

Twinax ~0.25 s~0.1W10mSFP+ CUCopper

MM OM2MM OM3

~0.1 s1W82m300m

SFP+ SRshort reach

MM OM2MM OM3

~0.1 s1W10m100m

SFP+ USRultra short reach

Cat6Cat6a/7Cat6a/7

2.5s2.5s1.5s

~8W~8W~4W

55m100m30m

10GBASE-T

•Low power consumption•Low cable cost•Low transceivers latency•Low error rate (10 exp-17)


Internal Architecture


Hardware Architecture

10 GE& FC

Inte

l 31

00

PC

I Co

ntro

ller

IntelLV Xeon

(1.66 GHz)

FLASH

NVRAM

Serial

PCIe

1GE

10 GE Interfaces

10 GE Interfaces 10 GEInterfaces

RS-232 Console

10/100/1000 Management

XAUI

XFI

Dual NICXAUI

XFI

UnifiedPort

Controller

XAUI

XFI

UnifiedPort

Controller

UnifiedPort

Controller

10 GE

UnifiedPort

Controller

UnifiedPort

Controller

UnifiedPort

Controller

1/2/4 Gbps Fibre Channelto Storage Network

Memory

UnifiedCrossbar

Fabric

UnifiedPort

Controller

Dual NIC

SFP+xcvr

SFP+xcvr

SFP+xcvr

SFP+xcvr

SFP+xcvr

SFP+xcvr

SFP+xcvr

SFP+xcvr

SFP+xcvr

SFP+xcvr

SFPxcvr

SFPxcvr

SFPxcvr

SFPxcvr

SFP+xcvr

SFP+xcvr

SFP+xcvr

SFP+xcvr

SFP+xcvr

SFP+xcvr

SFP+xcvr

SFP+xcvr

SFP+xcvr

SFP+xcvr


Supervisor Details

CPU 1.66 GHz Intel LV Xenon - LF80538KF0281M

IO Chip Set Intel 3100 South Bridge for embedded applications

DRAM 2 GBytes of DDR2 400 (PC2 3200) in two DIMM slots

Program Store 1 GBytes of USB based (NAND) Flash

Boot/BIOS 2 Mbytes of EEPROM with locked recovery image

On-board Fault Log 64 MBytes of Flash for failure analysisKernel Stack traces, boot record and fault logs

NVRAM 2 Mbytes of SRAM – Syslog and licensing information

Secure Keystore Renesas AE46C1 – Credentials and secure RNG

Management Interfaces RS-232 console port – console0

10/100/1000BASE-T – mgmt0 partitioned from inband VLANs


Unified Crossbar Fabric

58 port crossbar and scheduler

3 unicast and 1 multicast crosspoints

Central tightly coupled scheduler

Request, propose, accept, grant, acknowledge semantics

Packet enhanced iSLIP scheduler

Distinct unicast and multicast schedulers

Eight classes of service

Egress buffer credits

DWRR class of service

DWRR ingress interface

Total SRAM 24.6 Mbits

Gates 12.4 Million

Transistors ~200 Million

Metal Layers 7

Signal Pins 1286

SerDes 232 @ 3.75Gbps


Unified Port Controller

Media Access Controllers

1/10G Ethernet and 1/2/4G Fibre Channel

Packet Buffering and Queuing

Total of 1.875 MBytes used in four slices

Forwarding Controller

Ethernet, Fibre Channel

Layered policy engine

Four data path slices

One 1/10G Ethernet or two 1/2/4G Fibre Channel ports

Connects to one Altos port

All switching done in Altos crossbar

480 KBytes of buffering

Total SRAM 35 Mbits

Total TCAM 1 Mbit

Logic Gates 18 Million

Transistors ~300 Million

Metal Layers 7

Total Pins 900

SerDes 32 @ 3.75Gbps

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 17Unified Crossbar Fabric


Slice 4Slice 2 Slice 3

Switch ASIC Architecture

Slice 1

Forwarding

?

1/10G MAC

Transceiver

VirtualQueues

EgressQueues

PacketBuffer

VirtualQueues

EgressQueues

PacketBuffer


Slice 2 Slice 3Slice 1

VirtualQueues

EgressQueues

PacketBuffer

Slice 4

4 @ 3.75G – 12Gbps 4 @ 3.75G – 12Gbps 4 @ 3.75G – 12Gbps

XAUI – 10 Gbps4 @ 3.125G

Transceiver

XAUI – 10 Gbps4 @ 3.125G

1/10GE Attached Server

10GE LAN Uplink

58 source busses in total

Parsing &Editing

Forwarding

?

ForwardingParsing &

Editing

1/10G MACFC

MAC

SAN B

FCMAC

Fibre ChannelSAN Uplinks

1/2/4G Fibre Channel1 @ 1.0625/2.125/4.25G

Parsing &Editing

Fabric Buffer Fabric Buffer Fabric BufferUnicast and

Multicast Schedulers


Switch Fabric Data Path


Altos

Gatos

Port 4

Data Path Deep Dive


Crossbar operation

Unicast

Multicast

Latency

Port 1

Forwarding

?

1/10G MAC

Transceiver

VirtualQueues

EgressQueues

PacketBuffer

Fabric Buffer

VirtualQueues

EgressQueues

PacketBuffer

Fabric BufferUnicast and


10GE Attached Servers


Parsing &Editing

1/10G MAC

Parsing &Editing

Transceiver



Each Unified Port Controller slice has…One 1 Gigabit Ethernet MAC

One 10 Gigabit Ethernet MAC

Two 1/2/4 Gigabit Fibre Channel MACs

Two of the slices in each Gatos have an 802.1AE LinkSec encryption engine

Integrated Flow Control handlingEthernet – 802.3X “PAUSE” and Cisco Priority Flow Control

Fibre Channel – BB_credits


Crossbar Overview

Tightly coupled scheduler and crosspoint

20% link speedup

12 Gbps

Unicast Scheduler

Virtual Output Queuing

3x fabric speed up

3 crosspoints

Multiple frames transferred per scheduling event

“Superframing”

Multicast Scheduler

System Class queuing

Separate crosspoint


Unicast Virtual Output Queuing

CrossbarFabric

Ingress Port

Packet Buffer

Egress Port

Packet BufferPacket Buffer

SchedulerEgress Queue

Q1

Q8

Q1

Q8

Egress Queue

VOQ 1

VOQ N

Egress Port

Packet

Buffer

Packet

Buffer

Egress Queue

Q1

Q8

VOQ 1

VOQ N

Q1

Q8

Q1

Q8


Day in the Life of a Unicast Frame

1. Frame arrive into Packet buffer

2. Frame pointer posted to Virtual Output Queue

3. VOQ posts request to Scheduler

4. Scheduler arbitrates and grants access

5. Frame sent to Fabric Buffer

6. Fabric Buffer sends to egress queue

7. Egress port sends frame on wire

8. Egress buffer indicates freed buffer resources

CrossbarFabric

Ingress Port


Egress Port


Scheduler

Egress QueueEgress Fabric buffer

Q1

Q8

Q1

Q8

Egress Queue

VOQ 1

VOQ N

VOQ 1

VOQ N

1

2

3

4

56

7

8


Scheduler overview

RequestProposal/mandate

Grant

Accept

EgressScheduler

EgressScheduler

EgressScheduler

IngressScheduler

IngressScheduler

IngressScheduler

VOQs IF1

VOQs IF2

VOQs IF3


Unicast scheduler algorithm

Egress Scheduler A priority is selected

Fixed priority, or

DWRR

An ingress is selected within that priority

Highest priority “current preferred” ingress is given a “mandate”

iSLIP maximally matches remaining requesters

Ingress SchedulerEgress schedulers make a proposal

Ingress scheduler selects an egress

Fixed Round Robin selection

The selected Egress Scheduler updates its own “Current preferred”

In multi-pass scheduling, this step happens only for first-pass selections


Multicast


Multicast MAC lookups

MAC table32K entries total (unicast, multicast, Fibre Channel)

1K entries (software setting) for multicast

Populating multicast MAC tableIGMP snooping

Static

Multicast MAC lookup missSource only multicast (for L3 multicast)

Forward frame to interfaces linked to Multicast Routers

Learned via PIM snooping

Flooding (for L2 multicast)


Multicast, fabric replication

CC

Ingress Fabric Egress

BMMcast

A

Ucast

BMcast

C

Mcast

AU-VOQU-VOQ

BBU-VOQU-VOQ

U-VOQU-VOQ

Use cases• Ethernet multicast

M-VOQM-VOQAA

Mcast

A

Mcast

A


Multicast Class Queuing

Port 1MC Ingress PriorityQueues

EgressPriorityQueues



Port

4MC

Ingr

ess

Prio

rity

Queu

es

Egre

ssPr

iorit

yQu

eues

MC Ingress PriorityQueues


Port 3Port 1MC Ingress PriorityQueues
















Port

4MC

Ingr

ess

Prio

rity

Queu

es

Egre

ssPr

iorit

yQu

eues

Port

4MC

Ingr

ess

Prio

rity

Queu

es

MC In

gres

s Pr

iorit

yQu

eues

MC In

gres

s Pr

iorit

yQu

eues

Egre

ssPr

iorit

yQu

eues

Egre

ssPr

iorit

yQu

eues

Egre

ssPr

iorit

yQu

eues









Port 3


Multicast Scheduling AlgorithmGl

obal

RR

poin

ter

Pref

erre

d in

gres

s =

X

DWRR priorityselection

Egress 1Scheduler

Egress 2Scheduler

…Egress 57Scheduler

Ingress 3Scheduler

Match

Ingress selected

Glob

al R

R po

inte

rPr

efer

red

ingr

ess

= X

Request vector Proposal vector

DWRR priorityselection

Egress 1SchedulerEgress 2

SchedulerEgress 2

SchedulerEgress 3

Scheduler……

Egress 57Scheduler

Egress 58Scheduler

Match

Egress 2SchedulerEgress 2

SchedulerEgress

Scheduler

Multicast Fabric Buffer

Free?

Egress BufferFree?


Free?

Egress BufferFree?


Free?

Egress BufferFree?

…..

1


Altos

Gatos

Port 4

Latency 3.2 µsec port-to-port

First-In-First-Out

Full featured forwarding

6.7 µsec kernel to kernel

Stateless offloads

no DDP

1.4 µsec host send

2.1 µsec host receive

Increases with OS, interrupt, and transport overheads

Socket layer app-to-app

Linux 2.6

Raw – 10.1 µsec

UDP – 11.2 µsec

TCP – 11.8 µsec

Port 1

Forwarding

?

1/10G MAC

Transceiver

VirtualQueues

EgressQueues

PacketBuffer

Fabric Buffer

VirtualQueues

EgressQueues

PacketBuffer





Parsing &Editing

1/10G MAC

Parsing &Editing

Transceiver


Lossless Data Path


QoS Processing Flow

STOP

MACMAC TrafficClassification

TrafficClassification

IngresspolicingIngresspolicing

MTU checking

MTU checking

Per-classBuffer usageMonitoring


VoQs for unicast

(8 per egress port)

Egress Queues

MarkingMarking

PFC ON/OFF signal

Trust CoSL2/L3/L4 info with ACL


Truncate or droppackets if MTU is violated


If buffer usage crosses threshold:• Tail drop for droppable class• Assert PFC signal to MAC

for no-drop system class



Mark packet with CoS value


EgressScheduler

Strict priority +DWRR scheduling


Central Scheduler

8 muticast VoQs

STOP

MACMAC TrafficClassification

TrafficClassification

IngresspolicingIngresspolicing

MTU checking

MTU checking



VoQs for unicast

(8 per egress port)

Egress Queues

MarkingMarking

PFC ON/OFF signal





















EgressScheduler





Central Scheduler

Central Scheduler

8 muticast VoQs


Class Based Data Path

Different classes of traffic require different treatment, e.g.

FC class of traffic requires lossless or no drop treatment

Market Data Ethernet class traffic may also require no drop

Remaining Ethernet Data may only require best effort

Nexus 5000 data path resource and features are all per class based; for example,

Per class VOQs and egress queues, buffers, MTU, drop behavior

Per Class behavior should be consistently configured system wide and network wide


Nexus 5000 supports Modular QoS CLI (MQC) for all QoS configuration

System is a new target introduced at the global cfg level

System classes are instantiated within a system policy

System policy is a service-policy attached to the ‘system’ target

At FCS, parameters configurable under system class:

MTU

Drop, no Drop

At ingress, packets are classified into a system class

At FCS, classification can be based on .1p or interface

Once classified, this class assignment travels with the packet through the entire system to select per class treatment at every step

At Egress, 802.1p rewrite is supported. 802.1p value can then be consistently used throughout the network to select the same system class treatment

System Class


PFC and BB_Credits

IEEE 802.3x Pause provides no drop flow control similar to BB credits for FC

Priority Flow Control is a finer grained mechanism of flow control over standard pause or link level BB credits

Priority Flow Control uses .1p CoS value mapping to a system class to send appropriate pause to previous hop


Priority Flow Control

Priority based Flow ControlPriority based Flow Control

• Enables lossless behavior for each class of service• PAUSE sent per priority when buffers limit exceeded


Priority based bandwidth managementPriority based

Bandwidth ManagementPriority based

Bandwidth Management

• Enables Intelligent sharing of bandwidth between traffic classes control of bandwidth• 802.1Qaz Enhanced Transmission


Forwarding


Data Path

Altos

Gatos

Port 4Port 1

Forwarding

?

1/10G MAC

Transceiver

VirtualQueues

EgressQueues

PacketBuffer

Fabric Buffer

VirtualQueues

EgressQueues

PacketBuffer





Parsing &Editing

1/10G MAC

Parsing &Editing

Transceiver


Forwarding Pipeline

Wire rate “fixed” latency

Parsed frame fields, configuration, and control plane state are evaluated to determine destination(s)

Policy engine filters based on configuration, bindings, and layered ACLs

Layered equal cost multi path expansion

Fibre Channel

EtherChannel/PortChannel

Parsed Packet

Collect Interface Configuration and

State

Virtual Interface Table (512)

Vlan Translation Table (4K)

Vlan State Table (1K)

Determine Destination

(ingress only)

Fibre Channel Switch Table (4K)

EthernetLearning

Policy EnforcementACL Search Engine

(2K)

MultipathExpansion

(ingress only)

Zoning Table(2K)

RBACL Label Table(2K)

Binding Table(2K)

Fibre Channel Multipath Table (1K)

PortChannel Table(16)

Multicast Vector Table (4K)Station Table

(16K)

Editing Instructions &Virtual Output Queue List


Parsed Packet

Interface StateVirtual

Interface Table (512)



Forwarding(ingress only)

Fibre Channel Switch Table

(4K)

EthernetLearning

Policy Enforcement

ACL TCAM(2K)

MultipathExpansion

(ingress only)

Zoning Table(2K)


Binding Table(2K)

Fibre Channel Multipath Table

(1K)


Multicast Vector Table

(4K)Station Table

(16K)

Virtual Output Queue List

Destination address

Source address

Ethertype = 2

FCS

Ethertype = .1Q VLANCoS d

TOS Total lenVer IHL

Identification Flg Frgm offset

TTL Proto Header cksum

Source address

Destination address

IP options

Src port Dst port

Seq number

Ack number

Hdr len Flags Win size

Cksum Urgent ptr

TCP options and data

checksum check

FCS check

Parsing ethernet IP packets


Parsed Packet

Interface StateVirtual

Interface Table (512)



Forwarding(ingress only)

Fibre Channel Switch Table

(4K)

EthernetLearning

Policy Enforcement

ACL TCAM(2K)

MultipathExpansion

(ingress only)

Zoning Table(2K)


Binding Table(2K)

Fibre Channel Multipath Table

(1K)


Multicast Vector Table

(4K)Station Table

(16K)

Virtual Output Queue List

Destination address

Source address

FCS

Ethertype = .1Q VLANCoS d

Ethertype = FCoE Ver

Reserved

SOF

EOF Reserved

r_ctl d_id

seq_id df_ctl seq_cnt

ox_id rx_id

Payload

Parameters

cs_ctl s_id

type f_ctl

CRC

CRC checkFCS check

Fibre Channel frames are FCoE encapsulated prior to

forwarding

Parsing FCoE packets


Unified Fabric & FCoE


Mapping of FC Frames over Ethernet

Enables FC to Run on a Lossless Ethernet Network

Mapping of FC Frames over Ethernet

Enables FC to Run on a Lossless Ethernet Network

Fewer Cables

Both block I/O & Ethernet traffic co-exist on same cable

Fewer adapters needed

Overall less power

Interoperates with existing SAN’s

Management SAN’s remains constant

No Gateway

Fewer Cables

Both block I/O & Ethernet traffic co-exist on same cable

Fewer adapters needed

Overall less power

Interoperates with existing SAN’s

Management SAN’s remains constant

No Gateway

FCoEFCoE BenefitsBenefits

FC over Ethernet (FCoE)

Fibre Channel Traffic

Ethernet


Destination MAC Address

Source MAC Address

IEEE 802.1Q Tag

ET = FCoE Ver Reserved

Reserved

Reserved SOF

Encapsulated FC Frame(Including FC-CRC)

EOF Reserved

FCS

Reserved

FCoE Standard Specification Status

An extension of FC over lossless Ethernet

FCoE Standard Specification in ANSI INCITS FC T11.3

Frame Format agreement Aug. 2007

Addressing scheme ratified Feb. 2008

Target completion 2H08

24 companies behind the standard

FCoE Frame FormatBit 0 Bit 31

Eth

ern

etH

ead

er

FC

oE

Hea

der

FC

Hea

der

FC Payload CR

C

EO

F

FC

S

Byte 0 Byte 2197


I/O consolidation Phase 1 (Mid 2008)

Reduction of server adapters

Simplification of access layer & cabling

Gateway free implementation – fits in installed base of existing LAN and SAN

L2 Multipathing Access – Distribution

Lower TCO

Fewer Cables

Investment Protection (LANs and SANs)

Consistent Operational Model

Today:Parallel LAN/SAN Infrastructure

Inefficient use of Network Infrastructure

5+ connections per server – higher adapter and cabling costs

Adds downstream port costs;

cap-ex and op-ex

Each connection adds additional points of failure in the fabric

Longer lead time for server provisioning

Multiple fault domains – complex diagnostics

Management complexity

I/O Consolidation

Enhanced Ethernet and FCoE Ethernet FC

LAN SAN BSAN A

Today I/O Consolidation with FCoE

LAN SAN BSAN A

Nexus 5000Nexus 5000


Nexus 5000: I/O Consolidation

16 Servers Enet FC Total

Adapters 20+ 20 40*

Switches 2 2 4

Cables 40 40 80

Mgmt Pts 2 2 4

16 Servers Enet FC Total

Adapters 20 0 20

Switches 2 0 2

Cables 40 0 40

Mgmt Pts 2 0 2

4

2

8

2

LAN SAN BSAN A LAN SAN BSAN A

Nearly twice the Cables

Nearly twice the Cables

• Half the Cables• Half the Adapters• Power & Cooling Savings• Consistent Management

• Half the Cables• Half the Adapters• Power & Cooling Savings• Consistent Management


Menlo: I/O Consolidation Network Adapter

Off the shelf NIC and HBA ASICs from: Qlogic, Emulex

Dual 10GE/FCoE ports

Support for native drivers and utilities

Customer certified stacks

Replaces multiple adapters per server

Consolidates 10GE and FC on a single interface

Minimum disruption in existing customer environments

Supports PFC & DCBX

Linux (SLES & Redhat) and Windows versions

June 2008 availability

10GE/FCoE

PCIe Bus

Menlo ASIC Cisco/Nuova designed multiplexer and FCoE offload protocol engine

Menlo ASIC Cisco/Nuova designed multiplexer and FCoE offload protocol engine

FC10GE

Documents

Nexus 5000 Architecture