Hawaii Tech Day- DC Switching Design

Data Center Switching Design

Scott HarrisConsulting Systems Engineer, Ciscoscotharr@cisco.com

based on Cisco Live presentation BRK-2208

Agenda

• Data Center Ethernet Design• evolution of traditional designs

• migrating to an ACI design

• Fibre Channel & Ethernet Convergence

• Cisco Nexus & MDS Switch Overview: very brief

Design Considerations

Data Center Design: Scale Up vs Scale Out

Small Spine/Leaf

VXLAN

Single Layer DC

VXLAN

Dual Tier DC

Scale Up

Scale Out

Connectivity & Features Drive Design Choices

Connectivity Model

– 25 or 10 or 1-GigE Server ports

– NIC/HBA Interfaces per-server

– Copper vs Fiber cabling to ToR

Virtual Networking Requirements

– vSwitch/DVS/OVS/Nexus1Kv/AVS

Programmabiltiy/Automation/Orchestration

– Complete abstraction

– Device or fabric level programmability

Computing Form Factor

– Unified Computing Fabric

– 3rd Party Blade Servers

– Rack Servers (Non-UCS Managed)

Storage & Storage Protocols

– Native Fibre Channel

– Unified Ports, FCoE

– IP-based storage (iSCSI, NAS)

VM VM VM

iSCSI

FCoE

FC

NFS/

CIFS

VM VM VM

Data Center Network Needs

• Server virtualization is here

• Oversubscription ratio from 20:1 to 1:1

• Horizontal workload mobility

• Any VLAN everywhere, no physical

boundaries

• Highly distributed applications

• Hadoop and microservices

• More server to server communication

• 80/20 traffic pattern reversed?

• Increasing high availability

requirements

• High throughput, low latency

• More speed, less human errorsEAST – WESTTRAFFIC

NO

RT

H-

SO

UT

HT

RA

FF

IC

FC

FCoE iSCSI /

NAS

Server/Compute

Site B

Enterprise

Network

Internet

DATA CENTER NETWORK

Public

Cloud

Mobile

Services

Storage

Orchestration/

Monitoring

OffsiteDC

API

Why Data Center Network Fabrics?

• Flexibility: allows workload mobility, VLANs everywhere

• Robustness: reduce failure domains, L2/L3 boundaryon leafs, anycast gateway

• (Virtual) Network services move out to border leafs,policy-based service chaining

• Performance: full cross sectional bandwidth (any-to-any) with ECMP, avoid oversubscription

• Latency: deterministic at scale, single hop away

• Scalability: add end nodes, maintain oversubscription

• Cost: fixed switches vs modular switches

EAST – WEST TRAFFIC

NO

RT

H-

SO

UT

HT

RA

FF

IC

FC

FCoE iSCSI /

NAS

Server/Compute

Site B

Enterprise

Network

Internet

DATA CENTER FABRIC

Cloud

Mobile

Services

Storage

Orchestration/

Monitoring

OffsiteDC

API

Definition: ensemble of switches that behaveand get configured like a single giant switch

Evolution of Data Center Ethernet Networks

Scaling Data Center Ethernet

Data Center Interconnect (DCI):

just a teaser

Fibre Channel & Ethernet Convergence

Based on Cisco Live presentation BRKDCN-1902

Traditional Data Center Design

Ethernet LAN and Fibre Channel SAN

• Physical and Logical separation of LANand SAN traffic

• Additional Physical and Logicalseparation of SAN fabrics

Isolation Convergence

Fabric ‘B’

HBA

L2

L3

NIC

Fabric ‘A’

FC

Nexus

7000

Nexus

5000

MDS 9000

Ethernet

FC

Data Center Design with E-SAN

Ethernet LAN and Ethernet SAN

• Same topologies as existing networks, but using NexusUnified Fabric Ethernet switches for SANs

• Physical and Logical separation of LAN and SAN traffic

• Additional Physical and Logical separation of SANfabrics

• Ethernet SAN Fabric carries FC/FCoE & IP basedstorage (iSCSI, NAS, …)

• Common components: Ethernet Capacity and Cost

Isolation Convergence

Fabric ‘B’

CNA

L2

L3

NIC

or

CNA

Fabric ‘A’

FCoE

Nexus

7000

Nexus

5000

Nexus

7000

Nexus

5000

Ethernet

FC

Converged Access with vPCSharing Access Layer for LAN and SAN

• Shared Physical, Separate Logical LAN and SAN traffic atAccess Layer

• Physical and Logical separation of LAN and SAN traffic at Aggregation Layer

• Additional Physical and Logical separation of SAN fabrics

• Storage VDC on Nexus 7000 for additional management /operation separation

Isolation Convergence

Fabric ‘B’

Ethernet

FC

Converged FCoE link

DedicatedFCoE link

L2L3

CNA

Fabric ‘A’

FCFCoE

Nexus

7000

Nexus

5000

MDS

9000

Agg BW: 40G

FCoE:

Ethernet:

40G

40G

One wire for all traffic types

QoS guarantees minimum bandwidth

allocation

No Clear Port ownership

Desirable for DCI Connections

Dedicated wire for a traffic type

No Extra output feature processing

Distinct Port ownership

Complete Storage Traffic Separation

HA: 4LinksAvailable

Different methods, Producing the same aggregate bandwidth

Dedicated Links provide additional isolation of Storage Traffic

Dedicated vs. Converged ISLs

Why support Dedicated ISLs as opposed to Converged?

Agg BW: 40G

FCoE:

Ethernet:

20G

20G

Converged Network – Dual Fabrics with Dedicated Links

Maintaining Dual SAN fabrics with Overlay

• LAN and SAN traffic share physical switches

• LAN and SAN traffic use dedicated links between switches

• All Access and Aggregation switches are FCoE FCFswitches

• Dedicated links between switches are VE_Ports

• Storage VDC for additional management / operationseparation

Isolation Convergence

Nexus 7000

Nexus 5000

Ethernet

FC

Converged FCoE link

DedicatedFCoE link

…

…

L2

L3

CNA FCFCoE

FCFFCF

FCF

VE

Fabric ‘A’

Fabric ‘B’

LAN/SAN

Converged Network – Dual Fabrics with Dedicated Links

Maintaining Dual SAN fabrics with Overlay

• LAN and SAN traffic share physical switches

• LAN and SAN traffic use dedicated links between switches

• All Access and Aggregation switches are FCoE FCFswitches

• Dedicated links between switches are VE_Ports

• Storage VDC for additional management / operation separation

Isolation Convergence

Nexus 7000

Nexus 5000

Ethernet

FC

Converged FCoE link

Dedicated FCoE link

FabricPath

L2

L3

CNA FCFCoE

Fabric ‘A’

Fabric ‘B’

FCF

FCF

FCF

FCF

VE

Cisco Nexus & MDS Platforms

Operational Simplicity

Architectural Flexibility

Open/ Programmable

Resilience and Scale

Investment Protection

One Operating System - NX-OS

CiscoNexus 2000

Cisco Nexus 9000

Cisco Nexus 3000

Cisco Nexus 1000VL4-7 vServices

Cisco Nexus 7000/7700

Cisco Nexus 5000

Cisco MDS 9200 Multiservice Switch

Cisco MDS 9700 Series

Cisco MDS 9100 Series

Cisco MDS 9500 Series