Embed Size (px)
Citation preview
Data Center Switching Design
Scott HarrisConsulting Systems Engineer, [email protected]
based on Cisco Live presentation BRK-2208
Agenda
• Data Center Ethernet Design• evolution of traditional designs
• migrating to an ACI design
• Fibre Channel & Ethernet Convergence
• Cisco Nexus & MDS Switch Overview: very brief
Design Considerations
Data Center Design: Scale Up vs Scale Out
Small Spine/Leaf
VXLAN
Single Layer DC
VXLAN
Dual Tier DC
Scale Up
Scale Out
Connectivity & Features Drive Design Choices
Connectivity Model
– 25 or 10 or 1-GigE Server ports
– NIC/HBA Interfaces per-server
– Copper vs Fiber cabling to ToR
Virtual Networking Requirements
– vSwitch/DVS/OVS/Nexus1Kv/AVS
Programmabiltiy/Automation/Orchestration
– Complete abstraction
– Device or fabric level programmability
Computing Form Factor
– Unified Computing Fabric
– 3rd Party Blade Servers
– Rack Servers (Non-UCS Managed)
Storage & Storage Protocols
– Native Fibre Channel
– Unified Ports, FCoE
– IP-based storage (iSCSI, NAS)
VM VM VM
iSCSI
FCoE
FC
NFS/
CIFS
VM VM VM
Data Center Network Needs
• Server virtualization is here
• Oversubscription ratio from 20:1 to 1:1
• Horizontal workload mobility
• Any VLAN everywhere, no physical
boundaries
• Highly distributed applications
• Hadoop and microservices
• More server to server communication
• 80/20 traffic pattern reversed?
• Increasing high availability
requirements
• High throughput, low latency
• More speed, less human errorsEAST – WESTTRAFFIC
NO
RT
H-
SO
UT
HT
RA
FF
IC
FC
FCoE iSCSI /
NAS
Server/Compute
Site B
Enterprise
Network
Internet
DATA CENTER NETWORK
Public
Cloud
Mobile
Services
Storage
Orchestration/
Monitoring
OffsiteDC
API
Why Data Center Network Fabrics?
• Flexibility: allows workload mobility, VLANs everywhere
• Robustness: reduce failure domains, L2/L3 boundaryon leafs, anycast gateway
• (Virtual) Network services move out to border leafs,policy-based service chaining
• Performance: full cross sectional bandwidth (any-to-any) with ECMP, avoid oversubscription
• Latency: deterministic at scale, single hop away
• Scalability: add end nodes, maintain oversubscription
• Cost: fixed switches vs modular switches
EAST – WEST TRAFFIC
NO
RT
H-
SO
UT
HT
RA
FF
IC
FC
FCoE iSCSI /
NAS
Server/Compute
Site B
Enterprise
Network
Internet
DATA CENTER FABRIC
Cloud
Mobile
Services
Storage
Orchestration/
Monitoring
OffsiteDC
API
Definition: ensemble of switches that behaveand get configured like a single giant switch
Evolution of Data Center Ethernet Networks
Scaling Data Center Ethernet
Data Center Interconnect (DCI):
just a teaser
Fibre Channel & Ethernet Convergence
Based on Cisco Live presentation BRKDCN-1902
Traditional Data Center Design
Ethernet LAN and Fibre Channel SAN
• Physical and Logical separation of LANand SAN traffic
• Additional Physical and Logicalseparation of SAN fabrics
Isolation Convergence
Fabric ‘B’
HBA
L2
L3
NIC
Fabric ‘A’
FC
Nexus
7000
Nexus
5000
MDS 9000
Ethernet
FC
Data Center Design with E-SAN
Ethernet LAN and Ethernet SAN
• Same topologies as existing networks, but using NexusUnified Fabric Ethernet switches for SANs
• Physical and Logical separation of LAN and SAN traffic
• Additional Physical and Logical separation of SANfabrics
• Ethernet SAN Fabric carries FC/FCoE & IP basedstorage (iSCSI, NAS, …)
• Common components: Ethernet Capacity and Cost
Isolation Convergence
Fabric ‘B’
CNA
L2
L3
NIC
or
CNA
Fabric ‘A’
FCoE
Nexus
7000
Nexus
5000
Nexus
7000
Nexus
5000
Ethernet
FC
Converged Access with vPCSharing Access Layer for LAN and SAN
• Shared Physical, Separate Logical LAN and SAN traffic atAccess Layer
• Physical and Logical separation of LAN and SAN traffic at Aggregation Layer
• Additional Physical and Logical separation of SAN fabrics
• Storage VDC on Nexus 7000 for additional management /operation separation
Isolation Convergence
Fabric ‘B’
Ethernet
FC
Converged FCoE link
DedicatedFCoE link
L2L3
CNA
Fabric ‘A’
FCFCoE
Nexus
7000
Nexus
5000
MDS
9000
Agg BW: 40G
FCoE:
Ethernet:
40G
40G
One wire for all traffic types
QoS guarantees minimum bandwidth
allocation
No Clear Port ownership
Desirable for DCI Connections
Dedicated wire for a traffic type
No Extra output feature processing
Distinct Port ownership
Complete Storage Traffic Separation
HA: 4LinksAvailable
Different methods, Producing the same aggregate bandwidth
Dedicated Links provide additional isolation of Storage Traffic
Dedicated vs. Converged ISLs
Why support Dedicated ISLs as opposed to Converged?
Agg BW: 40G
FCoE:
Ethernet:
20G
20G
Converged Network – Dual Fabrics with Dedicated Links
Maintaining Dual SAN fabrics with Overlay
• LAN and SAN traffic share physical switches
• LAN and SAN traffic use dedicated links between switches
• All Access and Aggregation switches are FCoE FCFswitches
• Dedicated links between switches are VE_Ports
• Storage VDC for additional management / operationseparation
Isolation Convergence
Nexus 7000
Nexus 5000
Ethernet
FC
Converged FCoE link
DedicatedFCoE link
…
…
L2
L3
CNA FCFCoE
FCFFCF
FCF
VE
Fabric ‘A’
Fabric ‘B’
LAN/SAN
Converged Network – Dual Fabrics with Dedicated Links
Maintaining Dual SAN fabrics with Overlay
• LAN and SAN traffic share physical switches
• LAN and SAN traffic use dedicated links between switches
• All Access and Aggregation switches are FCoE FCFswitches
• Dedicated links between switches are VE_Ports
• Storage VDC for additional management / operation separation
Isolation Convergence
Nexus 7000
Nexus 5000
Ethernet
FC
Converged FCoE link
Dedicated FCoE link
FabricPath
L2
L3
CNA FCFCoE
Fabric ‘A’
Fabric ‘B’
FCF
FCF
FCF
FCF
VE
Cisco Nexus & MDS Platforms
Operational Simplicity
Architectural Flexibility
Open/ Programmable
Resilience and Scale
Investment Protection
One Operating System - NX-OS
CiscoNexus 2000
Cisco Nexus 9000
Cisco Nexus 3000
Cisco Nexus 1000VL4-7 vServices
Cisco Nexus 7000/7700
Cisco Nexus 5000
Cisco MDS 9200 Multiservice Switch
Cisco MDS 9700 Series
Cisco MDS 9100 Series
Cisco MDS 9500 Series
