Upload
danglien
View
221
Download
3
Embed Size (px)
Citation preview
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Storage Area Networking
Core Edge Design
Best Practices
BRKSAN-1121
1
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Session BRKSAN-1121 Abstract
SAN Core-Edge Design Best Practices
This session gives non-storage-networking professionals the fundamentals to understand and implement storage
area networks (SANs). This curriculum is intended to prepare attendees for involvement in SAN projects and I/O
Consolidation of Ethernet & Fibre Channel networking. You will be exposed to the introduction of Storage
Networking terminology & Designs. Specific topics covered include Fibre Channel (FC), FCoE, FC services, FC
addressing, fabric routing, zoning, virtual SANs (VSANs). The session includes discussions on Designing Core-
Edge Fibre Channel Networks and the best practice recommendations around them. This is an introductory
session and attendees are encouraged to follow up with other SAN breakout sessions and labs to learn more
about specific advanced topics.
2
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Who Am I?
Chad Hintz
Technical Solutions Architect-Data Center/Virtualization
CCIE #15729
Routing & Switching, Security, Storage
3
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
What Are Storage Area Networks?
4
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Session Agenda
History of Storage Area Networks
Fibre Channel Basics
SAN Design Requirements
Introduction to Typical SAN Designs
Introduction to NPIV/NPV
General SAN Best Practices
Core-Edge Design Review
Recommended Core-Edge Designs for Scale and Availability
Next Generation Core-Edge Designs
5
History of Storage Area Networks
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Direct Attached Storage: DAS
DAS – Direct-Attach Storage
Dedicated High Speed Access
Can‘t share capacity
Difficult to share
data
Difficult to manage
7
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Network Attached Storage: NAS
NAS
NAS – Network-Attached Storage
Centralized Storage Attached over LAN
File sharing
More efficient
capacity usage
Performance
limits
usefulness of
NAS – mainly
used for file
storage and
low-end
databases
LAN
8
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Storage Area Network: SAN
SAN – Storage Area Network
Dedicated ‘Back End’ Network
Match DAS
performance
Capacity
deployed and
redeployed
Centralized
management
Diskless servers
– simplified
management,
reduced power
and cooling
NAS
SAN
LAN
9
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
SAN Protocols
Fibre Channel
‒ A gigabit-speed network technology primarily used for storage networking
Fibre Channel over Ethernet (FCoE)
‒ An encapsulation of FibreChannel frames over Ethernet networks. This allows
Fibre Channel to use 10 Gigabit Ethernet networks while preserving the Fibre
Channel protocol
iSCSI
‒ A TCP/IP-based protocol for establishing and managing connections between
IP-based storage devices, hosts and clients
10
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Session Agenda
History of Storage Area Networks
Fibre Channel Basics
SAN Design Requirements
Introduction to Typical SAN Designs
Introduction to NPIV/NPV
General SAN Best Practices
Core-Edge Design Review
Recommended Core-Edge Designs for Scale and Availability
Next Generation Core-Edge Designs
11
Fibre Channel Basics
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
SAN Components
Servers with host bus adapters
Storage systems
‒ RAID
‒ JBOD
‒ Tape
Switches
SAN management software
13
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Types of Fibre Channel Switches
Redundancy/Services
Cisco MDS 9000
MDS 9506, 9509, 9513
MDS 9200
MDS 91XX
Small/Medium Business
Enterprise and Service Provider
FC Bladeswitch
Edge Modular Director
Nexus 5500
14
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Fibre Channel Port Types
‗N‘ port: Node ports used to connect devices to switched
fabric or point to point configurations.
‗F‘ port: Fabric ports residing on switches connecting ‗N‘
port devices
‗L‘ port: Loop ports are used in arbitrated loop
configurations to build networks without FC switches.
These ports often also have ‗N‘ port capabilities and are
called ‗NL‘ ports.
‗E‘ port: Expansion ports are essentially trunk ports used
to connect two Fibre Channel switches
‗GL‘ port: A generic port capable of operating as either an
‗E‘ or ‗F‘ port. Its also capable of acting in an ‗FL‘ port
capacity. Auto Discovery.
N N
N F
NL FL
E E
15
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
G_Port
G_Port
E_Port
F_Port
F_Port
E_Port
N_Port
N_Port
Fibre Channel Port Types
Fibre Channel Switch
NPV
Switch
Input
Port Output
Port
Fabric
X
F_Port NP_Port Fabric
Switch
TE_Port Fabric
Switch TE_Port
End
Node
End
Node
16
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Start from the Beginning…
Start with the host and a target that need to
communicate
‒ Host has 2 HBAs (one per fabric) each with a
WWN
‒ Target has multiple ports to connect to fabric
Connect to a FC Switch
‒ Port Type Negotiation
‒ Speed Negotiation
FC Switch is part of the SAN ―fabric‖
Most commonly, dual fabrics are deployed
for redundancy
FC
HBA
Core
Initiator
Target
FABRIC A
Edge
17
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
My Port Is Up…Can I Talk Now?
FLOGIs/PLOGIs Step 1: Fabric Login (FLOGI)
determines the presence or absence of a Fabric
exchanges Service Parameters with the Fabric
switch identifies the WWN in the service parameters of the accept frame and assigns a Fibre Channel ID (FCID)
initializes the buffer-to-buffer credits
Step 2: Port Login (PLOGI)
required between nodes that want to communicate
similar to FLOGI – transports a PLOGI frame to the designation node port
In p2p topology (no fabric present), initializes buffer-to-buffer credits
N_Port
F_Port
FC
HBA
Core
Initiator
Target
E_Port
Edge
18
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Buffer to Buffer Credits
B2B Credits used to ensure that FC transport is lossless
# of credits negotiated between ports when link is
brought up
# Credits decremented with each packet placed on the
wire
‒ Independent of packet size
‒ If # credits == 0, no more packet transmission
# of credits incremented with each
―transfer ready‖ received
B2B Credits need to be taken into
consideration as distance and/or bandwidth increases
Fibre Channel Flow Control
16
16 P
acke
t
15
R_
RD
Y
Host
Fibre Channel Switch
16
19
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Fabric Name and Addressing: WWN
Every Fibre Channel port and node has a hard-coded address called
World Wide Name (WWN)
During FLOGI the switch identifies the WWN
in the service parameters of the accept frame
and assigns a Fibre Channel ID (FCID)
Switch Name Server maps WWNs to FCID
‒ WWNN uniquely identify devices
‒ WWPN uniquely identify each port in a device
20
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Switch Topology
Model
Switch
Domain Area Device
Fabric Channel ID (FCID)
FCID assigned to every WWPN
corresponding to an N_Port
FCID made up of switch domain, area and
device
Domain ID is native to a single FC switch
limitation of domain IDs in a single fabric
Forwarding decisions made on domain ID
found in first 8 bits of FCID
Fibre Channel Addressing Scheme
FC Fabric
Fabric A
21
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Fabric Shortest Path First (like OSPF)
Fibre Channel Forwarding
FSPF ―routes‖ traffic based on destination
domain ID found in the destination FCID
For FSPF a domain ID identifies a single
switch
‒ The number of Domains IDs are limited to 239/75
(theoretical limited/tested and qualified) within the
same fabric (VSAN)
FSPF performs hop-by-hop routing
FSPF uses total cost as the metric to
determine most efficient path
FSPF supports equal cost load balancing
across links
22
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Directory Server/Name Server (Like DNS)
Repository of information regarding the components that make up the
Fibre Channel network
Located at address FF FF FC (some readings call this the name server)
Components can register their characteristics with the directory server
An N_Port can query the directory server for specific information
‒ Query can be the address identifier, WWN and volume names for all SCSI
targets
23
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Login Complete…
Almost There Fabric Zoning Zones are the basic form of data path
security
Zone members can only ―see‖ and talk to other members of the zone
Devices can be members of more than one zone
Default zoning is ―deny‖
Zones belong to a zoneset
Zoneset must be ―active‖ to enforce zoning
Only one active zoneset per fabric or per VSAN
FC Target
fcid 0x10.00.01 [pwwn 10:00:00:00:c9:76:fd:31] [Initiator]
fcid 0x11.00.01 [pwwn 50:06:01:61:3c:e0:1a:f6] [target]
pwwn 50:06:01:61:3c:e0:1a:f6
FC Fabric
Initiator
pwwn
10:00:00:00:c9:76:fd:31
24
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Zoning—Enforcement Zoning is used to control access in a SAN
Soft zoning
‒ Enforced by name server query responses
‒ Name server sends membership list to N_Port
‒ N-port accesses members only
Hard zoning
‒ Enforced by hardware (forwarding ASIC) at wire speed pWWN, fWWN, FC_ID, FC_Alias
Zone-1
Array
Zone-2
Array
Host
FC
MDS MDS
Host
FC
Soft Zone Hard Zone
Zone-1
Array
Zone-2
Array
Host
FC
MDS MDS
Host
FC
25
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Hard Zone
26
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Soft Zone
27
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Enhanced vs. Basic Zoning
Basic Zoning Enhanced Zoning Enhanced Advantages
Administrators can make simultaneous configuration changes
All configuration changes are made within a single session. Switch locks entire fabric to implement change
One configuration session for entire fabric to ensure consistency within fabric
If a zone is a member of multiple zonesets , an instance is created per zoneset.
References to the zone are used by the zonesets as required once you define the zone.
Reduced payload size as the zone is referenced. The size is more pronounced with bigger database
Default zone policy is defined per switch.
Enforces and exchanges default zone setting throughout the fabric
Fabric-wide policy enforcement reduces troubleshooting time.
28
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Enhanced vs. Basic Zoning
Basic Zoning Enhanced Zoning Enhanced Advantages
Managing switch provides combined status about activation. Will not identify a failure switch.
Retrieves the activation results and the nature of the problem from each remote switch.
Enhanced error reporting reduces troubleshooting process.
To distribute zoneset must re-activate the same zoneset.
Implements changes to the zoning database and distributes it without activation.
This avoids hardware changes for hard zoning in the switches.
During a merge MDS specific types can be misunderstood by non-cisco switches.
Provides a vendor ID along with a vendor-specific type value to uniquely identify a member type
Unique Vendor type
29
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Virtual SANs (VSANs)
Virtual Fabric Separation
Analogous to VLANs in Ethernet
Virtual fabrics created from larger cost-effective redundant physical fabric
Reduces wasted ports of a SAN island approach
Fabric events are isolated per VSAN which gives further isolation for High Availability
Statistics can be gathered per VSAN
Each VSAN provides Separate Fabric Services
‒ FSPF, Zones/Zoneset, DNS, RSCN
A Virtual SAN (VSAN) Provides a Method to
Allocate Ports within a Physical Fabric and
Create Virtual Fabrics
Physical SAN
Islands Are
Virtualizedonto
Common SAN
Infrastructure
VSANs supported on MDS
and Nexus 5000 Product lines
30
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
SAN Islands – Before VSANs
SAN A
DomainID=1
Production SAN Tape SAN Test SAN
DomainID=8 DomainID=7
SAN B
DomainID=2
SAN D
DomainID=4
SAN F
Domain ID=6
SAN E
DomainID=5
SAN C
DomainID=3
31
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
SAN Islands – with Virtual SANs
Production SAN Tape SAN Test SAN
SAN B
DomainID=2
SAN D
DomainID=4
SAN F
Domain ID=6 SAN E
DomainID=5
SAN C
DomainID=3
SAN A
DomainID=1
32
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
VSANs and Zones—Complimentary
Hierarchical relationship—
‒ First assign physical ports to VSANs
‒ Then configure independent zones per VSAN
VSANs divide the physical infrastructure
Zones provide added security and allow sharing
of device ports
Zones can change frequently (e.g. backup)
Ports are added/removed non-disruptively to
VSANs
Virtual SANs and Fabric Zoning Are Very Complementary
VSAN 3
Physical Topology
VSAN 2
Disk1
Host2 Disk4
Host1
Disk2 Disk3
Disk6
Disk5
Host4
Host3
ZoneA
ZoneB
ZoneC
ZoneA
ZoneD
Relationship of VSANs to Zones
33
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Over-Subscription
FAN-OUT Ratio
Over-subscription (or fan-out) ratio for sizing ports and links
Factors used
‒ Speed of Host HBA interfaces
‒ Speed of Array interfaces
‒ Type of server and application
Storage vendors provide guidance in the process
Ratios range between 4:1 - 20:1
FC
6 x 4G Array ports 3 x 8G ISL ports
Example:
10:1 O/S ratio 60 Servers with
4 Gb HBAs
240 G 24 G 24 G
34
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Inter-Switch Link PortChanneling
Criteria for forming a PortChannel
‒ Same speed links
‒ Same modes (auto, E, etc.) and states
‒ Between same two switches
‒ Same VSAN membership
Treated as one logical ISL by upper layer protocols (FSPF)
Can use up to 16 links in a PortChannel
Can be formed from any ports on any modules—HA enabled
Exchange-based in-order load balancing
‒ Mode one: based on src/dstFC_IDs
‒ Mode two: based on src/dst FC_ID/OX_ID
Much faster recovery than FSPF-based balancing
Given logical interface name with aggregated bandwidth and derived routing metric
A PortChannel Is a Logical Bundling of Identical Links
E.g., 8-Gbps
PortChannel
(Four x 2
Gbps)
E.g., 4-Gbps
PortChannel
(Two x 2 Gbps)
35
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
PortChannel vs. Trunking
ISL = inter-switch link
PortChannel = E_Ports and ISLs
Trunk = ISLs that support VSANs
Trunking = TE_Ports and EISLs
36
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Session Agenda
History of Storage Area Networks
Fibre Channel Basics
SAN Design Requirements
Introduction to Typical SAN Designs
Introduction to NPIV/NPV
General SAN Design Best Practices
Core-Edge Design Review
Recommended Core-Edge Designs for Scale and Availability
Next Generation Core-Edge Designs
37
SAN Design Requirements
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
SAN Design
High Availability - Providing a Dual Fabric (current best practice)
Meeting oversubscription ratios established by disk vendors
Effective zoning
Providing Business Function/Operating System Fabric Segmentation and
Security
Fabric scalability (FLOGI and domain-id scaling)
Providing connectivity for virtualized servers
Providing connectivity for diverse server placement and form factors
Key Requirements
39
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
The Design Requirements
Fibre Channel SAN
‒ Transport and Services are on the same layer in the same devices
‒ Well defined end device relationships (initiators and targets)
‒ Does not tolerate packet drop – requires lossless transport
‒ Only north-south traffic, east-west traffic mostly irrelevant
Network designs optimized for Scale and Availability
‒ High availability of network services provided through dual fabric architecture
‒ Edge/Core vs Edge/Core/Edge
‒ Service deployment
Classical Fibre Channel
Client/Server
Relationships are
pre-defined
I(c)
I(c) T(s)
T2
I5
I4 I3 I2
I1
I0
T1 T0
Switch Switch
Switch
DNS FSPF
Zone RSCN DNS
FSPF Zone
RSCN
DNS
Zone
FSPF
RSCN
Fabric topology, services and
traffic flows are structured
40
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Session Agenda
History of Storage Area Networks
Fibre Channel Basics
SAN Design Requirements
Introduction to Typical SAN Designs
Introduction to NPIV/NPV
General SAN Best Practices
Core-Edge Design Review
Recommended Core-Edge Designs for Scale and Availability
Next Generation Core-Edge Designs
41
Typical SAN Designs
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
SAN Design – Single Tier Topology
Collapsed Core Design
Servers connect to the Core switches
Storage devices connect to one or more
core switches
Core switches provide storage services
Large amount of blades to support Initiator
(Host) and Target (Storage) ports
Single Management per Fabric
Normal for Small SAN environments
HA achieved in two physically separate, but
identical, redundant SAN fabrics
FC
Core Core
43
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
How Do We Avoid This?
44
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
SAN Design – Two Tier Topology
―Core-Edge‖ Topology- Most Common
Servers connect to the edge switches
Storage devices connect to one or more core
switches
Core switches provide storage services to
one or more edge switches, thus servicing
more servers in the fabric
ISLs have to be designed so that overall fan-
in ratio of servers to storage and overall end-
to-end oversubscription are maintained
HA achieved in two physically separate, but
identical, redundant SAN fabrics
FC
Core Core
Edge Edge
45
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
SAN Design – Three Tier Topology
―Edge-Core-Edge‖ Topology
Servers connect to the edge switches
Storage devices connect to one or more edge
switches
Core switches provide storage services to one or
more edge switches, thus servicing more servers
and storage in the fabric
ISLs have to be designed so that overall fan-in
ratio of servers to storage and overall end-to-end
oversubscription are maintained
HA achieved in two physically separate, but
identical, redundant SAN fabrics
FC
Core
Edge
Core
Edge
EdgeEdge
46
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Session Agenda
History of Storage Area Networks
Fibre Channel Basics
SAN Design Requirements
Introduction to Typical SAN Designs
Introduction to NPIV/NPV
General SAN Best Practices
Core-Edge Design Review
Recommended Core-Edge Designs for Scale and Availability
Next Generation Core-Edge Designs
47
Introduction to NPIV/NPV
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
What Is NPIV? and Why?
N-Port ID Virtualization (NPIV) provides a means to assign multiple FCIDs
to a single N_Port
‒ Limitation exists in FC where only a single FCID can be handed out per F-port.
Therefore and F-Port can only accept a single FLOGI
Allows multiple applications to share the same Fiber Channel adapter port
Usage applies to applications such as Virtualization
Application Server FC NPIV Core Switch
Web
File Services
Email I/O N_Port_ID 1
Web I/O N_Port_ID 2
File Services I/O N_Port_ID 3
F_Port
F_Port
N_Port
49
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
What Is NPV? and Why? N-Port Virtualizer (NPV) utilizes NPIV functionality to allow a ―switch‖ to act
like a server performing multiple logins through a single physical link
Physical servers connected to the NPV switch login to the upstream NPIV core switch
No local switching is done on an FC switch in NPV mode
FC edge switch in NPV mode does not take up a domain ID
‒ Helps to alleviate domain ID exhaustion in large fabrics
N-Port
Application Server
FC NPIV Core Switch
FC1/1
FC1/2
FC1/3
Server1 N_Port_ID 1
Server2 N_Port_ID 2
Server3 N_Port_ID 3
F_Port
F-Port
F-Port NP-Port
NPV Switch
50
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
NPV Auto Load Balancing
‒ Server loads are not tied to any
uplink
Benefit
‒ Optimal uplink bandwidth utilization
Uniform balancing of server loads on NP links
Bla
de
1
Bla
de
4
Blade Server Chassis
Bla
de
2
SAN
Balanced
load on
NP links
Bla
de
3
1
3 2
4
51
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
NPV Auto Load Balancing
Automatically moves the failed
servers to other available NP links
‒ Servers made to re-login
immediately after experiencing
―short‖ traffic disruption.
Benefit
‒ Downtime greatly reduced
Automatic failover of loads on NP links
Bla
de
1
Bla
de
4
Blade Server Chassis
Bla
de
2
SAN
Bla
de
3
1
2
3
4
Disrupted
servers re-
login on
other uplink
X
52
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
F-Port Port Channel
F-Port PortChannels
‒ Bundle multiple ports in to 1 logical
link
‒ Similar to ISL portchannels in FC
and EtherChannels in Ethernet
Benefits
‒ High-Availability- no disruption if
cable, port, or line cards fail
‒ Optimal bandwidth utilization &
higher aggregate bandwidth with
load balancing
Enhance NPV uplink Resiliency Storage
Bla
deS
yste
m
Blade 1
Blade 2
Blade N
F-Port Port
Channel
F-Port NP-Port
Core Director
SAN
Interface port-channel 1
no shut
Interface fc1/1
channel-group 1
Interface fc1/2
channel-group 1
53
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
NPV F-Port Port Channel
Link failures do not affect the server
connectivity
No application disruption
No traffic disruption
Bla
de
1
Bla
de
4
Blade Server Chassis
Bla
de
2
SAN
Bla
de
3
X
54
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
F-Port Trunking
F-Port Trunking
‒ Uplinks carry multiple VSANs
Benefits
‒ Extend VSAN benefits to Blade
servers
‒ Separate management domains
‒ Traffic Isolation and ability to host
differentiated services on blades
Extend VSAN Benefits to Blades
Storage
Bla
de S
yste
m
Blade N
Core Director
VSAN 1
VSAN 2
VSAN 3
F-Port Trunking
on
F-Port Channel
F-Port NP-Port
SAN
NPV
Interface fc1/1
trunk mode on
trunk allowed-vsan 1-
3
Interface port-channel
1
trunk mode on
trunk allowed-vsan 1-
3
Blade 2
Blade 1
55
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Session Agenda
History of Storage Area Networks
Fibre Channel Basics
SAN Design Requirements
Introduction to Typical SAN Designs
Introduction to NPIV/NPV
General SAN Best Practices
Core-Edge Design Review
Recommended Core-Edge Designs for Scale and Availability
Next Generation Core-Edge Designs
56
General SAN Best Practices
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
SAN Design
High Availability - Providing a Dual Fabric (current best practice)
Fabric scalability (FLOGI and domain-id scaling)
Providing connectivity for diverse server placement and form factors
Meeting oversubscription ratios established by disk vendors
Effective zoning
Providing Business Function/Operating System Fabric Segmentation and
Security
Providing connectivity for virtualized servers
Key Requirements
58
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Zones/ZoneSet
Create Device-Alias for End Devices
‒ Create a readable name for end devices tied to their PWWN
‒ As device moves between VSANs their Device Alias stays the same
Create 2 Member Zones
‒ Hardware zoning on MDS
Recommended to have more zones with 2 members in larger SANs
Single Management of Zones/Zoneset per Fabric
‒ Use Distribute full Zoneset command per VSAN to keep from isolation in
‒ Basic Zoning or use Enhanced Zoning.
If a device has different active members the ISL will become isolated
59
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
SAN Design
Key Requirements
High Availability - Providing a Dual Fabric (current best practice)
Fabric scalability (FLOGI and domain-id scaling)
Providing connectivity for diverse server placement and form factors
Meeting oversubscription ratios established by disk vendors
Effective zoning
Providing Business Function/Operating System Fabric Segmentation and
Security
Providing connectivity for virtualized servers
60
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Virtual SANs
Consolidate SAN Islands into OS or
Department VSANs
Reduction of SAN islands into a single
Fabric while keeping Isolation
Example is to have Test, Development and
Production in their own VSANs
Separate Tape or SAN extension VSANs
Security
Create Separate Administrative Roles per
VSANs
Use TACACS+ for authorization and
auditing of Switches
VSANs supported on MDS
and Nexus 7/5x00 Product
lines
61
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
SAN Design
High Availability - Providing a Dual Fabric (current best practice)
Fabric scalability (FLOGI and domain-id scaling)
Providing connectivity for diverse server placement and form factors
Meeting oversubscription ratios established by disk vendors
Effective zoning
Providing Business Function/Operating System Fabric Segmentation and
Security
Providing connectivity for virtualized servers
Key Requirements
62
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
SAN Design
Use port-channeling/trunking to enhance bandwidth available between
devices
Factors used
‒ Speed of Host HBA interfaces
‒ Speed of Array interfaces
‒ Type of server and application
Keep ISL Oversubscription ratio lower than Array oversubscription ratio
Ratios range between 4:1 - 20:1
FAN-OUT Ratio
FC
6 x 4G Array ports 3 x 8G ISL ports
Example:
10:1 O/S ratio 60 Servers with
4 Gb HBAs
240 G 24 G 24 G
63
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
SAN Security Scope
Fabric security augments overall application security ‒ Host and disk security also required
Six key areas of focus 1. SAN management access—secure access to
management services
2. Fabric access—secure device access to fabric service
3. Target access —secure access to targets and LUNs
4. SAN protocols—secure switch-to-switch communication protocols
5. IP storage access—secure FCIP and iSCSI services
6. Data integrity and secrecy—encryption of data in transit and at rest
*Check Reference Slides for more details around SAN security
1.SAN
Management
Security
4. SAN Fabric
Protocol Security
2.Fabric
Access
Security
6.Data Integrity and Secrecy
3.Target
Access
Security
5.IP Storage
Security
(iSCSI/FCIP)
iSCSI
Cisco
MDS 9000
SAN
Target
Host
64
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Session Agenda
History of Storage Area Networks
Fibre Channel Basics
SAN Design Requirements
Introduction to Typical SAN Designs
Introduction to NPIV/NPV
General SAN Best Practices
Core-Edge Design Review
Recommended Core-Edge Designs for Scale and Availability
Next Generation Core-Edge Designs
65
Core-Edge Design Review
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
SAN Design – Two Tier Topology
―Edge-Core‖ Topology- Most Common
Servers connect to the edge switches
Storage devices connect to one or more core
switches
Core switches provide storage services to
one or more edge switches, thus servicing
more servers in the fabric
ISLs have to be designed so that overall fan-
in ratio of servers to storage and overall end-
to-end oversubscription are maintained
HA achieved in two physically separate, but
identical, redundant SAN fabrics
FC
Core
Edge
Core
Edge
67
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Blade Switch Explosion Issues
Scalability
‒ Each Blade Switch uses a single Domain ID
‒ Theoretical maximum number of Domain IDs is 239 per VSAN
‒ Supported number of domains is quite smaller (depends on OSM)
EMC: 40 domains
Cisco Tested: 75
HP: 40 domains
Other OSM Do Not Post
Manageability
‒ More switches to manage
‒ Shared management of blade switches between storage and server administrators
68
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Server Consolidation with Top-of-Rack Fabric
Switches
2X ISL to Core at 10 G
32 Host Ports at 4 G
A B 96 Storage Ports at 2 G
28 ISL to Edge at 10 G
14 Racks
32 Dual Attached Servers per Rack
A
B
MDS
91XXs
Top of Rack
Top of Rack Design
Ports Deployed:
Storage Ports (4 G Dedicated):
Host Ports (4 G Shared):
Disk Oversubscription (Ports):
Number of FC switches in the
fabric
1200
192
896
9.3 : 1
30
69
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Server Consolidation with Blade servers
2X ISL to Core at 4G
16 Host Ports at 4G
120 Storage Ports at 2 G
60 ISL to Edge at 4 G
Five Racks
96 Dual Attached Blade Servers per Rack
A B
A B
Blade Servers
Blade Server
Design Using 2 x 4
G ISL per Blade
Switch;
Less cables/power Ports Deployed:
Storage Ports (4 G
Dedicated):
Host Ports (4 G Shared):
Disk Oversubscription
(Ports):
Number of FC switches in the
fabric
1608
240
480
8 : 1
62
70
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Session Agenda
History of Storage Area Networks
Fibre Channel Basics
SAN Design Requirements
Introduction to Typical SAN Designs
Introduction to NPIV/NPV
General SAN Best Practices
Core-Edge Design Review
Recommended Core-Edge Designs for Scale and Availability
Next Generation Core-Edge Designs
71
Recommended Core-Edge Designs for
Scale and Availability
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
SAN Design
High Availability - Providing a Dual Fabric (current best practice)
Fabric scalability (FLOGI and domain-id scaling)
Providing connectivity for diverse server placement and form factors
‒ Meeting oversubscription ratios established by disk vendors
‒ Effective zoning
‒ Providing Business Function/Operating System Fabric Segmentation and
Security
Providing connectivity for virtualized servers
Key Requirements
73
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
N-Port Virtualizer (NPV) Reduces Number of
FC Domain IDs
2 ISL to Core at 10 G
32 Host Ports at 4 G
14 Racks
32 Dual Attached Servers per Rack
Top of Rack
Top of Rack Design
Fabric Switches in
NPV mode
Ports Deployed:
Storage Ports (4 G Dedicated):
Host Ports (4 G Shared):
Disk Oversubscription (Ports):
Number of FC switches in the fabric
1200
192
896
9.3 : 1
2
A B 96 Storage Ports at 2 G
28 ISL to Edge at 10 G
A
B
MDS 91xx
in NPV
mode
NPIV Core
74
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
NPV Blade Switch
2 ISL to Core at 4G
16 Host Ports at 4G
Blade Servers
Blade Server Design
Using 2 x 4 G ISL per
Blade Switch;
Less cables/power
Ports Deployed:
Storage Ports (4 G Dedicated):
Host Ports (4 G Shared):
Disk Oversubscription (Ports):
Number of fabric switches to manage
1608
240
480
8 : 1
2
120 Storage Ports at 2 G
60 ISL to Edge at 4 G
Five Racks
96 Dual Attached Blade Servers per Rack
A B
A B
NPIV Core
MDS 91xx
in NPV
mode
75
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
F-Port Port Channel
F-Port PortChannels
‒ Bundle multiple ports in to 1 logical
link
‒ Similar to ISL portchannels in FC
and EtherChannels in Ethernet
Benefits
‒ High-Availability- no disruption if
cable, port, or line cards fail
‒ Optimal bandwidth utilization &
higher aggregate bandwidth with
load balancing
Enhance NPV Uplink Resiliency
Storage
Bla
deS
yste
m
Blade 1
Blade 2
Blade N
F-Port Port
Channel
F-Port N-Port
Core Director
SAN
Interface port-channel 1
no shut
Interface fc1/1
channel-group 1
Interface fc1/2
channel-group 1
76
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Control and monitor VMs in the SAN using NPIV
NPIV gives Virtual Servers SAN identity
‒ Designed for virtual server environments
Allows SAN control of VMs
‒ Zoning and LUN Masking at VM level
Multiple applications on the same port can use different IDs
‒ Better utilization of the server connectivity
FC
LUN1(pwwnD1)
LUN2 (pwwnD2)
LUN3(pwwnD3)
Control and monitor VMs in the SAN
FC
HBA
N_Port Controller
vpwwn1 FCID=1.1.1
vpwwn2 FCID=1.1.2
vpwwn2 FCID=1.1.3
F_Port
Virtual Servers
Web
Zone_Email vpwwn1 pwwnD1 Zone_Web vpwwn1 pwwnD1 Zone_Print vpwwn1 pwwnD1
77
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Nested NPIV
Two levels of NPIV usage
‒ From server to first level switch (NPV)
‒ From NPV to the core SAN
Virtual servers connected to the NPV devices
‒ Servers Supporting NPIV
‒ VmWare ESX in RDM mode
Connecting NPIV capable hosts to NPV
NP
F
P2 N
P
F
P1
NPV Edge Switch
NPV-Core Switch
F F
P3 = vP1 P4 = vP5
vP2 vP3
vP4
vP6 vP7
vP8
NP
IV
NP
IV
78
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
VM-Aware SANS
It is important to follow the guidelines from the virtual machine vendors for
assigning port WWNs to virtual machines. For example, VMware requires
the use of Raw Device Mode (RDM) instead of Virtual Machine File
System (VMFS) to get access to raw LUNs.
Using NPIV/Nested NPV with RDM we can give QOS, incident isolation
(VSANS) and visibility into a Virtualized environment
79
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Summary of Recommendations
High Availability - Provide a Dual Fabric
Use of Port-Channels and F-Port Channels with NPIV to provide the
bandwidth to meet oversubscription ratios
Use NPIV/NPV to provide Domain ID scaling and ease of management
Use of host level NPIV and Nested NPV to provide visibility to Virtualized
servers
80
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Session Agenda
History of Storage Area Networks
Fibre Channel Basics
SAN Design Requirements
Introduction to Typical SAN Designs
Introduction to NPIV/NPV
General SAN Best Practices
Core-Edge Design Review
Recommended Core-Edge Designs for Scale and Availability
Next Generation Core-Edge Designs
81
Next Generation Core-Edge Designs
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Fibre Channel over Ethernet
What Enables It?
10Gbps Ethernet
Lossless Ethernet
‒ Matches the lossless behavior guaranteed in FC by B2B credits
Ethernet jumbo frames
‒ Max FC frame payload = 2112 bytes
Eth
ern
et
Hea
de
r
FC
oE
Hea
de
r
FC
He
ad
er
FC Payload
CR
C
EO
F
FC
S
Same as a physical FC frame
Control information: version, ordered sets
(SOF, EOF)
Normal ethernet frame, ethertype = FCoE
83
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Unified Fabric
Why?
Fewer CNAs (Converged Network adapters) instead of NICs, HBAs and
HCAs
Limited number of interfaces for Blade Servers
All
traffic
goes
over
10GE
IPC Traffic HCA
FC Traffic FC HBA
CNA
CNA
FC Traffic FC HBA
NIC LAN Traffic
NIC LAN Traffic
NIC Mgmt Traffic
NIC Backup Traffic
84
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Today‘s Unified I/O Architecture
Ethernet FC
LAN SAN B SAN A
Today I/O Consolidation with FCoE
SAN B LAN SAN A
FCoE
Nexus
5000/UCS
85
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Nexus 5K FCOE Switches Also Use NPV to
Achieve Both Server and IO Consolidation
A B
Attached Servers per Rack
A
B
Nexus 5K/2k in
NPV mode
LAN Core
32 Host CNA Ports at 10G
Core connectivity using FC modules on N5K
86
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Cisco UCS Fabric Interconnects Also
Use NPV
A B
UCS 6X00 fabric
Interconnect in
NPV mode
LAN Core
87
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
UCS Core-Edge Design
F_Port Channeling and Trunking from MDS to
UCS
FC Port Channel behaves as one logical
uplink
FC Port Channel can carry all VSANs (Trunk)
UCS Fabric Interconnects remains in NPV end
host mode
Server vHBA pinned to an FC Port Channel
Server vHBA has access to bandwidth on any
link member of the FC Port Channel
Load balancing based on FC Exchange_ID
‒ Per Flow
Loss of Port Channel member link has no
effect on Server vHBA (hides the failure)
‒ Affected flows to remaining member links
‒ No FLOGI required
N-Port Virtualization Forwarding with MDS
F_ Port
Channel &
Trunk
SAN B SAN A
Server 1 VSAN 1
vFC 1 vFC 1
N_Proxy
F_Proxy
N_Port
6100-A 6100-B
F_Port
vFC 2 vFC 2
Server 2 VSAN 2
vHBA 1 vHBA 0 vHBA 1 vHBA 0
VSAN 1,3 VSAN 1,2
NPIV NPIV
88
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Nexus 5X00 Core_Edge
Nexus 5000 access switches operating
in NPV mode
With NX-OS release 4.2(1) Nexus 5000 supports F-Port Trunking and Channeling on the links between an NPV device and upstream FC switch (NP port -> F port)
F_Port Trunking: Better multiplexing of traffic using shared links (multiple VSANs on a common link)
F_Port Channeling: Better resiliency between NPV edge and Director Core
‒ No host re-login needed per link failure
‒ No FSPF recalculation due to link failure
Simplifies FC topology (single uplink from NPV device to FC director)
F_Port Trunking and Channeling
Fabric ‘A’ Supporting
VSAN 2
F Port Trunking & Channeling
VLAN 10,30
VLAN 10,20
VSAN 3
Fabric ‘B’ Supporting
VSAN 3
VF
VN
TF
TNP
Server ‘1’
VSAN 2 Server ‘2’
VSAN 3
Nexus 5000
NPV
VLAN 30=VSAN 3
VLAN 20=VSAN 2
VSAN 2
VLAN 10
89
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
FCoE Multi-Tier
Multi-hop edge/core/edge topology
Core SAN switches supporting FCoE
N7K with DCB/FCoE line cards
MDS with FCoE line cards (Sup2A)
Edge FC switches supporting either
N5K - E-NPV with FCoE uplinks to the FCoE enabled core (VNP to VF)
N5K or N7K - FC Switch with FCoE ISL uplinks (VE to VE)
Scaling of the fabric (FLOGI, …) will most likely drive the selection of which mode to deploy
Larger Fabric Multi-Hop Topologies
Edge FCF Switch Mode
Servers, FCoE attached Storage
N7K or MDS FCoE enabled Fabric Switches
FC Attached Storage Servers
VE
VE
Edge Switch in E-NPV Mode
VF
VNP VE
VE
90
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Fibre Channel Aware Device
What does an FCoE-NPV device do?
‖FCoE NPV bridge" improves over a "FIP snooping bridge" by intelligently proxying FIP functions between a CNA and an FCF
Active Fibre Channel forwarding and security element
FCoE-NPV load balance logins from the CNAs evenly across the available FCF uplink ports
FCoE NPV will take VSAN into account when mapping or ‗pinning‘ logins from a CNA to an FCF uplink
Emulates existing Fibre Channel Topology (same mgmt, security, HA)
Avoids Flooded Discovery and Configuration (FIP & RIP)
FCoE NPV
FCF
Fibre Channel Configuration and Control
Applied at the Edge Port
Proxy FCoE VLAN Discovery
Proxy FCoE FCF Discovery
FCoE NPV
VF
VNP
91
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Session Agenda
History of Storage Area Networks
Fibre Channel Basics
SAN Design Requirements
Introduction to Typical SAN Designs
Introduction to NPIV/NPV
General SAN Best Practices
Core-Edge Design Review
Recommended Core-Edge Designs for Scale and Availability
Next Generation Core-Edge Designs
92
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Reference Sessions
BRKCOM-2002-UCS Supported Storage Architectures and Best Practices
with Storage
BRKDCT-1044-FCoE for the IP Network Engineer
BRKSAN-2282- Operational Models for FCOE Deployments- Best
Practices and Examples
BRKCOM-2001 UCS Deep Dive
93
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Recommended Reading
NX-OS and Cisco Nexus Switching (ISBN:
1587058928), by David Jansen, Ron Fuller,
Kevin Corbin. Cisco Press 2010.
Storage Networking Fundamentals(ISBN-10:1-
58705-162-1; ISBN-13: 978-11-58705-162-3),
by Marc Farley. Cisco Press. 2007.
Storage Networking Protocol Fundamentals
(ISBN: 1-58705-160-5), by James Long, Cisco
Press. 2006.
Available Onsite at the Cisco Company Store
94
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Complete Your Online
Session Evaluation Give us your feedback and you
could win fabulous prizes.
Winners announced daily.
Receive 20 Passport points for each
session evaluation you complete.
Complete your session evaluation
online now (open a browser through
our wireless network to access our
portal) or visit one of the Internet
stations throughout the Convention
Center.
Don‘t forget to activate your
Cisco Live Virtual account for access to
all session material, communities, and
on-demand and live activities throughout
the year. Activate your account at the
Cisco booth in the World of Solutions or visit
www.ciscolive.com.
95
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Final Thoughts
Get hands-on experience with the Walk-in Labs located in World of
Solutions, booth 1042
Come see demos of many key solutions and products in the main Cisco
booth 2924
Visit www.ciscoLive365.com after the event for updated PDFs, on-
demand session videos, networking, and more!
Follow Cisco Live! using social media:
‒ Facebook: https://www.facebook.com/ciscoliveus
‒ Twitter: https://twitter.com/#!/CiscoLive
‒ LinkedIn Group: http://linkd.in/CiscoLI
96
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Reference Slides
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
NPV Traffic Engineering
Allows user to select external
interface per server interface
Benefits
‒Allows customized Bandwidth
Management
‒Allows use of shortest path
‒Enables use of Persistent FCIDs
Bla
de
1
Bla
de
N
Blade Server Chassis
Storage
Bla
de
2
….
Traffic-map
SAN
Customize
d Traffic
Pattern
1
2
N
npv traffic-map server-interface fc1/2 external-interface
fc1/1
npv traffic-map server-interface fc1/3 external-interface
fc1/1
…….
npv traffic-map server-interface fc1/N external-interface
fc1/5
Fc1/3 Fc1/4 Fc1/24
Fc1/1 Fc1/5
99
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Number of NPIV Logins: MDS 9200/9500
Type of Logins Verified Logins
Logins per Port 126 (1) / 256 (2)
Logins per Line Card 400
Logins per Switch 2,000
Logins per physical
fabric
10,000
These are the number of logins allowed on all Gen1, Gen2 and Gen3 line cards. The limits applied to on a per switch will also apply to all MDS 9200 and MDS 9500. MDS 9124/9134 and Blade switches will have different limits and will be shown later.
(1) SAN-OS 3.x, NX-OS 4.1(1) (2) NX-OS 4.1(2)
100
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Number of NPIV Logins:
MDS 9124/9134 and Blade Switches
Switching Mode NPV Mode
Logins per Port 42 (1) / 89 (2) 114
Logins per Port-Group 168 114
Logins per MDS 9124 1008 684
Logins per MDS 9134 1680 1140
Logins per MDS 9124e 1008 684
Logins per IBM Blade Switch 840 570
The stated numbers are verified / supported number of logins.
(1) Using 2 member zoning (2) Using default zone-permit instead of zoning
101
Introduction to FCOE
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Fibre Channel over Ethernet
What Enables It?
10Gbps Ethernet
Lossless Ethernet
‒ Matches the lossless behavior guaranteed in FC by B2B credits
Ethernet jumbo frames
‒ Max FC frame payload = 2112 bytes
Eth
ern
et
Hea
de
r
FC
oE
He
ad
er
FC
Hea
de
r
FC Payload
CR
C
EO
F
FC
S
Same as a physical FC
frame
Control information: version, ordered
sets (SOF, EOF)
Normal ethernet frame, ethertype =
FCoE
103
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Unified Fabric
IEEE DCB Standard / Feature Status of the Standard
IEEE 802.1Qbb Priority-based Flow Control (PFC)
In Sponsor Ballot
IEEE 802.3bd Frame Format for PFC
In Sponsor Ballot
IEEE 802.1Qaz Enhanced Transmission Selection (ETS) and Data Center Bridging eXchange (DCBX)
Just completed WG recirculation ballot. New recirculation expected next week in order to go to Sponsor Ballot after the May interim
IEEE 802.1Qau Congestion Notification
Done!
IEEE 802.1Qbh Port Extender
In its first task group ballot
Developed by IEEE 802.1 Data
Center Bridging Task Group (DCB)
All technically stable
Final standards expected by mid
2010
CEE (Converged Enhanced Ethernet) is an informal
group of companies that submitted initial inputs to the
DCB WGs. 104
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
FC Traffic FC HBA
Unified Fabric Why?
Fewer CNAs (Converged Network adapters) instead of NICs, HBAs and
HCAs
Limited number of interfaces for Blade Servers
All
traffic
goes
over
10GE
CNA
CNA
FC Traffic FC HBA
NIC LAN Traffic
NIC LAN Traffic
NIC Mgmt Traffic
NIC Backup Traffic
IPC Traffic HCA
105
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
What‘s the difference between DCE, CEE and
DCB ?
All three acronyms describe the same thing, meaning the architectural
collection of Ethernet extensions (based on open standards)
Cisco has co-authored many of the standards associated and is focused
on providing a standards-based solution for a Unified Fabric in the data
center
The IEEE has decided to use the term ―DCB‖ (Data Center Bridging) to
describe these extensions to the industry.
http://www.ieee802.org/1/pages/dcbridges.html
106
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Priority Flow Control Fibre Channel over Ethernet Flow Control
Pa
cke
t
R_R
DY
Fibre Channel
Transmit Queues Ethernet Link
Receive Buffers
Eight
Virtual
Lanes
One One
Two Two
Three Three
Four Four
Five Five
Seven Seven
Eight Eight
Six Six
STOP PAUSE
B2B Credits
Enables lossless Ethernet using PAUSE based on a COS as defined in 802.1p
When link is congested, CoS assigned to FCoE will be PAUSEd so traffic will not be dropped
Other traffic assigned to other CoS will continue to transmit and rely on upper layer protocols for retransmission
107
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
DCB ―Virtual Links‖ An Example
VL1
VL2
VL3
LAN/IP Gateway
VL1 – LAN Service – LAN/IP VL2 - No Drop Service - Storage
Ability to support QoS queues within the ―lanes‖
Campus Core/
Internet
Storage Area
Network
108
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Fiber Channel over Ethernet Protocol Host Side – FIP and DCBX Configuration
1st portion of the
MAC is the FC-
MAP of the
Nexus 5000
FC-MAP
(0E-FC-xx)
FC-ID
7.8.9 FC-MAC
Address
FC-MAP
(0E-FC-xx)
FC-ID
10.00.01
2nd portion of the MAC
is the FC-ID
109
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
FCoE Building Blocks
The Acronyms Defined
FCF (FCoE Forwarder): A Fibre Channel switching element that is able to forward FCoE frames (Nexus 5000, Nexus 7000, MDS 9000)
FPMA : ‖Fabric Provided MAC Address‖ -- A unique MAC address that is assigned by an FCF to a single Enode
Enode: ―End Node‖ -- a Fiber Channel end node that is able to transmit FCoE frames using one or more ENodeMACs.
FCoE Pass-Through : any DCB device capable of passing FCoE frames to an FCF
FIP Snooping Bridge
FCoE N-Port Virtualizer
Single hop FCoE : running FCoE between the host and the first hop access level switch
Multi-hop FCoE : the extension of FCoE beyond a single hop into the Aggregation and Core layers of the Data Centre Network
110
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
FCoE Building Blocks
Fibre Channel Forwarder
FCF (Fibre Channel Forwarder) is the Fibre Channel switching element
inside an FCoE switch
‒ Fibre Channel logins (FLOGIs) happens at the FCF
‒ Consumes a Domain ID
FCoE encap/decap happens within the FCF
‒ Forwarding based on FC information
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Ethernet Bridge
FC
por
t FC
por
t FC
por
t FC
por
t
FCF
FCoE Switch FC Domain ID : 15
111
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Fibre
Channel
Drivers
Ethernet
Drivers
Operating System
PCIe
Eth
ern
et
Fib
re C
ha
nn
el
10G
bE
10G
bE
Link
Ethernet Driver
bound to
Ethernet NIC PCI
address
FC Driver
bound to FC
HBA PCI
address
Replaces multiple adapters per
server, consolidating both Ethernet and FC on a single interface
Appears to the operation system as individual interfaces (NICs and HBAs)
First Generation CNAs from support PFC and CIN-DCBX
Second Generation CNAs support PFC, CEE-DCBX as well as FIP
Single chip implementation
Half Height/Length
Half power consumption
FCoE Building Blocks Converged Network Adapter
112
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Translate to FCoE…
Same host to target communication
Host has 2 CNA‘s (one per fabric)
Target has multiple ports to connect to fabric
Connect to a capable switch
Port Type Negotiation (FC port type will be handled by FIP)
Speed Negotiation
DCBX Negotiation
Access switch is a Fibre Channel Forwarder (FCF)
Dual fabrics are still deployed for redundancy
FC
CNA
FC Fabric
ENode
Target
Ethernet Fabric
DCB capable Switch
acting as an FCF
Unified Wire
113
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
VE_Port
VF_Port
VF_Port
VE_Port
VN_Port
VN_Port
Fibre Channel Over Ethernet Port Types
Fibre Channel over Ethernet Switch
FCoE_
NPV
Switch
VF_Port VNP_Port FCF
Switch
End
Node
End
Node
FCoE Switch : FCF
114
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
My Port Is Up…Can I Talk Now? FIP and FCoE Login Process
VN_Port
VF_Port
FIP Discovery
E_ports or
VE_Port
Step 1: FIP Discovery Process
enables FCoE adapters to discover which VLAN to transmit & receive FCoE frames
enables FCoE adapters and FCoE switches to discovers other FCoE capable devices
verifies Lossless Ethernet is capable of FCoE transmit
Step 2: FIP Login Process Simular to existing Fibre Channel Login process - sends
FLOGI to upstream FCF
adds the negotiation of the MAC address to use Fabric Provided MAC Address (FPMA)
FCF assigns the host a Enode MAC address to be used for FCoE forwarding
FC
CNA
FC or FCoE Fabric
Target
ENode
**Multi-hope FCoE with VE_Ports not supported until Eaglehawk Release
115
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Enode MAC Address Fibre Channel over Ethernet Addressing Scheme
Enode MAC assigned for each FCID
Enode MAC composed of a FC-MAP and FCID
FC-MAP is the upper 24 bits of the Enode‘s MAC
FCID is the lower 24 bits of the Enode‘s MAC
FCoE forwarding decisions still made based on FSPF and the FCID within the Enode MAC
FC Fabric
Domain ID
FC-MAP
(0E-FC-xx)
FC-ID
7.8.9
FC-
MAC
Address
FC-MAP
(0E-FC-xx) FC-ID
10.00.01
Fibre Channel
FCID
Addressing
116
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Login Complete…Almost There Fabric Zoning
FC
FC/FCoE Fabric
Initiator
Target
FCoE fabric zoning done the same as FC fabric zoning
Zoning is enforced at the FCF
Zoning can be configured on the Nexus 5000 using the CLI or Fabric Manager
If Nexus 5000 is an ―FCoE Pass-Through‖ device, zoning will be configured on the upstream core switch and pushed to the Nexus 5000
fcid 0x10.00.01 [pwwn 10:00:00:00:c9:76:fd:31] [tnitiator]
fcid 0x11.00.01 [pwwn 50:06:01:61:3c:e0:1a:f6] [target] pwwn 10:00:00:00:c9:76:fd:31
pwwn 50:06:01:61:3c:e0:1a:f6
**Multi-hope FCoE with VE_Port not supported until Eaglehawk
FCF with Domain ID 10
117
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Host connected over unified wire to first hop access switch
Access switch (Nexus 5000) is the FCF
Fibre Channel ports on the access switch can be in NPV or Switch mode for native FC traffic
DCBX is used to negotiate the enhanced Ethernet capabilities
FIP is use to negotiate the FCoE capabilities as well as the host login process
FCoE runs from host to access switch FCF – native Ethernet and native FC break off at the access layer
FC
CNA
FC Fabric
ENode
Target
Ethernet Fabric
DCB capable Switch
acting as an FCF
Unified Wire
Single Hop Design Today‘s Solution
118
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Fabric A
CEE-DCBX
Generation 1 CNA
CIN-DCBX
Generation 2 CNA
Fabric B LAN Fabric
VN
VF Direct attach
VN_Port to
VF_Port
Generation 1 CNA
limited to direct attached CNAs at the access
Utilized Cisco, Intel, Nuova Data Center Bridging Exchange protocol (CIN-DCBX)
Generation 2 CNA
Utilizes Converged Enhanced Ethernet Data Center Bridging Exchange protocol (CEE-DCBX)
Utilizes FCoE Initialization Protocol (FIP) as defined by the T.11 FC-BB-5 specification
Supports both direct and multi-hop attachment (through a Nexus 4000 FIP Snooping Bridge)
Single Hop Design The CNA Point of View
Nexus 5000
FCF-A Nexus 5000
FCF-A
119
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Unified Fabric with FCoE CNA: Converged Network Adapter
Standard drivers
Same management
Operating System sees:
‒ Dual port 10 Gigabit
Ethernet adapter
‒ Dual Port 4 GbpsFibre
ChannelHBAs
120
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
vfc2
Eth1/2 PC1
vfc1
vfc3
mac-address
The Virtual Fibre Channel Interface Binding the vfc Virtual Fibre Channel Interface (vfc) : Where is the
logical Fibre Channel wire is terminated (the FCF)
Today this corresponds to an ―F_Port‖
Three options for binding a vfc interface :
Physical Interface: Direct Attach CNAs and FCoE_NPV devices (future)
Single link port-channel: Direct Attach CNAs connected via a two port vPC
MAC-Address over an Ethernet Cloud: through a FIP-Snooping Device
Nexus 5000
FCF
Nexus 4000
FIP-Snooping
Nexus 5000
FCF
121
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
VLAN 10,30
VLAN 10,20
FCoE VLANs are treated differently than native Ethernet VLANs
No flooding, MAC learning, broadcasts, etc.
The FCoE VLAN must not be configured as a native VLAN
FIP uses native VLAN
FCoE VLANs should not be configured on Ethernet links that are not carrying FCoE traffic
Unified Wires must be configured as trunk ports and STP edge ports
! VLAN 20 is dedicated for VSAN 2 FCoE traffic
(config)# vlan 20
(config-vlan)# fcoevsan 2
VSAN 2
STP Edge Trunk
Fabric A Fabric B LAN Fabric
Nexus 5000
FCF
Nexus 5000
FCF
VSAN 3
Single Hop Design The FCoE VLAN
122
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
VLAN 10,30
VLAN 10,20
FCoE Fabric ‗A‘ will have a different VLAN topology than FCoE Fabric ‗B‘ which are different from the LAN Fabric
PVST+ allows unique topology per VLAN
MST requires that all switches in the same Region have the same mapping of VLANs to instances
MST does notrequire that all VLANs be defined in all switches
A separate instance must be used for FCoE VLANs
spanning-tree mst configuration
name FCoE-Fabric
revision 5
instance 5 vlan 1-19,40-3967,4048-4093
instance 10 vlan 20-29
instance 15 vlan 30-39
Fabric A Fabric B LAN Fabric
VSAN 3 VSAN 2
VLAN 10
Nexus 5000
FCF-A
Nexus 5000
FCF-B
Single Hop Design The FCoE VLAN and STP
123
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
VLANs and VSANs FCoE Considerations
VSANs use VLAN hardware table resources
FCoE requires a VLAN and a VSAN that you bind the VLAN to.
Hence for each FCoE VSAN you should count using 2 VLANs
Enabling FCoE burns two internal VSAN/VLAN resources
vFC binds to the Port-Channel, as long as there is one single port in the
port-channel attached to the switch
124
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Optimal layer 2 LAN design often
leverages Multi-Chassis Etherchannel
(MCEC)
Nexus utilizes Virtual Port-Channel (vPC)
to enable MCEC either between switches
or to direct attached servers (using LACP
or static port-channels)
MCEC provides network based load
sharing and redundancy without
introducing layer 2 loops in the topology
MCEC maintains the separation of LAN
and SAN high availability topologies
‒ FC maintains separate SAN ‗A‘
and SAN ‗B‘ topologies
‒ LAN utilizes a single logical
topology
Direct Attach vPC
Topology
MCEC
vPC Peers
vPC Peer Link
Fabric A Fabric B LAN Fabric
Nexus 5000
FCF-A
Nexus 5000
FCF-B
Single Hop Design What is MCEC??
125
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
vPC enabled topologies with FCoE
must follow specific design and
forwarding rules…
A ‗vfc‘ interface can only be
associated with a single-port port-
channel
While the port-channel configurations are the same on N5K-1 and N5K-2, the FCoE VLANs are different
vPC configuration works with Gen-2 FIP enabled CNAs ONLY
FCoE VLANs are ‘not’ carried on the vPC peer-link
FCoE and FIP ethertypes are ‘not’ forwarded over the vPC peer link
Direct Attach vPC
Topology
VLAN 10,30
VLAN 10,20
STP Edge Trunk
VLAN 10 ONLY HERE!
Fabric A Fabric B LAN Fabric
Nexus 5000
FCF-A
Nexus 5000
FCF-B
Single Hop Design Unified Wires and MCEC
vPC contains only 2 X
10GE links – one to each
Nexus 5000
126
CLI Configuration Sample
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Sample Working Topology
NPV Core
NPV Edge
128
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Enabling the Required Features
NPV Edge Switch:
pod7-5020-51(config)# feature fcoe
FC license checked out successfully
fc_plugin extracted successfully
FC plugin loaded successfully
FCoE manager enabled successfully
FC enabled on all modules successfully
pod7-5020-51(config)# feature npv
Verify that boot variables are set and the changes are saved. Changing to npv mode erases the current
configuration and reboots the switch in npv mode. Do you want to continue? (y/n):y
NPV Core Switch:
pod3-9216i-70(config)# feature npiv
pod3-9216i-70(config)# feature fport-channel-trunk
Admin trunk mode has been set to off for
1- Interfaces with admin switchport mode F,FL,FX,SD,ST in admin down state
2- Interfaces with operational switchport mode F,FL,SD,ST.
129
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Configure the VSANs
NPV Edge Switch:
pod7-5020-51(config)# vsan database
pod7-5020-51(config-vsan-db)# vsan 10
NPV Core Switch:
pod3-9216i-70(config)# vsan database
pod3-9216i-70(config-vsan-db)# vsan 10
pod3-9216i-70(config-vsan-db)# vsan 10 interface fc1/12
130
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Configure Trunking F_Port Port Channel
NPV Core Switch:
pod3-9216i-70(config)# interface port-channel 1
pod3-9216i-70(config-if)# switchport mode f
pod3-9216i-70(config-if)# switchport trunk mode on
pod3-9216i-70(config-if)# channel mode active
pod3-9216i-70(config-if)# interface fc2/13, fc2/19
pod3-9216i-70(config-if)# switchport mode f
pod3-9216i-70(config-if)# switchport rate-mode dedicated
pod3-9216i-70(config-if)# switchport trunk mode on
pod3-9216i-70(config-if)# channel-group 100 force
fc2/13 fc2/19 added to port-channel 100 and disabled
please do the same operation on the switch at the other end of the port-channel, then do "no shutdown"
at both end to bring them up
pod3-9216i-70(config-if)# no shut
131
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Configure Trunking F_Port Port Channel
NPV Edge Switch:
pod7-5020-51(config)# interface san-port-channel 1
pod7-5020-51(config-if)# switchport mode np
pod7-5020-51(config-if)# switchport trunk mode on
pod7-5020-51(config-if)# interface fc2/1-2
pod7-5020-51(config-if)# switchport mode np
pod7-5020-51(config-if)# switchport trunk mode on
pod7-5020-51(config-if)# channel-group 1
fc2/1 fc2/2 added to port-channel 1 and disabled
please do the same operation on the switch at the other end of the port-channel, then do "no shutdown"
at
both ends to bring it up
pod7-5020-51(config-if)# no shut
132
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Configure FCoE on NPV Edge Switch
pod7-5020-51(config)# vlan 10
pod7-5020-51(config-vlan)# fcoevsan 10
pod7-5020-51(config-vlan)# interface ethernet 1/3
pod7-5020-51(config-if)# switchport mode trunk
pod7-5020-51(config-if)# switchport trunk allowed vlan 1,10
pod7-5020-51(config-if)# spanning-tree port type edge trunk
Warning: Edge port type (portfast) should only be enabled on ports connected to a single host. Connecting hubs,
concentrators, switches, bridges, etc... to this interface when edge port type (portfast) is enabled, can cause
temporary bridging loops. Use with CAUTION
pod7-5020-51(config-if)# interface vfc3
pod7-5020-51(config-if)# bind interface ethernet 1/3
pod7-5020-51(config-if)# vsan database
pod7-5020-51(config-vsan-db)# vsan 10 interface vfc3
pod7-5020-51(config-vsan-db)# interface vfc3
pod7-5020-51(config-if)# no shut
133
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Verify NPV Fabric Connectivity (Edge) pod7-5020-51# sh npv flogi-table
--------------------------------------------------------------------------------
SERVER EXTERNAL
INTERFACE VSAN FCID PORT NAME NODE NAME INTERFACE
--------------------------------------------------------------------------------
vfc3 10 0x0f0100 21:00:00:c0:dd:12:04:f3 20:00:00:c0:dd:12:04:f3 Spo1
Total number of flogi = 1.
Verify that the VFC interface is pinned to the SAN Port Channel
pod7-5020-51# sh npv traffic-usage
NPV Traffic Usage Information:
----------------------------------------
Server-If External-If
----------------------------------------
vfc3 san-port-channel 1
----------------------------------------
134
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Verify on Core that N5K and FCoE Workstation
Are Logged into the Fabric.
pod3-9216i-70(config)# show flogi database
--------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------------
fc1/12 10 0x0f00dc 21:00:00:20:37:a9:cd:6e 20:00:00:20:37:a9:cd:6e
fc1/12 10 0x0f00e0 21:00:00:20:37:a9:89:7e 20:00:00:20:37:a9:89:7e
fc1/12 10 0x0f00e2 21:00:00:20:37:af:de:85 20:00:00:20:37:af:de:85
fc1/12 10 0x0f00e4 21:00:00:20:37:a9:d6:49 20:00:00:20:37:a9:d6:49
fc1/12 10 0x0f00e8 21:00:00:20:37:a9:d7:bf 20:00:00:20:37:a9:d7:bf
fc1/12 10 0x0f00ef 21:00:00:20:37:a9:94:89 20:00:00:20:37:a9:94:89
port-channel 1 1 0x670102 24:01:00:0d:ec:a3:da:40 20:01:00:0d:ec:a3:da:41
port-channel 1 10 0x0f0100 21:00:00:c0:dd:12:04:f3 20:00:00:c0:dd:12:04:f3
Total number of flogi = 8.
135
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Configure Zoning
NPV Core Switch:
pod3-9216i-70(config)# zone name npv_vsan10 vsan 10
pod3-9216i-70(config-zone)# member pwwn 21:00:00:20:37:a9:cd:6e
pod3-9216i-70(config-zone)# member pwwn 21:00:00:20:37:a9:89:7e
pod3-9216i-70(config-zone)# member pwwn 21:00:00:20:37:af:de:85
pod3-9216i-70(config-zone)# member pwwn 21:00:00:c0:dd:12:04:f3
pod3-9216i-70(config-zone)# exit
pod3-9216i-70(config)# zoneset name npv_v10_zs vsan 10
pod3-9216i-70(config-zoneset)# member npv_vsan10
pod3-9216i-70(config-zoneset)# zoneset activate name npv_v10_zs vsan 10
Zoneset activation initiated. check zone status
pod3-9216i-70(config)# show zoneset active
zoneset name npv_v10_zs vsan 10
zone name npv_vsan10 vsan 10
* fcid 0x0f00dc [pwwn 21:00:00:20:37:a9:cd:6e]
* fcid 0x0f00e0 [pwwn 21:00:00:20:37:a9:89:7e]
* fcid 0x0f00e2 [pwwn 21:00:00:20:37:af:de:85]
* fcid 0x0f0100 [pwwn 21:00:00:c0:dd:12:04:f3]
136
SAN Security
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
SAN Management Potential Threats Three Main Areas of Vulnerability: 1. Intentional disruption of switch processing
‒ CPU hogging from unnecessary queries ‒ Denial-of-service attacks
Result: switch can‘t react to fabric events
2. Compromised fabric stability
‒ Altered/lost switch configurations ‒ Removal of other security services ‒ Disabled switches/ISLs/device ports
Result: loss of service, unplanned down time
3. Compromised data integrity and secrecy
‒ Altered target (and LUN) visibility ‒ Altered zoning configuration
Result: LUN corruption, data corruption, data theft, or loss
Out-of-Band Ethernet Management Connection
Accidental or Intentional Harmful
Management Activity
138
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
SAN Management Security Securing access to all management
facilities on MDS SAN
‒ Must secure console sessions
‒ Must secure GUI application access
‒ Must secure API access (SMI-S)
‒ Must also secure file transfer
to/from switch
Equally important to
enable audit mechanisms
‒ Integrated RADIUS for user accounting and
switch scope assignment
‒ Integrated syslog for
switch-event accounting
‒ Integrated SNMP traps for
access-denial accounting
‒ Network time protocol (NTP) support to
synchronize clocks, log entry time stamps
SAN Management Security Infrastructure
Management
Network
RADIUS Server for
User Authentication
switch> config t
switch(config)>
analyzer on
switch(config)>
exit
switch>
Cisco Fabric
Manager
Using SNMPv3
NX-OS CLI
Using SSH/SFTP
TACACS+ Server for
User Authentication
SNMP Polling Server
Using SNMPv3
NTP Server
for Time/Date
Synchronization
Integrated RFC 2625 IP-over-FC Provides
Redundant IP Connectivity for Security Services over In-Band
FC Link
Out-of-Band Ethernet
Management
Connection
139
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Used for AAA (Authentication, Authorization, and Accounting) services
‒ Limit management access to a subset of switches
‒ MDS supports up to five HA server definitions
RADIUS—Remote Authentication Dial In User Service (IETF RFC-2865 standard)
‒ Initially used for dial-in networks—now greatly expanded to a variety of uses
System user account centralized authentication
Network-device user account AAA services
Dial-in/VPN service AAA services
iSCSI host authentication
TACACS+—Terminal Access Controller Access Control System (based on RFC-1492)
‒ Widely used and supported by Cisco
‒ Freely available from Cisco—similar to RADIUS
Flexible RADIUS and TACACS+ Services
Cisco MDS SAN
Authentication Calls and Accounting
Records Are Sent to Centralized RADIUS or TACACS+ Servers
RBAC Role Membership Info Is Authorized by RADIUS/TACACS+ Servers
RAD
LDAP Server
DB
Database Server
(Oracle, mySQL,
etc.)
Roles Are Populated into MDS Switches
Dial/VPN Servers for Remote
Access System Console Terminal Servers
Network Management
Stations
NMS
Datacenter Routers and
Switches
RADIUS and TACACS+ Deployments
RAD
Windows 2000 IAS Server (RADIUS)
Linux TACACS+Server
Microsoft Active
Directory
Redundant Server
140
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Configuration Consistency Analysis
Important to keep consistent
configurations across all switches
‒ Especially important for security configurations:
RADIUS/TACACS+, remote syslog, NTP,
SNMP communities, authentication, and roles
Configurations can be
extracted from switches as a flat text file
‒ Allows for easy and regular archiving
Cisco Fabric Manager provides
fabric configuration analysis tool
‒ Checks all switch configurations against
policy switch or file
‒ Can take corrective action to fix configurations
‒ Also has zone-merge analysis tool to
validate zone-merge validity
Policy Reference
Switch
Define Analysis
Rules
Fabric Configuration Analysis Part of
Cisco Fabric Manager
Review Results and Take
Corrective Actions
Administrator Compares Policy Reference Config
to All Switches in Fabric
141
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
SAN Management Recommendations
Use RBAC to grant adequate privilege to SAN administrators
‒ Example: not every administrator needs capability to disable modules
‒ Reserve select functions to fewer super-admin RBAC role:
‒ VSAN definition, firmware upgrades, roles definition, RADIUS, and SSH configuration
Use RADIUS or TACACS+ for centralized user account administration
‒ Ensures consistent and timely removal of users if required
‒ Use RADIUS accounting feature for audit log of configuration events
Use all secure forms of management protocols—disable others
‒ SSH, SFTP, SCP, SNMPv3, SSL for SMI-S support
‒ Disable Telnet, FTP, TFTP, SNMPv1,v2
Enable NTP across all switches for consistent time stamping of events
Log and archive everything
‒ Enable centralized syslog
‒ Take regular copies of switches configurations (can use CiscoWorks RME)
‒ Turn on MDS call-home feature to alert of anomalies
142
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Fabric and Target Access Potential Threats Three Main Areas of Vulnerability: Compromised application data
‒ Unauthorized access to targets and LUNs
‒ High potential for data corruption, loss, or theft
Result: unplanned down time, costly data loss
Compromised LUN integrity
‒ LUN corruption due to unintentional OS mount
‒ Accidental formatting of LUN—loss of data
Result: unplanned down time, costly data loss
Compromised application performance
‒ Unauthorized I/O potentially causing congestion
‒ Injected fabric events causing disruption; i.e., rogue HBA hammering fabric controller
Result: unplanned down time, poor I/O performance
Unauthorized Fabric Service
Unauthorized Target Access
143
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Fabric Access Security: Port Modes Port-mode security—allow edge ports
to form F_Ports or FL_Ports only,
i.e., no ISL/EISL
‒ MDS supports an Fx_Port mode which allows F_Port or
FL_Port only
‒ Limit users who can change port mode via
roles-based access control assignments
VSAN-based security—only allow access to
devices within attached VSAN
‒ Strict isolation based on fabric service
partitioning and explicit frame tagging
‒ Independent name server table per VSAN
‒ Independent active zoneset per VSAN
‒ Part of ANSI T11 fabric expansion
study group
Management port access security
‒ Provides IP access control lists (ACLs) for management
traffic (SNMP, SSH, Telnet, etc.)
IP Access Lists (ACL) Based on Source and Destination IP Addresses, TCP/UDP Ports, and TCP Connection Flags
Any Port Type
Auto Mode
E_Port Mode
F_Port Mode
Fx_Port Mode
F, FL Only F Only
Fx_Port Mode
E_Port or Auto Mode
Management Network
Port Mode and VSAN-Based Security
VSAN 1
VSAN 2
Both
Disk Array Connected to
Multiple VSANs
Unique Services
per VSAN
One Active VSAN Only
EISLs Carrying Multiple VSANs
144
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Fabric Access Security
MDS access security technology
‒ Grant selective access to fabric based on device identity
‒ Failure results in link-level login failure
‒ Prevents FC frame S_ID spoofing through hardware frame filtering
Supports switch-to-switch (fabric binding) and device-to-switch (port security)
‒ Auto-learning mode to ease initial configuration
Uses grouping of attributes to define binding configuration
‒ WWN or Port_ID – port identifier on switch (i.e. fc1/2)
‒ Multiple groups are created and activated as a group set to enforce desired policy
Default configuration
‒ Set port administrative default value to SHUT
‒ Do not put ports in VSAN 1
‒ Ports by default in VSAN 4094 (isolated)
145
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Fabric Access Security: Fabric Binding
Used to allow
ISL establishment
Attributes to define binding
configuration:
‒ fWWN—fabric WWN
of switch port
‒ sWWN—switch WWN
‒ Port_ID—port identifier
on switch (i.e., fc1/2)
fWWN-1 Port_ID-1
sWWN-2
pWWN-1
nWWN-1
pWWN-2
sWWN-1
fWWN-2 Port_ID-2
fWWN-3 Port_ID-3
fWWN-4 Port_ID-4
fWWN-6 Port_ID-6
fWWN-5 Port_ID-5
pWWN-3
pWWN-4
nWWN-2
sw-2
Security Group—sw-1 sWWN-2
Bind sw-2 to sw-1 ISL
Security Group – sw-1 sWWN-2
Port_ID-5 or fWWN-5
Bind sw-2 to sw-1/port 5 ISL
sw-1
146
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Fabric Access Security: Port Security
Used to allow device-to-switch login
Attributes to define binding configuration
‒ pWWN—port WWN of attaching device
‒ nWWN—node WWN of attaching device
‒ fWWN—fabric WWN of switch port
‒ Port_ID—port identifier on switch (i.e. fc1/2)
Bind Host to sw-1 (Any Port)
Bind Host to sw-1/port 2
Bind Host, disk to sw-1 (Any Port)
Security Group – sw-1 pWWN-1 or nWWN-1
Security Group – sw-1
pWWN-1 or nWWN-1 Port_ID-2 or fWWN-2
Security Group – sw-1
pWWN-1 or nWWN-1 pWWN-3 or nWWN-2
Security Group – sw-1
pWWN-1 Port_ID-2 or fWWN-2 Bind Host HBA-1 to sw-1/port 2
fWWN-1
Port_ID-1
sWWN-2
pWWN-1
nWWN-1
pWWN-2
sWWN-1
fWWN-2
Port_ID-2
fWWN-3
Port_ID-3 fWWN-4
Port_ID-4
fWWN-6
Port_ID-6
fWWN-5
Port_ID-5
pWWN-3
pWWN-4
nWWN-2
sw-2
sw-1
147
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Fabric Access Security: Authentication
Device authentication provides
stronger means of ensuring
device identity
‒ WWNs can be spoofed by
simple means
ANSI T11 FC-SP security
protocols working group
‒ Cisco was the prime contributor
DH-CHAP provides
authentication mechanism
‒ Switch-to-switch authentication
‒ Device-to-switch authentication
(when adopting HBA supporting
DH-CHAP)
Fibre Channel Fabric Authentication
Management Network
RADIUS Server for User
Authentication
RAD
TACACS+ Server for User Authentication
RADIUS and TACACS+ Servers Can Be Used to Hold DH-CHAP User
Accounts and Passwords for Centralized Authentication
Out-of-Band Ethernet Management Connection
New Switch Wanting to
Join the Fabric
New Host Wanting to Join
the Fabric
Equipped with HBA
Supporting DH-CHAP (Emulex, Qlogic)
DH
-CH
AP
FCIP Network
New Switches Wanting to Join the Fabric over FCIP
148
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Fabric Access Recommendations Use IP ACLs on management interfaces to block unused services
‒ Enable logging of denied attempts—block denial-of-service attacks
Hard-fix switch-port administrative modes to assigned port function
‒ Lock (E)ISL ports to only be (T)E_Ports—set to E_Port mode
‒ Lock access ports to only be F(L)_Ports—set to Fx_Port mode
Use VSANs to isolate departments
‒ Provides security and availability benefits
‒ RBAC management control per VSAN allows individual admin assignment
Use port security features everywhere
‒ Bind devices to switch as a minimum level of security
‒ Bind devices to a port as an optimal configuration
Consider binding to line card in case of port failure
‒ Bind switches together at ISL ports—bind to specific port, not just switch
Use FC-SP authentication for switch-to-switch fabric access
‒ Use device-to-switch when available 149
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
Fabric Protocols Potential Threats
Three Main Areas of Vulnerability:
Compromised fabric stability
‒ Injection of disruptive fabric events
‒ Creation of traffic black-hole
Result: unplanned down time, fabric instability
Compromised data security
‒ Injection of harmful zone reconfiguration data
‒ Open access to fabric targets
Result: unplanned down time, costly data loss
Compromised application performance
‒ Unauthorized I/O potentially causing congestion
‒ Numerous disruptive topology changes
Result: unplanned down time, poor I/O performance
Fabric Control Protocol Integrity
Rogue Switch
150
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121
SAN Fabric Protocols Security
Very important to secure the fabric control protocols to ensure fabric stability ‒ Securing access to control protocol
configuration via Cisco RBAC is first step
‒ Enable port-security for switch binding
‒ Using FC-SP for switch-to-switch authentication is next critical step to block rogue ISLs
Plug-n-play fabric protocol configuration is convenient—however, static configuration is more secure ‒ Configure static principle switch
‒ Enable static domain IDs
‒ Enable static FCIDs optional but recommended
Great benefit for HP/UX and AIX environments
‒ Enable RCF-reject, especially on long-haul links
‒ Enable RSCN-suppression where necessary
Use VSANs to divide and manage individual fabric configuration and resiliency
Cisco MDS 9216
Dept ‗A‘
VSAN Trunk
Bundles
Cisco MDS 9500
Multilayer Director
Port Channeling for HA and
Performance
Fabric Protocols Security Dept ‗B‘
DWDM or CWDM Network
VSAN Trunks
over Optical
Enterprise Tape VSAN
1 1 2 2
3 3 4 4 3 4
5 5 6 6 1 7 2 8
Statically Assigned domain_IDs, One per Active VSAN
Minimizes Potential for Disruptive RCF
Statically Assigned Principle Switch
per VSAN
RCF-Reject Configured to Protect Against Remote Initiated Fabric Rebuild
151