John L. Hufferd, Brocade - snia.org · Fibre Channel over Ethernet (FCoE) John L. Hufferd, Brocade includes material courtesy of: Claudio DeSanti, Cisco

Fibre Channel over Ethernet (FCoE)

John L. Hufferd, Brocadeincludes material courtesy of:

Claudio DeSanti, Cisco

2Fibre Channel over Ethernet (FCoE)© 2007 Storage Networking Industry Association. All Rights Reserved.

SNIA Legal Notice

The material contained in this tutorial is copyrighted by the SNIA. Member companies and individuals may use this material in presentations and literature under the following conditions:

Any slide or slides used must be reproduced without modificationThe SNIA must be acknowledged as source of any material used in the body of any document containing material from these presentations.

This presentation is a project of the SNIA Education Committee.


Abstract

A new concept is currently moving through the Fibre Channel (T11) standards committee called Fire Channel over Ethernet (FCoE). The FCoE standard will specify the encapsulation of Fibre Channel frames into Ethernet Frames and the amalgamation of these technologies into a network fabric that can support Fibre Channel protocols and other protocols such as TCP/IP, UDP/IP etc.The tutorial will show the Fundamentals of the FCoE concept and describe how it might be exploited in a Data Center environment.


Agenda

Introduction

Goals & Requirements

Consolidation

Architecture

Topologies

Flows

Scenarios

Summary


Introduction

This presentation provides an overview of Fibre Channel over Ethernet (FCoE)

One should think about FCoE as placing the FC protocol on a new physical link

Ethernet links instead of physical FC linksBut it is still Fibre Channel

The protocol is being defined in the INCITS Fibre Channel (T11) technical committee

Many details of the protocol still need to be defined but some of the significant major issues have been resolved

Target completion is the 2H08


The Origins of Data Center Ethernet

The technology has evolved continuously, showing a great ability to adapt to new technologies and increasing business requirements

Increasing Scalability, Feature, Function

Incr

easi

ng P

erfo

rman

ce

Data Center Ethernet{Converged Enhanced Ethernet (CEE)}

Integration of Carrier-grade features

Incremental Protocol Enhancements

Logical Partitioning

Evolution from shared media to dedicated media

1973

2008-2009

Introduction of Ethernet

10Mbps

100Mbps

1Gbps

10Gbps


Goals/Requirements


FCoE Goals/Requirements (part 1)

Connection to a physical Fibre Channel fabric is not required (but must be possible)FCoE Fabrics are built with FCoE Switches

Switches with Ethernet ports that provide FCoE capabilities and servicesSwitches that include the functions of traditional FC switchesStandard Ethernet switches may also exist in the fabric but switches with FCoE capabilities are required

FCoE fabrics must operate seamlessly with real FC FabricsTranslation between FCoE fabrics and Physical Fibre Channel fabrics must be efficient & high-performing FC services must operate identically on FCoE fabrics and Fibre Channel fabricsFCoE must support all Fibre Channel advanced features(e.g. virtual fabrics, IFR, security, etc.) transparently



FCoE requires specific Ethernet extensions to be implemented

Lossless switches and fabrics (e.g., supporting IEEE 802.3 PAUSE) configurations are requiredJumboframe support is required (not a standard, but widely available)

Deployments of FCoE should utilize the advances in Ethernet currently being discussed in IEEE 802.1, specifically:

Priority based Flow Control (PFC)&

Selective Transmission

These are important for Consolidated Flows (Messaging, Clustering and Storage)

This set of functions has been called DCE --Data Center Ethernet, or CEE – Converged Enhanced Ethernet (intended for a Data Center Environment)



FCoE must be a direct mapping of Fibre Channel over an Ethernet network

FCoE must be layered on top of EthernetFSPF used to route FCoE packetsEthernet Spanning Tree (STP), MSTP, etc, is at a layer below

FCoE to allow an evolutionary approach towards consolidation of fabrics

The Fibre Channel N_Port, F_Port and E-Port constructs must be retained

With FCoE, ports may be connected with Logical Ethernet Links– May pass through Ethernet switches– Identified by pairs of end point MAC addresses

Physical Ethernet Links can replace physical FC Links Physical Ethernet Links can carry all Ethernet traffic, including FCoE, but combined traffic needs the CEE capabilities



“Combo FCoE Switches” may be built that support normal Ethernet traffic, FCoE traffic , & FC trafficThe FCoE solutions should appear as a Fibre Channel to a Fibre Channel experienced customerFCoE should keep the Fibre Channel operations independent from Ethernet forwarding

Keeps management /Troubleshooting simpleCommon physical structures, different logical structures

Based on Ethertype (Ethertype = FCoE)

Storage Management should be unchangedFCoE is NOT a replacement for FCIP or iFCP

FCIP & iFCP use TCP/IPFCIP/iFCP is for inter-switch links beyond the Data Center


Consolidation


Dramatic reduction in adapter, switch ports and cabling4-6 cables to 2 cables per server

Seamless connection to the installed based of existing SANs and LANsHigh performance lightweight frame mappers vs. heavy weight gateways

No need to terminate and re-initiate a SCSI connection(e.g. iSCSI to FC)

Effective sharing of high bandwidth links

High End 10GE Server & NIC/HBA Consolidation

Today

With CEEOS3 DB Server

OS2 App Server

OS1 Web Server

Messaging

MPIRDMA

FC HBA

OS3 DB Server

OS2 App Server

OS1 Web Server

IB/Ethernet Cluster

Hyper Visor(vmWare, Xen, etc)

E-HBA(CEE)

Data CenterCEE Network

EthernetTCP/IP

FC SAN

Data CenterCEE Network

E-HBA(CEE)

E-HBA(CEE)

•NIC•TCP acceleration.•MPI, RDMAover Ethernet

•FCoE

OS3 DB Server

OS2 App Server

OS1 Web Server


Architecture


FC Encapsulation Into Network Packets

IEEE 802.3Layers

FC Levels(Unchanged)

FCoE Mapping

FC-4

FC-3

FC-2

FC-1

FC-0

FC-4

FC-3

FC-2

MAC

PHY

Protocol control information: Version, SOF, EOF, etc.

FC Imbedded Frames: Same as in Physical FC

EthernetHeader FCSFCoE

HeaderFC Header SCSI

Commands/Data

Provides things needed for the physical network, including “Ethertype = FCoE”


FC’s Encapsulation in Ethernet (FCoE)

Word 31-24 23-16 15-8 7-0

Ver (4b)

n+1 EOF (8 bits) Reserved

0 Destination MAC Address (6 Bytes)

12 Source MAC Address (6 Bytes)

3 ET=FCoE (16 bits) Reserved (12 bits)

4 Reserved5 Reserved6 Reserved SOF (8 bits)

7…

n

n+2 Ethernet FCS

Encapsulated FC FrameFC Frame = Minimum 28 Bytes (7 Words)

Maximum 2180 Bytes (545 Words)(including FC-CRC)

Optional IEEE 802.1q4 ByteTag goes here

This field varies

In size

Ethernet framesizeIs 64Bytes to 2220Bytes


FCoE Switch

An FCoE switch:A Switching device that also Forwards & may MapFC and FCoE frames to one another

Maps FC Frames into FCoE frames or FCoE Frames into FC Frames Forwards FCoE/FC Frames between FCoE/FC compatible devices

Provides FC services to FC & FCoE compatible devices

Fabric LoginsPort LoginsStorage Names ServicesEtc.


FCoE Switch (cont.)

An FCoE -- N_Port, F_Port or E_Port has the same function as in FC

But is layered on top of EthernetFCoE Pathing and Forwarding utilizes the FSPF (Fabric Shortest Path First) protocol

Note: Non-FCoE Ethernet traffic is relayed using conventional 802.1 defined mechanisms such as STP (Spanning Tree Protocol) and MSTP (Multiple STP)

Requires no changes to FC softwareMany vendors will implement an Ethernet Switch combined with an FCoE-Switch


Connections to a Combo FCoE Switch

Applications

TCPUDPIP

SCSI

Fibre Channel

FCoE

Lossless Ethernet MAC (CEE)

• Fibre Channel is carried over lossless Ethernet as a L3 protocol

Combo Lossless Ethernet (CEE) Switchwith FCoE Switch capabilities

(FCoE N_Port)

Ethernet port with IP & FCoE F_Port capabilities

IP address 123.45.67.89

iSCSI

SCSI


Integrated Ethernet & FCoE switch

Some implementations may combine the features and capabilities of an Ethernet Switch with the features and capabilities of a FC switch

• Support Ethernet and IP standards for switching, pathing and routing

• Support FC standards for switching, pathing and routing

• Support current and enhanced Ethernet Standards

• Adapt between FCoE and FC

FCFCFCEthernet – Standard & CEE

(with FCoE and IP, etc.)


Examples of FCoE Switch Models

Logical FC Pathing & RoutingLogical FC Services

Physical FC Pathing & Routing

Physical FC Services

Physical FC Switching Physical Ethernet Switching

Shown are two different very high level examples of Switching implementations that can create an FCoE Switch

Implementations will have the capability to handle multiple logical port parings (i.e. FCoE N_Ports/F_Ports and E_Ports logically paired not physically 1:1)

Different vendors will apply different approaches (these and others)But

All implementations should be able to interoperate since the observed behaviors on the wire/fiber must be identical

Logical Ethernet Pathing & RoutingPhysical Ethernet Pathing & Routing

FCFCFCEthernet – Standard & CEE

(with FCoE and IP, etc.) FCFCFCEthernet – Standard & CEE

(with FCoE and IP, etc.)


Topologies


FCoE Switch & FC Fabric

FCFCFCFCoE Switch

An FCoE Switch may connect to a normal FC switchVia the FC E-Port Ethernet

FC

Note: FCoE servers and storage will probably use an FCoE HBA (or chip set)


Combo FCoE Fabric

FCFCFCLossless Combo FCoE Fabric

A Lossless Ethernet Fabric can be made up of Combo FCoE Switches Ethernet

FC


Ethernet Fabric and FCoE Switch

FCFCFCLossless EthernetFabric

FCoE Switch

Lossless Ethernet switches configured into a Lossless Ethernet Fabric can Front the FCoE Switch Ethernet

FC


FCoE Edge Switches

FCFCFC

FCoESwitch

FCoESwitch

FCoESwitch

FCoE Switches deployed at the edges of the Lossless Ethernet Fabric

FCoE Switches connected via E_Ports and Lossless Ethernet

Lossless Ethernet

EthernetFC


Logical Fabric topology

FCFCFC

FCoESwitch

FCoESwitch

LosslessEthernet

1

LosslessEthernet

2

LosslessEthernet

4

EthernetFC

FCoESwitch

An FCoE Switch may connect to other FCoE Switch via the FCoE E_Ports

An FCoE Switch may connect to an FC switch via the FC E_Port

LosslessEthernet

3


Status of Current Data Center Networking

There can be 3 different networks

System Area Network (SyAN)Used for Clustering/Low Latency

Storage area Network (SAN)Used to Access to Storage

LAN/WAN External NetworksUsed for General messagingUsed for Client-Server MessagingUsed for NAS

Often divided into at least 3 management domains

Data Center Server (clustering) NetworkData Center Storage NetworkOutfacing (IP) Network

LAN/WANs– Messaging– NAS

Data Center

Remote Offices

System Area Network (SyAN)•Clustering Fabric•InfiniBand•Myrinet•Ethernet•CTC

Data CenterServer & Storage Network

Management GroupOutfacing (IP) Network

Management Group

LAN/WAN •Messaging•NAS

Storage Area Network- Fibre Channel

Note: with multiple Data Centers there may also be interconnects with DWDM, FCIP/iFCP, etc.


Current FabricsRemoteRemoteOfficesOffices

Clustering Network

There is a FC Storage Network, a Clustering Network & an “Outfacing” NetworkIP Network Management Group

FC

FC

FC

Ficon

Ficon StorageController

Mainframe

Data Center Server & Storage Network Management GroupFile Storage Arrays (NAS)

iSCSIStorage

FCNetwork

Ficon

(LAN/WAN)(LAN/WAN)••MessagingMessaging••NASNAS

Outfacing IP Network

Local & Remote Local & Remote Business CampusBusiness Campus

Focus: Low Latency & High

Bandwidth

Focus: Protection, Bandwidth/Congestion

Management

FC Link

EthernetLink

Ficon Link

Clustering Network



View of the Future Data Center Networks

Remote Remote OfficesOffices

Data Center Server & Storage Network

Management Group

Outfacing (IP) NetworkManagement Group

Storage & System Storage & System Area NetworkArea Network••Logically Single FabricLogically Single Fabric

(with FCoE)(with FCoE)

Other IP NetsOther IP Nets(LAN/WAN)(LAN/WAN)••MessagingMessaging••NASNAS

The The Server/Storage NetworksServer/Storage Networkswill become a Consolidated will become a Consolidated FabricFabric

Managed by the Data Center Managed by the Data Center System and Storage System and Storage Management GroupManagement Group

–– Includes Storage and Includes Storage and clustering provisioningclustering provisioning

–– SAN and SyAN managed SAN and SyAN managed as a single fabricas a single fabric

Focus: High Bandwidth and Focus: High Bandwidth and Low LatencyLow Latency

The Outfacing (The Outfacing (IP) Management IP) Management GroupGroup remains the sameremains the same

Focus: Protection, Bandwidth Focus: Protection, Bandwidth and Congestion Managementand Congestion Management



Phase in: FC & High Performance Lossless Ethernet Unify into a Data Center Fabric

Remote Remote OfficesOffices

FC

FC

FCoE

FC

(LAN/WAN)(LAN/WAN)••MessagingMessaging••NASNAS

Outfacing IP Network

Ficon

iSCSIStorage

Ficon

Ficon StorageController

Mainframe

File Storage Arrays (NAS)

• FCoE permits intermixing of multiple Connection types/protocols• Clustering messaging, General Messaging, and Storage

• The DataCenter Fabric will “Trunk” to the “Outfacing” Network (including iSCSI sys)• Some Customers may want keep a mixed environment on-going

DataCenterFabric

EthernetSW

EthernetSW

FC & EthernetSW Blades

FC Link

EthernetLink

Ficon Link

Business Campus

with iSCSI connections

Including iSCSI Gateways

Note: with multiple Data Centers there may also be interconnects

with DWDM, FCIP/iFCP, etc.


Flows


Logical Fabric Topology

FCFCFC

FCoESwitch

FCoESwitch

FCoESwitch

LosslessEthernet

LosslessEthernet

A

H1

H2

H3H4 S2

EthernetDestination

& Source

EncapsulatedFC Frame

D_ID

S_ID

FCoE-A MACFCoE-H2 MAC

FC_ID for S1FC_ID for H2




FCoE-B MACFCoE-A MAC

FCoE-C MACFCoE-B MAC

FCoE-S1 MACFCoE-C MAC

EthernetFC

LogicalTransaction Path

An FCoE Switch receives FCoE frames addressed to its FC-MAC address and forwards them based on the D_ID of the encapsulated FC frame

An FCoE Switch rewrites the SA and DA of an FCoE frame

LosslessEthernet

LosslessEthernet

S1

Path #1 Path #2 Path #3 Path #4

#1#2

#3

#4B

C


Single Ethernet Fabric with FCoE Switches

FCFCFCFCoE

SwitchFCoE

SwitchLossless Ethernet

AC

H1

H2

S1

S2

H3

FCoE-A MACFCoE-H2 MAC


D_ID

S_ID




FCoE-C MACFCoE-A MAC

FCoE-S1 MACFCoE-C MAC

EthernetDestination

& Source

Ethernet Switch

Ethernet Switch

Ethernet Switch

Ethernet Switch

Ethernet Switch

EthernetFC


Path #1 Path #2 Path #3

#1

#2 #3


FC Host to FCoE Storage

FCFCFCFCoE

Switch

FCoESwitch

Lossless EthernetH1

H2

S1

S3

H3

FCoE-S2 MACFCoE-A MAC


D_ID

S_ID




FCoE-A MACFCoE-C MAC

EthernetDestination

& Source

Ethernet Switch

Ethernet Switch

Ethernet Switch

Ethernet Switch

H5S2

Ethernet Switch

EthernetFC


Path #3 Path #2

Path #1

#1

#2

#3

CA


Scenarios


Scenario 1: FCoE & Messaging

FCFCFC

LosslessEthernet

ClassicalEthernet

FCoESwitch

Internet

FCoESwitches

LosslessEthernet


Scenario 2: FCoE Right & Wrong

LosslessEthernet

FCFCFC

LosslessEthernet

ClassicalEthernet

FCoESwitch

Internet

FCoESwitches

LosslessEthernet


Summary


Summary

FCoE is a simple, efficient mechanism for encapsulating Fibre Channel in Ethernet frames

FCoE is being standardized in INCITS Fibre Channel (T11) technical committee

Target completion is 2H08

Maximum benefit of Fibre Channel is achieved:Evolutionary model of FC Switches and FC SANsEmphasis placed on capitalizing on the benefits of Converged Enhanced Ethernet (CEE)

Being discussed in the IEEE 802.1 standards working group

Thank You!


Q&A / Feedback

Please send any questions or comments on this presentation to SNIA: [email protected]

For additional information refer to http://www.t11.org/fcoe

Many thanks to the following individuals for their contributions to this tutorial.

SNIA Education Committee

Claudio DeSanti Howard Goldstein Walter DeyRobert Snively Suresh VobbilisettyJoe Pelissier John Hufferd

mailto:[email protected]

http://www.t11.org/fcoe


Appendix


Additional Architecture, Flows & Topologies


Additional Topologies (1)

FCoESwitch (A)

FCoESwitch (B)

FCoESwitch (F)

LosslessEthernetSwitch

FCoESwitch (E)

LosslessEthernet Switch

FCFCFC

Example of Topologies with Rack Mount servers


Additional Topologies (2)

FCoESwitch (B)

FCoESwitch (E)

FCoESwitch (F)

LosslessEthernet Switch

LosslessEthernetSwitch

FCFCFC

FCoESwitch (A)

Equivalent to Blade servers with N_Port_ID Virtualization (NPIV) SupportExample of Topologies with Blade Servers


FCoE Relation to ISO Layers


Pause vs. BB_CreditBoth mechanisms allow to avoid dropping frames

With different trade-offs

The Pause mechanism requires at least the (2 x RTT x bandwidth) product on a link as buffer space

But allows Buffer handling in an arbitrary wayWell suited for networks with limited (bandwidth x delay) product (e.g. within the data center)

The Pause frame is handled by the MAC layerSimilar to the R_RDY handling by the FC-1 level

The BB_Credit mechanism prevents loosing frames over any link

But links go under-utilized if link credits (& buffers) are < that needed for (RTT x BW)Requires buffer handling in maximum frame size units

Documents

John L. Hufferd, Brocade - snia.org · Fibre Channel over Ethernet (FCoE) John L. Hufferd, Brocade includes material courtesy of: Claudio DeSanti, Cisco