VMware EMC Service Talk

Copyright © 2013 EMC Corporation. All Rights Reserved.

VMware EMC Service Talk

May 2014Kevin Wang


Question 1: The detailed procedure for HBA initiates login to SAN to complete the connection to the SP (Answers cover both FC and FCoE)


This lesson covers the following topics:• FC protocol make-up• FC components

Lesson 1: Fibre Channel Protocol Overview

Fibre Channel and FCoE Technology 3


Fibre Channel Layered Architecture: FC-0

ULP Upper Layer ProtocolSCSI-3, IP, ESCON/FIPS, etc.

FC-4 FCP (SCSI-3 Fibre Channel Link Encapsulation),Single-Byte Command Code Sets (FICON), etc.

FC-3 Common Services (Not Used)

FC-2 Exchange and Sequence Management, Frame Structure ,Flow Control

FC-1 Encode/Decode, Link Control Protocols,Ordered Sets

FC-0 Physical Interface, Optical and Electrical Interfaces, Cables, Connectors, etc.



Transceivers Cable Connectors Comments

SFP, SFP+ LC

• Most popular for Fibre Channel today

• SFPs up to 8 Gbps• SFP+ > 8Gbps• versions for different distances

mSFP mLC

• mini-SFP and mini-LC • Designed by Brocade for 64-

port blade

GBIC SC

• Technology found in older 1 Gbps equipment

• Replaced by LC technology

Optical Transceivers and Cable Connectors



Fibre Channel Cables and Distances

Transceiver Type Speed

Multimode media maximum distance

62.5 um(OM1)

50 um(OM2)

50 um(OM3)

50 um(OM4)

SW

2 Gbps 150m 300m 500m 500m

4 Gbps 70m 150m 380m 400m

8 Gbps 21m 50m 150m 190m

10 Gbps 33m 82m 300m 550m

16 Gbps 15m 35m 100m 125m

Single-mode media maximum distance (9 um core optic fiber)

LWL

4 Gbps

10 km to 40 km depending on vendor

8 Gbps

10 Gbps

16 Gbps

Note: For greater distances, you may connect using DWDM / CWDM between switches .



Basic Port Types

Node Port: Transmits and receives Fibre Channel data frames in a Switched Fabric environment (N_Port)

Fabric Port: Port on a switch that N_Port connects to. (F_Port)

Expansion Port: Port on a switch that has another switch plugged into it (two switches that are merged) (E_Port)

Node Loop Port: Transmits and receives Fibre Channel frames in an Arbitrated Loop (private) environment (NL_Port)

Fabric Loop Port: Allows a Fabric to communicate with an Arbitrated Loop (public) (FL_Port)

Generic Port: Port on a switch that detects the attached port type and auto-configures to match that port type (G_Port)



The Host Bus Adapter

Fabric

Target Storage Array

Server

HBA

NIC

LAN

Ethernet / IP

See EMC Support Matrix at https://elabnavigator.emc.com




ULP

FC-4

FC-3

FC-2

FC-1 Encode/Decode, Link Control Protocols,Ordered Sets

FC-0

Data encoding/decoding

• Less than 10 Gbps (20% overhead for encoding) 8-bits 10-bits

• 10 Gbps and 16 Gbps (3% overhead for encoding) 64-bits 66-bits

• Special Characters Used with data characters to create ordered sets Ordered sets include: Idle, SOF, EOF, R_RDY




ULP

FC-4

FC-3

FC-2 Exchange and Sequence Management, Frame Structure ,Flow Control

FC-1

FC-0

• Defines structure of Fibre Channel frame Frames, Sequences, Exchanges

• Flow control mechanisms

• Error handling and recovery

• Provides class of services Class 1, Class 2, Class 3, Class 4, Class 6, and Class F



FC-2

• Defines the structure of the Fibre channel frame How information to large to fit into a single frame is transferred by

using a sequence of frames How operations that consist of exchanges of sequences of frames

are managed

• Describes how the rate of frame transmission is controlled by the flow control mechanisms

• Error handling and recovery• Class of Service

Class 1, Class 2, Class 3, Class 4, Class 6, and Class F



Exchange, Sequence, Frame

• Exchange is an operation manager SCSI task, an IP connection Has an Originator Has a Responder Has one or more Sequences

• Sequence is a ULP object to be transmitted in one direction that may be one frame or longer

• Frame is the unit of information transfer• Manages flow control (buffer-to-buffer and end-to-end)



Conversations

Session

Exchange Exchange Exchange

Seq Sequence Sequence SequenceSeqSeq

Frame #1 Frame #1

Frame #2

Frame #3

Frame #1

Frame #2

Frame #3

Frame #4

Frame #5

Frame #6

Frame #1

Frame #2

Frame #3

Frame #4

Frame #1

Frame #2

Frame #1



Fibre Channel Frame

Data Field End-of-Frame

CRC

Start-of-Frame

Header

Optional Header

Fill Bytes

Payload(information being transported)

Optional Header

R_CTL

CS_CTL

TYPE

SEQ_ID DF_CTL SEQ_CNT

Frame Control (F_CNT)

Source Address (S_ID)

Destination Address (D_ID)

OX_ID RX_ID

Parameter Field (PARM)



Credit-Based Flow Control

Fibre ChannelSwitch

Physical ISL

FC Frame R_RDY

Credit = 16 1516

Fibre ChannelSwitch

Long Distance links may require additional BB_Credit




ULP

FC-4 FCP (SCSI-3 Fibre Channel Protocol) mapping,Single-Byte Command Code Sets (used for FICON) mapping

FC-3 Common Services (Not Used)

FC-2

FC-1

FC-0

• Mapping Layer Maps Upper Level Protocol to lower levels SAN typically uses SCSI FCP mapping



FC-4 and Upper Level ProtocolExchange, Sequence, and Frame

...Information Unit # 1 Information Unit # 3Information Unit # 2...FC-4

FC-2

Sequence # 1 Sequence # 3Sequence # 2... ...

Frame 1

...

Frame 1 Frame 2 Frame N Frame 1

Payload: SCSI CDB DATA Payload: Status

Exchange (Operation)

DATA DATA

SCSI Command SCSI ResponseSCSI DATAULPSCSI-3:

One SCSI Read I/O Operation



EMC Fibre Channel Implementations

• Point-to-Point – Link between two ports using Arbitrated Loop

• Arbitrated Loop (FC-AL) – Shared interconnect between 2 to 126 nodes

• Switched Fabric (FC-SW) – Dynamic connectivity support for more than 15 million nodes



Fibre Channel Fabric

Target Storage Array

Fabric

Server

HBA

Server

HBA

ISL



Fibre Channel Address (FCID)

• Required to route frames from source to target• 24-bits (three-bytes) - specified in Hex

Assigned at Fabric Login (FLOGI)

• Divided into three parts:

Domain_ID Area_ID Port_ID

Domain (01-EF)

(1-239 in decimal)

00-FF 00-FF

Example: 6b 04 00



Common Fabric Services: Well Known Addresses

Service NameWell

Known Address

Description

Login Service(F_Port Server)

FFFFFE All nodes register in login service when performing FLOGI

Name Service FFFFFCStores information about nodes attached to fabric during the Fabric Login process

Fabric Controller FFFFFD

State change notification to nodes registered in the fabric and inter-switch communications



World Wide Name

• A World Wide Name (WWN) is a 64-bit value used in Fibre Channel networks to uniquely identify each element in the network

• Usually assigned to a Host Bus Adapter (HBA), switch port, or storage port by the vendor at the time of manufacture

• Often linked within a component, such as an HBANode WWN – A name for all ports within the single HBAPort WWN – A name for each individual port



WWPN Examples

EMC VNX World Wide Port Name Example

5 0 0 6 0 1 6 0 4 7 2 0 5 5 9 0

EMC Company ID (24-bits) Port VNX Seed (from SPE/DPE midplane)

EMC VMAX World Wide Port Name Example

5 0 0 0 0 9 7 2 0 8 1 3 4 9 a d

EMC Company ID for VMAX (24-bits) VMAX Serial Number - Decodes to country where manufactured/Model /

etc. (HK192601234)

CPU/Dir/P(12g port B)

Emulex World Wide Port Name Example

1 0 0 0 0 0 0 0 c 9 2 0 d c 4 0

Reserved (12-bits)

Company OUI (24-bits) Company Specific (24-bits)



FC Port Initialization Process - Step 11 - Link online2 - Fabric Login - get FCID3 - Port Login to fabric name server4 - Registrations with name server5 - Query Name server for FC-4 devices6 - State Change RegistrationRepeat for each Target:

7 Port Login to Target port 8 Process Login to Target port 9 SCSI INQ

Fabric 1

5

1 2

3 4HBA

WWN = 50:06:04:8a:d5:2e:76:c4FCID = 040600

WWN = 10:00:00:00:c9:48:f1:c7FCID = ??????

6




7 - Port Login to Target port 8 - Process Login to Target port 9 - SCSI INQ

Fabric 1

5

1 2

3 4HBA

FLOGI

ACC

Accept frame includes FCID for HBA 030500

FLOGI FrameDestination Address = FFFFFESource Address = 000000

WWN = 50:06:04:8a:d5:2e:76:c4FCID = 040600

WWN = 10:00:00:00:c9:48:f1:c7FCID = ??????

6





Fabric 1

WWN = 50:06:04:8a:d5:2e:76:c4FCID = 040600

WWN = 10:00:00:00:c9:48:f1:c7FCID = 030500

5

1 2

3 4HBA

PLOGI

ACC

PLOGI FrameDestination Address = FFFFFCSource Address = 030500

6





Fabric 1

5

1 2

3 4HBA

WWN = 50:06:04:8a:d5:2e:76:c4FCID = 040600

WWN = 10:00:00:00:c9:48:f1:c7FCID = 030500

6

Fibre Channel Services Frames• Register Class of Service • Register FC-4 Type• Register Port type• Register other (vendor

dependent)

FC Serv

ACC

FC Serv

ACC





Fabric 1

5

1 2

3 4HBA

WWN = 50:06:04:8a:d5:2e:76:c4FCID = 040600

WWN = 10:00:00:00:c9:48:f1:c7FCID = 030500

6

Fibre Channel Services Frames• Query: Get PIDs • Query: Get Port WWN for PID1• Query: Get Port WWN for PIDn

FC Serv

ACC

FC Serv

ACC

Name Server Database:• PID = 040600

• FC-4 = SCSI FCP• WWPN =

50:06:04:8a:d5:2e:76:c4• . . .

• PID = 030100 . . .• PID = 040400 . . .





Fabric 1

5

1 2

3 4HBA

WWN = 50:06:04:8a:d5:2e:76:c4FCID = 040600

WWN = 10:00:00:00:c9:48:f1:c7FCID = 030500

6

SCR

ACC

SCR FrameDestination Address = FFFFFDSource Address = 030500





Fabric 1

5

1 2

3 4HBA

WWN = 50:06:04:8a:d5:2e:76:c4FCID = 040600

WWN = 10:00:00:00:c9:48:f1:c7FCID = 030500

6PLOGI





Fabric 1

5

1 2

3 4HBA

WWN = 50:06:04:8a:d5:2e:76:c4FCID = 040600

WWN = 10:00:00:00:c9:48:f1:c7FCID = 030500

6PRLI

SCSIProces

s

SCSIProces

s





Fabric 1

5

1 2

3 4HBA

WWN = 50:06:04:8a:d5:2e:76:c4FCID = 040600

WWN = 10:00:00:00:c9:48:f1:c7FCID = 030500

6

SCSIProces

s

SCSIProces

sSCSI INQ



Fibre Channel Login at a Glance

Process FLOGI PLOGI PRLI

Requirements - Link initialization- Cable- HBA and driver- Switch port

- FLOGI- Zoning- Persistent binding- Driver setting

- PLOGI- Device masking (Target)- Device mapping (Initiator)- Driver setting (Initiator)

Information passed - WWN- S_ID- Protocol- Class- Zoning

- WWN- S_ID- ULP- Class- BB Credit

- LUN

Session members - N_Port to F_Port - N_Port to N_Port - ULP (SCSI-3 to SCSI-3)

Verification UNIX

WINDOWS

- Syslog- Switch Utilities

- Syslog- Driver Utilities

- Syslog- Host based volume management

- Event Viewer- Switch Viewer

- Driver utilities - Driver utilities- Host based volume management- Device manager

Fibre Channel and FCoE Technology - 33


Zone Set: Fabric 1Zone A: Initiator A; Target wZone B: Initiator B; Target xZone C: Initiator C; Target y; Target z. . .

Zoning Overview

Storage(Targets)

Hosts(Initiators)

I

T

Initiator (HBA) Target

Zone A

Fabric 1



Fabric 1

How to Define Zone Members

I

T

Target Port WWPN:50:06:04:8a:d5:2e:76:c4

Zone A

Initiator Port WWPN:10:00:00:00:c9:48:f1:c7

5 3

1 2

3 4

Zone A10:00:00:00:c9:48:f1:c7;50:06:04:8a:d5:2e:76:c4

Zone Members Defined by WWPN

Zone A0305ef; (Domain_ID 03, port_05)0403ef; (Domain_ID 04, port_03)

Zone Members Defined by Domain/Port



This lesson covers the following topics:• Architecture of an Ethernet solution

Lesson 2: Ethernet Overview



Ethernet and the OSI Model

7 - Application

6 - Presentation

5 - Session

4 - Transport

3 - Network

2 - Data Link

1 - Physical

LLC - Logical Link ControlMAC - Media Access Control

Ethernet is defined at Layer 2 of the OSI model



Ethernet PDU

Destination MAC6 bytes

Source MAC

6 bytes

802.1Q tag

(optional)

4 bytes

Type2

bytes

Data Payloadvariable # bytes

FCS4

bytes

Ethernet_II Frame Format

Header



Ethernet MAC Address

IG

GL

Organizationally Unique Identifier (OUI)

(Assigned by IEEE)Vendor assigned

24 bits 24 bits

Examples:

Broadcast address: ff:ff:ff:ff:ff:ff

Physical address of port on PC: 00-26-5E-82-3E-6E

47 46 45 24 23 0



MAC Address Used to Direct Frame Through Network

MAC: 00-26-5E-82-3E-6E MAC: 00-26-5E-77-2F-58

…2F:58 …3E:6E 08-00

Payload FCS

Ethernet Frame

This is the source address This is the destination address

Destination MAC



Switching Frames Using MAC Addresses - Step 1

MAC TablePort 1 - EmptyPort 2 - EmptyPort 3 - EmptyPort 4 - Empty

PC A PC C

PC B

PC D

Port 1

Port 2

Port 3

Port 4

Frame

Dest Src

PC A




MAC TablePort 1 - PC APort 2 - EmptyPort 3 - EmptyPort 4 - Empty

PC C

PC B

PC D

Port 1

Port 2

Port 3

Port 4

No entry for PC D, so flood the frame out all

ports

PC A




MAC TablePort 1 - PC APort 2 - EmptyPort 3 - EmptyPort 4 - Empty

PC A PC C

PC B

PC D

Port 1

Port 2

Port 3

Port 4

Frame

Dest Src

PC A PC D

PC D



The Problem of Loops and Broadcast Storms

Switch A

Switch C

Switch B

Broadcast



How STP Solves the Problem

• STP is a loop-prevention protocol

• Uses spanning tree algorithm to elect a Root Bridge and block certain ports.

Switch A

Switch C

Switch B

Block



How Does STP Protocol Elect a Root Bridge?

Switch A

Switch C

Switch B

Priority = 32768MAC = 0000.1111.2222

Priority = 32768MAC = 0000.2222.3333

Priority = 32768MAC = 0000.2222.2222

• Switch with lowest Bridge ID (BID) wins election

• If all switches have equal priority, then switch with lowest MAC address wins

Since all switches are set to default priority, this switch will be elected Root Bridge because it has the lowest MAC address

Bridge ID



Result of running STP Protocol

Switch A

Switch C

Switch B

Root Bridge

Designated Port (F)

Root Port (F)

AlternatePort (D)

Nonroot Bridge

Nonroot Bridge

Root Port (F)

Designated Port (F)

Designated Port (F)

• Forwarding ports Root ports Designated ports

• Discarding port Alternate



Ethernet Hardware Overview

• Ethernet hardware supports the following speeds 10 Mbps 100 Mbps 1 Gbps 10 Gbps 40 Gbps 100 Gbps

Hardware Popular Implementations

NIC(Network Interface Controller)

• Stand-alone card• Built into host motherboard• Wireless

Cables

• Copper UTP (Unshielded Twisted Pair)

• Copper Twinax• Optic

Connectors• RJ45 (for copper UTP cables)• SFP+ (for copper twinax cables)• LC (for fiber optic cables)

Most common

Data Center Core (Backbone)



Network Interface Controller - NIC

• Provides access to the network• Different implementations include:

Stand-alone card (could have multiple ports) Built into motherboard (most common now) Wireless

• Ports can be “teamed” to act as a single logical interface for failover

NIC LAN



Example Ethernet Cabling

UTP (Unshielded Twisted Pair)

Twinax with SFP+Fiber Optic Cable with LC Connector

Media Copper Copper Optic

IEEE Standard 1000BaseT (802.3ab)

10GSFP+CU (Direct Attach)

1000BaseLX (802.3z)

Distance/speed

100 meters @1 Gbps 5 meters @ 10 Gbps 10 km @ 1 Gbps

Cat 5

RJ45

Twinax / SFP+ 9/125 SM FiberLC Connector



Issues With a Flat Switched Network

• Single broadcast domain

• No Security between users

Broadcast



VLAN200

VLAN200

VLAN100

VLAN100

VLAN Overview

• Each VLAN is a separate broadcast domain

• Each switch port is assigned to a single VLAN

• Devices in different VLANs must use a router to communicate

• VLAN Trunks allow the data from multiple VLANs to pass on a single link between switches

Trunk

Trun

k



How a VLAN Solves the Problems of the Flat Network

• Smaller broadcast domains

• Security between VLANs

VLAN100

VLAN200

Trunk

VLAN100

VLAN200

VLAN100

VLAN200

TrunkTrunk

Broadcast Broadcast



How it Works: VLAN Tagging

• 802.1Q - IEEE Standard Each port on switch assigned to specific VLAN ID Unassigned ports automatically belong to Default VLAN (VLAN ID = 0001)

• When port receives frame: Compare VLAN ID to filter table.

If valid, and MAC on another switch - forward to trunk port If destined to exit an access port on switch, remove VLAN ID

• Trunk carries both tagged and untagged traffic

Destination MAC6 bytes

Source MAC

6 bytes

802.1Q tag

(optional)

4 bytes

Type2

bytes

Data Payloadvariable # bytes

FCS4

bytes

Ethernet_II Frame Format

VLAN Tag



This lesson covers the following topics:• FCoE Protocol and FCoE Fabric design basics

Lesson 3: FCoE Protocol Overview



FCoE Protocol Stack

Volume managers, File systems, etc.

SCSI Command Descriptor Blocks, data, and responses

FCoE frame encapsulation / de-capsulation, FCoE entities

Frame switching, MAC address, transport Connection to the Physical Layer

Fibre Channel Protocol frame construction FC-4, FC-3, FC-2

Application

SCSI

FCP

FCoE

Enhanced Ethernet

FCoE Protocol Stack



FCoE Encapsulation

SO

F

EO

F

EthernetHeader

FCoEHeader

Encapsulated FC Frame(including FC-CRC)

FC

S

FC Header

Payload

Fill

Dest. MAC

Address

SourceMAC

AddressVersion Reserved

VLANTAG C

RCEther

Type

Fibre Channel Frame

Ethernet Frame

FCoE PDU



FCoE Frames

• Frame Efficiency iSCSI, FCIP, iFCP rely on IP over Ethernet which result in additional

overhead in frames FCoE frames only use Ethernet protocol Efficiency can be calculated by the formula

• Jumbo Frames Ethernet frames with more than 1500 bytes of payload Considered unreliable on regular Ethernet CEE allows reliable use of Jumbo Frames FCoE must use jumbo frames (2240 bytes)

Number of Data Bytes

Total Bytes Sent



Converged Network Adapter (CNA) Overview

CNACNA = HBA + NIC + FCoE

Host



• CNA is both a Network Adapter Fibre Channel adapter

• Host sends I/O to appropriate adapter: NIC – NFS, Web, Email, etc. HBA – SCSI block data

• Host receives data through lossless Ethernet MAC Directs I/O to appropriate adapter Based on Ethernet “Type”

Converged Network Adapter Architecture

2 x 10 Gbps Ethernet Ports

Ethernet NIC

Host Networking Drivers

FCoE Encapsulation

Existing FC Host Drivers

CNA Adapter

Lossless Ethernet MAC

Is it FCoE?NO YES

HBA Architecture



Fibre Channel Forwarder – FCF

• FCF function provided by FCoE switch or FCoE blade Services FCoE Login requests Provides services associated with a Fibre Channel switch

Stand-aloneFCoE Switch Fibre Channel Switch

with FCoE blade



• Lossless Ethernet: FCoE requires reliable frame delivery Can’t drop frames from buffer overflow

• Improves regular Ethernet: Eliminates Ethernet loss, therefore more

reliable Speed increased to 10 Gbps

• Additions to Ethernet: Priority-based Flow Control (PFC) Enhanced Transmission Selection (ETS) DCB Capability Exchange (DCBX)

Notify neighbor node of DCB support

Data Center Bridging (DCB)

FCoE Requires LosslessEthernet



Enhanced Ethernet Features

Feature Enhancement

Priority Flow Control IEEE 802.1Qbb – class of service flow control by enabling PAUSE functionality on IEEE 802.1p lanes

Data Center Bridging Exchange Auto-negotiation of Enhanced Ethernet capabilities DCBX (switch to NIC)

Bandwidth ManagementIEEE 802.1Qaz Enhanced Transmission Selection – manage bandwidth and assign priorities to groups of IEEE 802.1p lanes based on class of traffic.



Priority-based Flow Control

STOP

Transmit Queues Receive Buffers

0

1

2

3

4

5

6

7

Buffers are almost full - send “PAUSE”

Priority



Enhanced Ethernet - DCBX

• DCBX runs between the switch and CNA

• DCBX negotiates capabilities• Switch distributes configuration

to all attached adapters• Devices need to discover the

capabilities of its peers.• DCBCX utilizes the link-layer

discovery protocol and handles local operational configuration for each feature

Priority Flow Control

Data Center Bridging Exchange

Bandwidth Management

Rules for Capabilities Result

Capability and configuration match Enabled

Adapter configured to accept switch config. Enabled

Adapter not configured to accept switch config. Disabled

Adapter does not support/implement DCBX Disabled



Enhanced Ethernet – Bandwidth Management

• Enables consistent management of QoS

• Provides consistent scheduling of different traffic types

Priority Flow Control

Data Center Bridging Exchange

Bandwidth Management



Addressing

MAC = AFCID = 050100

MAC = BDomain ID = 05 Domain ID = 01

FCID = 011C00

FC SANConvergedNetwork

FC Frame

D_ID = 011C00S_ID = 050100

Dest = MAC BSource = MAC A

D_ID = 011C00S_ID = 050100

Ethernet Frame



A Note About CNA MAC Addresses

FCoE Controller

Lossless Ethernet MAC

FCoE_LEP

VN_Port

FC-3 / FC-4 Layers

Ethernet_Port

Ethernet NIC

Host Networking Drivers

Existing FC Host Drivers

Physical MAC

ENode MAC

FPMA MAC

CNA



How are MAC Addresses Assigned to the VN_Port?• Server-provided MAC Address (SPMA):

Uses single MAC Address for all FCoE traffic FCoE has MAC address independent of other protocols Support for different MAC addresses for each Virtual Port This method isn’t used

• Fabric-provided MAC Address (FPMA): Uses virtual port MAC addresses Each virtual port MAC address is supplied by the fabric Most common method Have the following format:



Server Provided MAC Addresses (SPMA)

• Adapter uses burned-in or configured MAC address Consistent with the Ethernet model

• FCF needs a table to map between MAC addresses and FC_IDs

FCoE Node Port

MA

C

00.05.69.00.00.03

MAC Address



Fabric Provided MAC Addresses (FPMA)

• MAC address assigned for each FC_ID: Consistent with the Fibre Channel model

• Uses OUIs with U/L = 1 FC-MAP (Local addressing) • No table needed for encapsulation

FC-MAP(0E-FC-00)

FCID07.08.09

24 bits 24 bits

48 bit MAC Address

FC-MAP(0E-FC-00)

FCID07.08.09



FCoE Initialization Protocol (FIP) and FC Logins

CNA

CNA (ENode)Lossless Ethernet

Switch (FIP Snooping Bridge)

FCoE Switch(FCF)

Storage

FCoE Switch(FCF)

FC SAN

FC SAN

Storage

ENode (FCoE Controller)

Send FIP VLAN Request

Send FIP Solicitation

Send FIP FLOGI

FCF

Reply FIP VLAN Notification

Reply FIP Advertisement

Reply FIP FLOGI ACC

FIP Multicast



FIP, the FCoE Initialization Protocol – How does it work?



FIP VLAN Request



FIP Discovery Solicitation



FIP Discovery Advertisement



FIP Virtual Link Instantiation Request (FLOGI)



FIP Virtual Link Instantiation Request (FLOGI ACC)



FIP Snooping Bridge (FSB)



Question 2: How SP coordinates the host to handle hardware assist locks (Reservation, ATS)


Verify VAAI Support Details

• The compare write(F1) operation is exclusively used by VAAI enabled VMware host


VAAI – Hardware Assisted Locking• Scalable Lock Management – Atomic Test and Set (ATS)

• A number of VMFS operations cause the LUN to temporarily become locked for exclusive write use by one of the ESXi nodes, including:– Moving a VM with vMotion– Creating a new VM or deploying a VM from a template– Powering a VM on or off– Creating a template– Creating or deleting a file, including snapshots

• VAAI feature, atomic_test_and_set (ATS) allows the ESXi host to offload the management of the required locks to the storage and avoids locking the entire VMFS file system


Atomic Test & Set• Original file locking technique

1. Acquire SCSI reservation2. Acquire file lock3. Release SCSI reservation4. Do work on VMFS file/metadata5. Release file lock

• New file locking technique

1. Acquire ATS lock2. Acquire file lock3. Release ATS lock4. Do work on VMFS file/metadata5. Release file lock

Note - The main difference with using the ATS lock is that it does not affect the other ESXi hosts sharing the datastore


VMFS Scalability with Atomic Test and Set

Makes VMFS more scalable overall, by offloading block locking mechanism

Using Atomic Test and Set (ATS) capability provides an alternate option to use of SCSI reservations to protect the VMFS metadata from being written to by two separate ESXi hosts at one time.

Normal VMware Locking (No ATS)

Enhanced VMware Locking (With ATS)


How to resolve SCSI Reserves on VMware ESXi servers• Refer to the following VMware KB articles:

– SCSI Reservation Issue with Fibre Channel HBAs (4365932) – Troubleshooting SCSI Reservation failures on Virtual Infrastructure 3.x and

vSphere 4.x (1005009) – VMware KB Article 1002293

• Warning! DO NOT release the reservation with INLINES when customer is using vSphere.

• It is also NOT recommended to try to release via Solutions Enabler

• The CORRECT way to release a SCSI Reserve in ESXi is to ask the customer to use OS tools as described in the VMware documentation and KB articles that are listed.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=4365932






http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1002293&sliceId=1&docTypeID=DT_KB_1_1&dialogID=23224094&stateId=0%200%2023226185

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1002293&sliceId=1&docTypeID=DT_KB_1_1&dialogID=23224094&stateId=0%200%2023226185


Question 3: The terms or tricks to match WWPN with SP port, which would help us to understand mpath failover sequence


During this lesson the following topics are covered:• PowerPath configuration and PowerPath’s function in the I/O

path• How PowerPath supports EMC and non-EMC arrays• Key functionality included in current releases of PowerPath• How PowerPath is licensed• Hardware, software, clusters, and storage arrays that

PowerPath supports

Lesson 1: PowerPath Architecture

87


EMC PowerPath - A Family of Products

• PowerPath traditionally viewed only as “Path Management Software” Performed Multipathing, load balancing, failover, etc.

• Functionality Expanded PowerPath Family now has 4 distinct products

PowerPath can now be used for solutions previously handled by separate software

Common Interface allows for simplified management

• Optimized for heterogeneous host / storage environments

88

PowerPath Multipathing Path

Management /

OptimizationPowerPath/VE

PowerPath Migration Enabler Data

ProtectionPowerPath Encryption


• PowerPath sits between the applications and the HBA driver

• Applications direct I/O to PowerPath

• PowerPath directs I/O to an optimal path based on current workload and path availability

PowerPath Configuration

89

O p

e n

S y

s t

e m

s H

o s

t

Logical volume manager

File system

PowerPath Driver

Applications

ManagementutilitiesDBMS

Storage Array

Host busadapter

Host busadapter

Host busadapter

Host busadapter

Storage Area Network


• PowerPath keeps a table of paths

• Presents a single device to the application

• Combines equivalent paths into a single “Path Set”

Volume Path Set

90

PowerPath

HBAHBAS

ER

VE

RS

TO

RA

GE

Front-end Ports


Volume Path Set

Application

HBAHBA

POWERPATH

“Power”Device


PowerPath Array Support Concepts

91

SE

RV

ER

ST

OR

AG

E

SE

RV

ER

ST

OR

AG

E

Active-Active Arrays Active-Passive Arrays

Owned by SP-A

HBA HBA HBAHBA

SAN

SP-A

Active

Active

SP-B

Passive

Passive

PowerPath

0 1 0 1

Host Application(s)

Mapped to allFront-end ports

Active

Active

Active

Active

SAN

Front-end Ports

HBA HBA HBAHBA

PowerPath

Host Application(s)


• Dead indicates the path is

not usable. PowerPath driver

will not direct user I/O to this logical I/O path.

• Alive indicates the path is

usable The PowerPath

driver can direct I/O to this logical I/O path.

Device (LUN) Path States

92

SE

RV

ER

ST

OR

AG

E

Front-end Ports


Host Application(s)

HBA HBA HBAHBA Host BusAdapter

PowerPath

A

B


PowerPath Licensing

• PowerPath Unlocks the full load balancing and path failover capabilities of PowerPath and

supports all EMC and certain non-EMC arrays Configures optimized default load balancing policies for EMC VMAX, VNX and

supported third-party arrays

• PowerPath/VE Full feature set specifically for virtual server environments based on VMware

vSphere and Windows Hyper-V

• PowerPath SE Backend (between switch and array) failover only Single HBA supported only No load balancing available Path Failover across two backend paths

93

PowerPath SE


During this lesson the following topics are covered:• Benefits of PowerPath/VE• Features and functionality of PowerPath/VE• PowerPath/VE configuration in an ESXi environment

Lesson 2: Multipathing in ESXi

94


What is PowerPath/VE?

• PowerPath advanced multipathing capabilities for virtual environments hosted on VMware vSphere

• Features include: Dynamic load balancing Device prioritization Automated optimization of server, storage, and path utilization Dynamic path failover and path recovery Monitoring performance and path status, I/O statistics and reports Automatic path testing Automatic path restore

95

x86 Architecture

PowerPath


PowerPath/VE Key Benefits

• Automates the virtual server-storage connection: Assures tunable, predictable performance Optimizes server, storage, and data-path utilization Maximizes data availability

• Uses standard management features across physical and virtual environments: PowerPath/VE experience similar to rest of environment (physical servers) Common user experience across all classes of supported platforms

• Saves Money: Simple to implement application service levels for performance and

availability Eliminates need to monitor and rebalance the dynamic environment Reduces number of multi-pathing software variants to manage

96


• PowerPath manages complexity: Constantly adjusts I/O path usage

and changes in I/O loads from VMs Simplifies provisioning by pooling

all connections Optimizes overall I/O performance

in VMware ESX environments

PowerPath/VE Load Balancing

97

OS

APP

OS

APP

OS

APP

OS

APP

OS

APP

OS

APP

OS

APP

OS

APP

OS

APP

OS

APP

OS

APP

OS

APP

OS

APP

OS

APP

OS

APP

OS

APP

ST

OR

AG

E

SAN

PowerPath PowerPath PowerPath PowerPath


PowerPath/VE supports:• Changes to Permanent

Device Loss (PDL) handling • VMFS 5 Filesystem

vSphere 5.0 Kernel Support

98

EMC and Non-EMC Storage

Host Bus

Adapter

Host Bus

Adapter

Host Bus

Adapter

Host Bus

Adapter

vSphere KernelPowerPath/VE

Guest OS

Applications

DBMS

Manageme

ntUtilitiesFile System

Logical Volume Manager

PowerPath/VE runs in the kernel


PowerPath/VE – Two Major Components

99

PowerPath Remote Management Server

ESX Hosts

PowerPath remote tools (rpowermt) installed

Used to remotely manage PowerPath/VE on any number of ESX hosts

PowerPath software is installed



• Supported platforms: Can be a physical or virtual host Check ESM for version compatibility

• Software: rpowermt

Sends PowerPath management commands to a remote ESX host Is the only way to manage PowerPath/VE (no local commands)

VMware vSphere CLI Installs PowerPath remotely onto an ESX host

License file Uses rpowermt to register license with an ESX host

• Can manage any number of ESX hosts.

100



PowerPath Remote Tools

• Are installed on the PowerPath Remote Management Server• Provide remote versions of existing PowerPath commands:

Example: Local command used by previous versions of PowerPath: powermt display dev=all Remote command, supported by PowerPath/VE: rpowermt host=<hostname> display dev=all

• Note: Remote Tools version and PowerPath version must be compatible: Possible to update either Remote Tools or PowerPath, without updating the

other

101


Installation and Configuration Steps

1. Upload Files to ESXi Server2. Install PowerPath software on the ESXi server3. Install PowerPath Remote Tools on PowerPath Remote

Management Server (Optional)4. Configure and register license file

102


Upload Files to ESXi Server

103

Create New Folder

Upload File or Folder


Install PowerPath on ESXi

104

# cd /vmfs/volumes/Storage1/PPVE/# lsPowerPath_VE_5_7_for_VMWARE_vSphere-Install_SW_Bundle.zip# unzip PowerPath_VE_5_7_for_VMWARE_vSphere-Install_SW_Bundle.zip. . .# esxcli software vib install --depot=/vmfs/volumes/Storage1/PPVE/ PowerPath_VE_5_7_for_VMWARE_vSphere-Install_SW_Bundle/EMCPower. VMWARE.5.7.b173.zip

Installation Result Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

Reboot Required: true VIBs Installed: EMC_bootbank_powerpath.cim.esx_5.7.0.00.00-b173, EMC_bootbank_powerpath.lib.esx_5.7.0.00.00-b173, EMC_bootbank_powerpath.plugin.esx_5.7.0.00.00-b173

VIBs Removed: VIBs Skipped:


Reboot ESXi Server

105


Verify PowerPath /VE Installation

106

# cd /opt/emc/powerpath/bin/opt/emc/powerpath/bin # lspowermt/opt/emc/powerpath/bin # ./powermt versionEMC powermt for PowerPath (c) client Version 5.7 (build 173)EMC PowerPath (c) Version 5.7 (build 173)

Warning: License not installed.

/opt/emc/powerpath/bin #


THANK YOU

Documents

VMware EMC Service Talk