Basic San

HP Training

Student guide

Introduction to SANsRev. 4.21HP Restricted

HP Training

Student guide

Introduction to SANsRev. 4.21HP Restricted

© Copyright 2004 Hewlett-Packard Development Company, L.P.

The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

This is an HP copyrighted work that may not be reproduced without the written permission of HP. You may not use these materials to deliver training to any person outside of your organization without the written permission of HP.

Printed in the United States

Introduction to SANsStudent guideJune 2004

HP Restricted — Contact HP Education for customer training materials.

Rev. 4.21 HP Restricted 1

Contents

Overview Facilities ................................................................................................................... 2 Introductions............................................................................................................. 3 Course prerequisites ................................................................................................. 4 Course objectives ..................................................................................................... 5 Course outline .......................................................................................................... 7 Additional resources — weblinks .......................................................................... 15

Module 1 — Introduction to SANs Objectives ................................................................................................................. 2 What is a Storage Area Network (SAN)?................................................................. 3 Introduction to SAN Components ............................................................................ 5 What Makes the SAN Attractive? ............................................................................ 7 Storage Consolidation and Expansion...................................................................... 9 High Performance Backup ..................................................................................... 11 Serverless Backup .................................................................................................. 13 Fault Tolerance/High Availability.......................................................................... 15 Direct Attached Storage ......................................................................................... 17 Network Attached Storage ..................................................................................... 18 Storage Area Networks........................................................................................... 20 DAS, NAS, and SAN ............................................................................................. 21

Introduction to SANs

2 HP Restricted Rev. 4.21

Module 2 — Fibre Channel Technologies Objectives ................................................................................................................. 2 Fibre Channel ........................................................................................................... 3 FC Protocol Architecture.......................................................................................... 4

FC-0 Level — Physical.................................................................................... 5 FC-1 Level — Encode/Decode ....................................................................... 6 FC–2 Layer ...................................................................................................... 8

Fibre Channel Frame ............................................................................. 10 Classes of service ................................................................................ 12

FC-3 and FC-4 ............................................................................................... 14 Fibre Channel cabling ............................................................................................ 15

Transceivers and cables.................................................................................. 16 Single-Mode Fiber ......................................................................................... 17 Single-Mode Step-Index Fiber ...................................................................... 18 Multimode Fiber ............................................................................................ 19 Multimode Step-Index Fiber ......................................................................... 20 Multimode Graded-Index Fiber .................................................................... 21

Attenuation ............................................................................................................. 22 Dispersion .............................................................................................................. 23 Macro Bending ...................................................................................................... 24 Micro Bending ....................................................................................................... 25 Fibre Channel extension ......................................................................................... 26 Long Wave transceivers ......................................................................................... 27 Wavelength division multiplexing ......................................................................... 28 Fibre Channel over Internet Protocol (FCIP) ......................................................... 29

Contents


Module 3 — San Topologies Objectives ................................................................................................................. 2 Fibre Channel topologies.......................................................................................... 3

Point-to-point ................................................................................................... 4 FC-AL .............................................................................................................. 6

Arbitrated loop hubs ................................................................................ 8 World-Wide Name (WWN) .................................................................... 9 WWN identifiers.................................................................................... 10 Loop initialization.................................................................................. 11 Private loop............................................................................................ 12 Public loop ............................................................................................. 13 Port logins – PLOGI .............................................................................. 14

Switched Fabric topology .............................................................................. 15 Fabric addressing ................................................................................... 16 Fabric login – FLOGI ............................................................................ 18 N_Port login – PLOGI........................................................................... 19 Fabric addressing- B series.................................................................... 20 Simple Name Server .............................................................................. 21 Translative Mode ................................................................................... 22

Module 4 — Brocade Fibre Channel Addressing Objectives ................................................................................................................. 2 Native Brocade Addressing Modes ......................................................................... 3 CORE_PID Addressing Mode ................................................................................. 4

CORE_PID Addressing Modes ............................................................... 5 Problem – solution.................................................................................................... 6 Extended Edge PID .................................................................................................. 7

Extended Edge advantage ................................................................................ 8 Configuring PID ....................................................................................................... 9

Module 5 — Configuring a Brocade Switch Objectives ................................................................................................................. 2 Quick Install SilkWorm 3800 Switches ................................................................... 3 Power-On Self Test .................................................................................................. 4 Brocade Silkworm 3800 Serial Port ......................................................................... 5 Setting Initial IP Address on Br3800........................................................................ 6 Logon........................................................................................................................ 7 Upgrading the Firmware........................................................................................... 9 Factory Default Settings ......................................................................................... 11 Configuration Parameters ....................................................................................... 13 Using Show Commands ......................................................................................... 15



Module 6 — Brocade Simple Name Server Objectives ................................................................................................................. 2 Brocade Distributed Simple Name Server at Common Address FFFFFC............... 3 SNS: Port and Nodes attributes ................................................................................ 4 Fabric login – FLOGI............................................................................................... 5 N_Port login – PLOGI.............................................................................................. 6

Entire n-port login sequence ............................................................................ 7 NsShow..................................................................................................................... 9 NsAllShow ............................................................................................................. 10 Registered State Change Notification .................................................................... 11 Worldwide Name identification ............................................................................. 12 PortShow ................................................................................................................ 13 SNS Common Addresses ....................................................................................... 14

Module 7 — Brocade Zoning Objectives ................................................................................................................. 2 Security Comparisons............................................................................................... 3 Overview – Brocade zoning product ........................................................................ 4 Zoning Example ....................................................................................................... 5 Zoning Components ................................................................................................. 6 Zoning enforcement mechanisms............................................................................. 8 2x00 zoning .............................................................................................................. 9

2x00 zoning examples.................................................................................... 10 3x00 zoning ............................................................................................................ 11

3x00 zoning examples.................................................................................... 12 Soft Porting .................................................................................................... 13 Zoning Rules(3x00) ...................................................................................... 14

Port Zoning............................................................................................................. 15 World-Wide Name Zoning..................................................................................... 16 Zoning commands ................................................................................................. 17 Zone Management Commands .............................................................................. 21 Creating a Configuration Example......................................................................... 26 Changes to the Fabric ............................................................................................. 27 Zoning Examples ................................................................................................... 29

Contents


Module 8 — Cascading Brocade Switches Objectives ................................................................................................................. 2 PATH and ROUTE Terminology............................................................................. 3 Brocade Path Selection............................................................................................. 5 Paths and routes........................................................................................................ 6

Fabric shortest path first (FSPF) ...................................................................... 7 Path selection and link costs ........................................................................... 8 Route selection default behavior.................................................................... 10 Brocade Routing Modes................................................................................. 13 Dynamic Load Sharing ................................................................................. 14 Load sharing .................................................................................................. 16 Configuring Static Routes .............................................................................. 18

What is ISL Trunking? ........................................................................................... 19 ISL Trunking.................................................................................................. 20 Before ISL Trunking...................................................................................... 21 ISL Trunking.................................................................................................. 22 ISL Trunking Commands............................................................................... 23 IslShow........................................................................................................... 24 Trunkable Switch ........................................................................................... 26 The Deskew counter....................................................................................... 27 TrunkShow..................................................................................................... 29 PortCfgTrunkPort........................................................................................... 30 SwitchCfgTrunk............................................................................................. 31 TrunkDebug ................................................................................................... 32 PortCfgShow.................................................................................................. 33 Dynamic Load Sharing ................................................................................. 34 FSPF For Trunking ........................................................................................ 35 Trunking Summary ........................................................................................ 36

Routing Algorithm.................................................................................................. 37 Routing Examples .................................................................................................. 38 In Order Delivery .................................................................................................. 43 Fabric data flow...................................................................................................... 45 Fabric Timing Values ............................................................................................. 46 URouteShow........................................................................................................... 47 TopologyShow ....................................................................................................... 49 uRouteShow and topologyShow ........................................................................... 50 Cascading HUBs .................................................................................................... 53

Port bypass circuit .......................................................................................... 54 Cascading Switches [1 hop] ................................................................................... 55 Meshing Switches [>1 hop].................................................................................... 56 How to increase fabric fan out ?............................................................................. 57 What would happen here? ...................................................................................... 58

Connecting two switches to a hub.................................................................. 59 Cascading two switches through a hub .......................................................... 60 Switched fabric vs. loop................................................................................. 61 Link utilization............................................................................................... 62 Balance inter-switch links .............................................................................. 63



HP standard SAN fabric topologies ....................................................................... 64 Cascaded fabric topology............................................................................... 65 Ring fabric topology ...................................................................................... 66 Meshed fabrics ............................................................................................... 67 Backbone fabrics............................................................................................ 68 Advantages of backbone fabrics .................................................................... 69 Fat tree and skinny tree designs ..................................................................... 70 Fat vs. skinny example................................................................................... 71

HA considerations .................................................................................................. 72 HA Recommendations............................................................................................ 73 Watch the Max Hops!............................................................................................. 74

Module 9 — Storage Presentation to Operating Systems Objectives ................................................................................................................. 2 Verifying adapter connectivity ................................................................................. 3 OpenVMS connectivity ............................................................................................ 4 Tru64 Unix connectivity........................................................................................... 5 HP-UX connectivity ................................................................................................. 7

Verify HBA installation ................................................................................... 8 HP-UX ioscan utility output descriptions ....................................................... 9

Windows connectivity ............................................................................................ 13 LPUTILNT..................................................................................................... 14

OpenVMS device discovery................................................................................... 15 OpenVMS multipath .............................................................................................. 16 OpenVMS multipath discovery.............................................................................. 17 Viewing device details in OpenVMS .................................................................... 18 ioscan device discovery.......................................................................................... 19 HP-UX device naming ........................................................................................... 20 HP-UX ioscan utility output ................................................................................... 21 Device discovery for Tru64 UNIX V5.x ............................................................... 22 Device naming for Tru64 UNIX V5.x .................................................................. 24 Windows disk manager ......................................................................................... 26

Contents


Module 10 — Secure Path Objectives ................................................................................................................. 2 Definition.................................................................................................................. 3 Features..................................................................................................................... 4 Support ..................................................................................................................... 5 Secure Path for Microsoft Windows ........................................................................ 6

Components ..................................................................................................... 7 Software components ....................................................................................... 8 Secure Path process.......................................................................................... 9 Profiles ........................................................................................................... 10 Configuration limits ....................................................................................... 11 Load balancing............................................................................................... 12 Device states .................................................................................................. 13 Moving LUNs to other controller .................................................................. 14 General installation tips ................................................................................. 16 Windows installation tips............................................................................... 17 Troubleshooting ............................................................................................. 18

Secure Path 3.0C HP-UX description .................................................................... 22 Secure Path V3.0C support limits .................................................................. 23 Secure Path 3.0C HP-UX switch support ..................................................... 24 Secure Path 3.0C HP-UX components .......................................................... 25 Secure Path 3.0C HP-UX driver model ......................................................... 27 Secure Path 3.0C HP-UX IOSCAN .............................................................. 28 Secure Path 3.0C HP-UX SCSI-2 CCL ......................................................... 30 Secure Path 3.0C HP-UX spmgr commands ................................................. 31 Secure Path 3.0C HP-UX dynamic LUNs ..................................................... 34 Secure Path 3.0C HP-UX troubleshooting..................................................... 35

Launching Secure Path Manager............................................................................ 39

Module 11 — SAN Troubleshooting Objectives ................................................................................................................. 2 About Troubleshooting............................................................................................. 3 Tracing problems...................................................................................................... 4 Analysis .................................................................................................................... 5 Common problem areas............................................................................................ 6 Troubleshooting tools............................................................................................... 7 Gather troubleshooting data ..................................................................................... 8

SupportShow command ................................................................................... 9 errShow command ......................................................................................... 10 switchShow command ................................................................................... 11 portshow......................................................................................................... 12 portstatsshow.................................................................................................. 13

Hardware diagnostics ............................................................................................. 14 Helpful Commands ................................................................................................ 15 Interpreting LED Activity ...................................................................................... 18 Rear Panel LEDs .................................................................................................... 21



Module 12 — Fibre Channel Switches Supported Fibre Channel switches ........................................................................... 2

Brocade SilkWorm 3800.................................................................................. 3 McDATA Sphereon 3216 ................................................................................ 4 McDATA Sphereon 3232 ................................................................................ 5 McDATA Sphereon 4500 ................................................................................ 6 Cisco MDS9120............................................................................................... 7 Cisco MDS9216............................................................................................... 8

Introduction to SANs Course overview

Rev. 4.21 Overview – 1

© 2004 Hewlett-Packard Development Company, L.P.The information contained herein is subject to change without notice


Course overview



2Rev. 4.21 HP Restricted

Facilities

• Telephones

• Restrooms

• Smoking area

• Breaks

• Lunch

• Class time




Introductions

• Name• Location• Years with company• Experience with SANs• Expectations for this course




Course prerequisites

• Students attending this course should have the following knowledge:

• HP-UX or Windows administration• SCSI, RAID, Fibre Channel and Backup technologies• Storage Area Network architecture and function

• Students attending this course should have attended the following courses or have equivalent field experience:

• SAN Fundamentals WBT




Course objectives

• After completing this course, students should be able to:• Connect nodes to the SAN infrastructure and recognize

implications of arbitrated loop and switched fabric topologies.• Interpret Fibre Channel addressing for HP-UX and Windows

operating systems.• Perform installation and configuration tasks on a B-series switch.• Determine the switch status using LEDs, CLI and GUI.• Configure and implement zoning in a SAN.• Configure switches to operate in a multi-switch fabric.• Perform a switch firmware upgrade.• Determine if a switch is functional and devices are logged in.




Course outline (1 of 4)

• Lecture

– Module 1 — Introduction to SANs

– Module 2 — Fibre Channel Technologies

– Module 3 — San Topologies

– Module 4 — Brocade Fibre Channel Addressing

– Module 5 — Configuring a Brocade Switch

• Labs

– Lab 1 — Getting Started

– Lab 2 — Preparing the Switch

– Lab 3 — Configuring Ports





• Lecture

– Module 6 — Brocade Simple Name Server

– Module 7 — Brocade Zoning

– Module 8 — Cascading Brocade Switches

• Labs

– Lab 4 — Zoning

– Lab 5 — Cascading

– Lab 6 — Merging and Splitting B-Series Fabrics

– Lab 7 — Trunking

– Lab 8 — Dynamic load sharing and routing





• Lecture

– Module 9 — Storage Presentation to Operating Systems

– Module 10 — Secure Path

• Labs

– Lab 9 — Preparing for Heterogeneous SAN

– Lab 10A — HP-UX and HSG80 SAN installation

– Lab 10B — HP-UX and VA SAN Installation

– Lab 11 — Windows 2000 Secure Path 4.0c Installation

– Lab 12 — HP-UX Secure Path V3.0D installation





• Lecture

– Module 11 — SAN Troubleshooting

– Module 12 — Fibre Channel Switches

• Labs

– Lab 13 — HP Web Tools




Additional resources — weblinks(1 of 2)• For the latest information, documentation, and firmware

releases see the HP StorageWorks website.1. Enter the URL: http://www.hp.com/country/us/eng/prodserv/storage.html

2. Under networked storage, click SAN infrastructure.3. Select the product.

• For warranty information:1. Go to the HP website at:

http://www.hp.com.2. Click Support & Drivers > See support and

troubleshooting information.3. Type the appropriate product name in the field.4. Click the appropriate listing.




Additional resources — weblinks(2 of 2)• HP Support & Drivers website

http://welcome.hp.com/country/us/en/support.html

• Cybraryhttp://cybrary.inet.cpqcorp.net/

• Single Point of Connectivity Knowledge (SPOCK)http://turbo.rose.hp.com/spock/index#Supported_Configurations



Introduction to SANs Introduction to SANs

Rev. 4.21 1



Module 1Introduction to SANs


Rev. 4.21 2

2[Rev. # or date] HP Restricted

Objectives

After completing this module, the student will be able to:• Define the Storage Area Network (SAN)• Classify the SAN components in categories• List the benefits of a SAN


Rev. 4.21 3


What is a Storage Area Network (SAN)?

““SAN is a high speed network that allowsSAN is a high speed network that allows

heterogeneous servers access to a common orheterogeneous servers access to a common or

shared pool of heterogeneous storage devicesshared pool of heterogeneous storage devices..””

---- HP Storage UniversityHP Storage University


Rev. 4.21 4


Introduction to SAN Components

• Servers with HBA• Storage systems

– RAID– JBOD– Tape– Optical

• SAN Infrastructure– Hubs – Switches– Bridges and

Repeaters• SAN Management

Software


Rev. 4.21 5


What Makes the SAN Attractive?

• Storage Consolidation and Expansion• Consolidating storage for easy management• Scaling storage to meet business needs

• High Performance Backup• Multiple servers using SAN to backup to a Shared backup

resource• LAN-free backup• Serverless (active-fabric) backup

• High Availability• Clustering for fault tolerance• Remote real-time mirroring of data


Rev. 4.21 6


Storage Consolidation and Expansion

SCSI SCSI

Distributed Storage


Rev. 4.21 7


High Performance Backup

Backup ServerBackup Server

Backup LibraryBackup LibraryDisk SystemDisk System

ServerServer ServerServer


Rev. 4.21 8


Backup LibraryBackup LibraryDisk SystemDisk System

Serverless Backup

ServerServer ServerServer


Rev. 4.21 9


Fault Tolerance/High Availability

Server AServer A Server BServer B

Switch ASwitch A Switch BSwitch B

Disk Array ADisk Array A Disk Array BDisk Array B


Rev. 4.21 10


Direct Attached Storage

DAS

Direct Attached Storage (DAS):Direst Attached Storage (DAS) is the traditional method of locally attaching storage to servers via a dedicated communication channel between the server and storage. DAS is commonly implemented as a SCSI connection, but other methods may also be used. DAS storage may be disk drives, a RAID subsystem, or another storage device. The server typically communicates with the storage subsystem using a block-level interface. As shown in the above figure, the file system resides on the server and determines which data blocks are needed from storage device to complete the file request from application.


Rev. 4.21 11


Network Attached Storage

Windows clients UNIX clients

Applicationserver

Applicationserver

NAS appliance


Rev. 4.21 12


Storage Area Networks

Storage Area Network (SAN):A SAN is a dedicated storage network designed specifically to connect storage, backup devices and servers. Storage area networking is predicated on the replacement of parallel SCSI transport with networked storage and tape behind the server.SANs represents, after WANs (Wide Area Network) and LANs (Local Area Network) a third type of network isolated from the messaging network and optimized for movement of data from server to disk and tape.Like NAS, SANs rely on networking concepts and components to move files to and from disk. Unlike NAS devices, SANs are built on separate network topology and, for the most part, do not rely on LAN protocol. That is to say, the LAN traffic and SAN traffic are isolated from each other and do not share the same bandwidth and resources. This concept is illustrated in the figure above.


Rev. 4.21 13


DAS, NAS, and SAN

Clients

NASSAN

DAS


Rev. 4.21 14

Learning check

Learning Check1. What are some of the benefits of a SAN?

…………………………………………………………………………………………….…………………………………………………………………………………………….

2. What components make up a SAN?…………………………………………………………………………………………….…………………………………………………………………………………………….

3. What is the difference between a SAN, NAS, and DAS network?…………………………………………………………………………………………….…………………………………………………………………………………………….


Rev. 4.21 15

Introductions to SANs Fibre Channel Technologies

Rev. 4.21 1


Fibre Channel Technologies



Rev. 4.21 2


Objectives

This unit prepares students to:• Understand Fibre Channel Protocol Architecture• Understand Fibre Channel Layers

• Note: Please refer to HP SAN Design Reference Guide and SAN Extensions Reference Guide for qualified solutions at:

http://h18006.www1.hp.com/storage/saninfrastructure.htmlSelect HP SAN Design Guide


Rev. 4.21 3


Fibre Channel

Fibre Channel is an industry-standard interconnect and high-performance serial I/O protocol that delivers a high level of reliability, throughput, and distance flexibility for the server industryThe levels of the Fibre Channel standard define:

Physical mediaTransmission ratesEncoding schemeFraming protocol and flow controlCommon serviceUpper-level application interfaces


Rev. 4.21 4


FC Protocol Architecture

IPIIPIIPI SCSISCSISCSI HIPPIHIPPIHIPPI SBCCSSBCCSSBCCS 802.2802.2802.2 IPIPIP ATMATMATMFC-4FCFC--44

Common ServicesCommon ServicesCommon ServicesFC-3FCFC--33

Framing Protocol / Flow ControlFraming Protocol / Flow ControlFraming Protocol / Flow ControlFC-2FCFC--22

Encode / DecodeEncode / DecodeEncode / DecodeFC-1FCFC--11

FC-0FCFC--00 133 Mbits/s133 133

Mbits/sMbits/s

NetworksNetworksChannelsChannelsN

odeN

odeP

ortP

ort

266 Mbits/s266 266

Mbits/sMbits/s531

Mbits/s531 531

Mbits/sMbits/s1063

Mbits/s1063 1063

Mbits/sMbits/s2125

Mbits/s2125 2125

Mbits/sMbits/s4250

Mbits/s4250 4250

Mbits/sMbits/s

FC Protocol ArchitectureIt is often easier to understand a communications protocol when it is first broken down into parts or layers. Fibre Channel is structured as a set of hierarchical functions: FC-0, FC-1, FC-2, FC-3, and FC-4. •FC-0 Physical Layer, includes definition of connectors, cables, and electrical characteristics of transition.•FC-1 Encoding Layer, defines the encoding/decoding and transmission protocol. •FC-2 Framing Protocol, determines how the data from the upper level will be framed for handling by the transport layer. It incorporates the management of frames, flow control and CRC.•FC-3 Common Services, is open for future implementation. •FC-4 Protocol Mapping, establishes the interface between F/C and the upper layer protocols. This function is usually provided by the vendors device driver.


Rev. 4.21 5


FC-0 Level — Physical

Lowest in the Fibre Channel protocol stackDefines the physical portions of Fibre Channel:

Media typeConnectorsTransceiversElectrical and optical characteristics needed to connect ports

Includes the data rate executed over the cables

FC-0 LayerThe lowest level defines the physical link in the system, including the fibre, connectors, optical and electrical parameters for a variety of data rates.


Rev. 4.21 6


FC-1 Level — Encode/Decode


Rev. 4.21 7


8B/10B Transmission Character

8 data bits2 parity bitsSpecial CharacterRunning Disparity

Encoding and decoding has to do with the manner that the data is placed on the wire so that the receiver will be able to interpret it. Fibre channel’s inherent reliability has much to do with the 8bit/10bit encoding scheme that reduces the bit error rate to somewhere in the range of 10-9 or about 1 bit error every trillion bits.

Fibre channel encapsultates 8 data bits into 10 bit transmission characters. This scheme provides an extra parity bit per character over the traditional 8-bit parity used in ethernet. Additional reliability is added by creating a balanced line of equal numbers of ones and zeros on the media. 10-bit characters can have positive, negative or neutral disparity. Positive disparity has more zeros than ones. Negative disparity has more ones than zeros. Neutrally disparate characters have an equal number of ones and zeros. Neutral disparity prevents an electrical shift caused by an imbalance of too many ones or zeros. Because not all data characters have equal numbers of ones and zeros, special characters are used to maintain a running disparity.


Rev. 4.21 8


Transmission Word

4 10-bit CharactersData Word• First transmission character is an encoded data byte

Ordered Set• First transmission character is the K28.5 special character• Frame Delimiter (SOF and EOF)

– SOF used for LIP• Primitive Signal (e.g., IDL, ARB, OPN(x), CLS, MRK(x)

• Primitive Sequence (e.g., LIP, LPB, LPE)– set of 3 ordered sets for link control

K28.5 D21.5 D22.2 D22.2

Transmission WordGroups of four characters (40-bits) form a transmission word. Words can be either data, where the first character is an encoded data byte, or an ordered set, where the fourth character (first byte) is the K28.5 special character used to provide character synchronization. Ordered sets permit control functions to be embedded in the bit stream.

Ordered SetAn ordered set is a transmission word beginning with a special character, K28.5. An ordered set can be a •Frame delimiter•Primitive signal, or•Primitive sequence

A frame delimiter defines what class of service is required. The frame contains a start-of-frame (SOF) and an end-of-frame (EOF) delimiter.


Rev. 4.21 9


8B/10B Transmission Character

8 data bits2 parity bitsSpecial CharacterRunning Disparity

Encoding and decoding has to do with the manner that the data is placed on the wire so that the receiver will be able to interpret it. Fibre channel’s inherent reliability has much to do with the 8bit/10bit encoding scheme that reduces the bit error rate to somewhere in the range of 10-9 or about 1 bit error every trillion bits.

Fibre channel encapsultates 8 data bits into 10 bit transmission characters. This scheme provides an extra parity bit per character over the traditional 8-bit parity used in ethernet. Additional reliability is added by creating a balanced line of equal numbers of ones and zeros on the media. 10-bit characters can have positive, negative or neutral disparity. Positive disparity has more zeros than ones. Negative disparity has more ones than zeros. Neutrally disparate characters have an equal number of ones and zeros. Neutral disparity prevents an electrical shift caused by an imbalance of too many ones or zeros. Because not all data characters have equal numbers of ones and zeros, special characters are used to maintain a running disparity.


Rev. 4.21 10


Primitive Signals

An ordered set with special meaningFill Words• IDLE - Maintain link synchronization• ARB

Non-Fill Words• R_RDY• VC_RDY• OPN• CLS

A Primitive Signal is an Ordered Set designated by this standard to have special meaning. They indicate events at the sending port. Primitive Signals are used to indicate events or actions and are normally transmitted once. Fill Word (Idle, ARB(X), ARB(F0), ARB(FF)– transmitted on a link whenever a port is operational and has no other specific information to send Non-Fill Word (R_RDY, VC_RDY, CLS, OPN, DHD, MRK, SYNx,y,z)- signal the events.

All FC_Ports recognize R_RDY and IDLE Primitive Signals amongst others.

IdleIdle is a Primitive Signal transmitted on the link to indicate that link initialization is complete and to maintain link synchronization. Idles are transmitted on the link during periods of time when frames, other Primitive Signals or Primitive Sequences are not required to be transmitted.


Rev. 4.21 11


Transmission Hierarchy

8B/10B Transmission Character Transmission Word Frame Sequence Exchange

Exchange

Sequence Sequence

Frame2148 bytes

10-bit Character 10-bit Character 10-bit Character 10-bit Character

Frame2148 bytes

Transmission Word40-bits

Transmission Word40-bits

FC-1

FC-2


Rev. 4.21 12


Rev. 4.21 13


FC-2 Framing and Flow Control

Frame format Sequence management Exchange management Flow Control

ExchangeSequence Sequence Sequence

FrameFrameFrameFrameFrameFrame

Frame FormattingAll information transferred in fibre channel is packaged in a data structure called a frame. A frame contains the information to be transmitted (payload), link control information and the addresses of the initiator and destination ports. Frame formatting is the functionality of FC-2 layer by which it breaks the data to be transmitted into pre-determined frame size and reassemble them at the receiving port.

Sequence ManagementA sequence is a set of one or more related frames transmitted unidirectionally from one port to another port. The port that initiates the sequence is called the sequence initiator. It follows that t he port that receives the sequence is called the sequencerecipient. Each frame within a sequence is uniquely identified by a sequence count. Management of sequence by FC-2 includes sequence identification, maintaining of sequence count, streaming the sequences and sequence completion.

Exchange Management


Rev. 4.21 14


Frame Structure

6 idles

>2 idles

Link Control Frame• provides flow control via ACK for class 1 and 2 frames• contains no payload

Data Frame• contains payload of 0-2112 bytes

A frame is the basic unit of data transfer and can be up to 2112 bytes.

The Start of Frame (SOF) and End of Frame (EOF) are special FCtransmission words that act as frame delimiters. The CRC is 4 octetslong and uses the same 32-bit polynomial used in FDDI.

The FC Header is 24 octets long and contains several fieldsassociated with the identification and control of the Data Field.

The Data Field is of variable size, ranging from 0 to 2112 octets,and includes the user data in the Frame Payload field, and OptionalHeaders. The currently defined Optional Headers are:- ESP_Header;- Network_Header;- Association_Header;- Device_Header.

The value of the SOF field determines the FC Class of serviceassociated with the frame. Five Classes of service are specified in[FC-FS]. They are distinguished primarily by the method of flowcontrol between the communicating Nx_Ports and by the level of dataintegrity provided. A given Fabric or Nx_Port may support one or moreof the following Classes of service:- Class 1: Dedicated physical connection with delivery confirmation;- Class 2: Frame multiplexed service with delivery confirmation;- Class 3: Datagram service;- Class 4: Fractional bandwidth;- Class 6: Reliable multicast via dedicated connections.


Rev. 4.21 15


Frame Header

The frame header is 24 bytes long and is present in all frames. It is used to control link operation, control device protocol transfers, and to detect missing frames or frames that are out of order.

DF_CTLData Field Control specifies the presence of optional headers inthe payload of the frame.

D_IDThe 24-bit N_port address to which the frame is being sent. Destination ID.

F_CTL24-bit field contains control information relating to the frame content.

OFFSET


Rev. 4.21 16


Sequence

A group of related frames transmitted in the same direction between two N_PortsPrimitive Sequences initiate or indicate port statesSequence initiativeSegmentation and reassemblySequence Management

Sequences are made up of one or more frames. Several non-concurrent signals make up an exchange. You can think of an exchange as a conversation made up of sentences (sequences) spoken over a CB radio, where each party has to let the other party know when to speak. For example, SCSI-3 FCP uses bidirectional exchanges with information passing in one direction at a time. To send data in the opposite direction, a sequence initiative is passed from one port to another and back again. Each port generates one or more sequences within the exchange.

An application level payload such as IPv6 is called Information Unit at the FC-4 level of Fibre Channel. Each FC-4 Information Unit is mapped to an FC Sequence by the FC-2 level. An FC Sequence consists of one or more FC frames related by the value of the Sequence_ID (SEQ_ID) field of the FC Header.

The maximum data that may be carried by an FC frame is 2112 octets. The maximum usable frame size depends on the Fabric and Nx_Port implementations and is negotiated during the Login process. Whenever an Information Unit to be transmitted


Rev. 4.21 17


Exchange

Asynchronous• Separate exchanges in each direction• Sequence Initiative held by the Common Transport

Synchronous• Single, bi-directional exchange• Sequence Initiative transferred at the end of each

transmission

Exchange ManagementAn Exchange is a set of one or more sequences. Sequences for the same exchange may flow in the same direction or in the opposite direction one sequence at a time. For an exchange to occur, the initiator initiates the exchange by sending an OX-ID (originator exchange id). The recipient on receipt replies with RX-ID (Responder exchange id). Both OX-ID and RX-ID are used to identify the exchange. Management of the exchange is done by maintaining and monitoring the Ebbs (Exchange Status Blocks) at the originator and the responder. An ESB is a logicalconstruct representing the format of the exchange status information. It is used to track the progress of an exchange on a sequence by sequence basis.

The Originator of an Exchange initiates the first Sequence as the Sequence Initiator. If the Sequence Initiative bit (bit 16) is set to zero, the Sequence Initiator holds the initiative to continue transmitting Sequences for the duration of this Sequence Initiative.

The Sequence Recipient gains the initiative to transmit a new


Rev. 4.21 18


Exchange/Sequence Control - F_CTL

Control FieldWord 2,

Bits Description0 = Originator of Exchange

Exchange Context 23 1 = Responder of Exchange0 = Sequence Initiator

Sequence Context 22 1 = Sequence Receipient

0 = Sequence other than first of Exchange

First_Sequence 211 = First Sequence of Exchange0 = Sequence other than last of exchange

Last_Sequence 20 1 = Responder of Exchange


Rev. 4.21 19


Flow Control

Buffer to Buffer• Flow control for two ports that share a link• Uses R_RDY primitive signal for flow control

End-to-End• Flow control for source and destination N_Ports• Uses ACK link control frame for flow control

Credit can be asymetrical

Flow ControlThe concept of flow control deals with the problem where a device receives frames faster than it can process them. When this happens, the result is that the device is forced to drop some of the frames. Fibre Channel has a built-in flow control solution to this problem.

The idea is simple enough. A device can transmit frames to another device only when the other device is ready to accept them. Before the devices can send data to each other, they must login to each other. One of the things accomplished in login is establishing credit. Credit refers to the number of frames a device can receive at a time. This value is exchanged with another device during login, so each knows how many frames the other can receive. After enough frames have been transmitted and credit runs out, no more frames can be transmitted until the destination device indicates it has processed one or more frames and is ready to receive new ones. Thus, no device should ever be overrun with frames. Fibre Channel uses two types of flow control, buffer-to-buffer and end-to-end.


Rev. 4.21 20


Buffer-to-Buffer

BB_CreditThe Login credit which represents the number of frames that may be transmitted before receiving an R_RDY.Frames are sent from one buffer to the other using R_RDY primitive signalsFrame flow is always from the source to the destination bufferMultiple intermediate buffers may be in between source and destination


Rev. 4.21 21


End-to-End

EE_CreditThe number of receive buffers allocated by a recipient port to an originating port. Used by Class 1 and 2 services to manage the exchange of frames across the fabric between source and destination.


Rev. 4.21 22


FC-2 Framing and Flow Control

Transmit Buffer

8-bitByte

8-bitByte

8-bitByte

8-bitByte

8 bit character

Serializer

10-bit character

8b/10bEncoder

Deserializer

Serial Transfer

8b/10bDecoder10-bit

character

Receive Buffer

8-bitByte

• Buffer-to-Buffer Credit • End-to-End Credit

• ACK

• R_RDY

Frame

8-bitByte

8-bitByte

8-bitByte

FC-2 defines the structure and organization of the information being delivered and how that delivery is controlled and managed. Exchange management is the mechanism that two fibre channel ports use to identify and assign an exchange ID number for a set of related information units. When the entire stream of data will fit in a single frame (2112 bytes) a single exchange id is created and a sequence number is assigned.However, when a stream of data will not fit into a single frame (2112 bytes), data is put into sequences of frames. Within the exchange ID sequence management is used to number the sequence segments in the stream of data. Sequence numbers associated with the exchange will be used at the recipient to re-order the sequence segments, to re-assemble as a contiguous stream of data. In other protocols, this is commonly known as fragmentation and re-assembly. Frame Structure has a start-of-frame delimiter ordered set and ends with an end-of-frame delimiter set.

Flow control is the process to deliver a frame. When a frame is ready for transmission, it is sent thru the encoder (8b/10b), to the serializer(sfp/gbic) and transmitted to the receiver port where it is deserialized, decoded and stored in a receive buffer.

The receiving port sends to the transmitting port a credit to send another frame and decrements a credit from the credit value established during the login session (buffer to buffer credit). When the receiving port moves the buffer to the next port, the debit is restored. Buffer credits regulate the flow of frames into and out of the fabric


Rev. 4.21 23


Flow Control

ACK AcknowledgementR_Ready Receiver ReadyFlow Control is related to Class of Service• Class of Service 3 uses R_RDY (buffer to buffer or

BB_Credit) flow control, each R_RDY received increments BB_Credit value.

• Class of Service 2 and F use R_RDY and ACK (end-to-end or EE_Credit) flow control, each ACK received increments EE_Credit value.


Rev. 4.21 24


Fibre Channel Frame

4 bytesStart Of Frame

4 bytesStart Of Frame

24 bytesFrame Header

24 bytesFrame Header

64 bytesOptional Header

64 bytesOptional Header

2112 bytes Data Field2112 bytes Data Field

2048 bytes Payload2048 bytes Payload

4 bytesCRCError Check

4 bytesCRCError Check

4 bytesEnd Of Frame

4 bytesEnd Of Frame

CTLCTL Source AddressSource Address

Destination Address

Destination Address TypeType Seq_CntSeq_Cnt Seq_IDSeq_ID Exchange_

IDExchange_

ID

Fibre Channel Frame

In the header of the frame contains:CTL-Defines what type of frame this is, Data or Control.Source Address- where the frame is starting from (PID)Destination Address-the destination (PID) of the frame.Type-FC-4 Types: Most common are SCSI (8) and IP (5)Seq_Cnt-A Each Frame within a sequence is uniquely numbered with a Sequence Count. Seq_ID-Each Sequence has it’s own unique identifier.Exchange_ID An Exchange is composed of one or more nonconcurrentsequences for a single operation.


Rev. 4.21 25


FC-3 and FC-4

FC-3 Level — Common ServicesIntended to provide common services and is currently under development because not all requirements have been clearly defined.FC-4 Level — MappingDefines the mapping of protocols between the lower levels of Fibre Channel and the command sets that use Fibre Channel. Separate standards for:

SCSI-3IPIntelligent Peripherals Interconnect-3 (IPI-3)High Performance Parallel Interface (HIPPI)Fiber distributed data interface (FDDI)

FC-3 and FC 4 LayersFC-3 Layer: The FC-3 level of the FC standard is intended to provide the common services required for advanced features such as:

• Striping -To multiply bandwidth using multiple N_ports in parallel to transmit a single information unit across multiple links.• Hunt groups - The ability for more than one Port to respond to the same alias address. This improves efficiency by decreasing the chance of reaching a busy N_Port.• Multicast - Multicast delivers a single transmission to multiple destination ports. This includes sending to all N_Ports on a Fabric (broadcast) or to only a subset of the N_Ports on a Fabric.

FC-4 Layer: The highest level in the FC structure defines the application interfaces that can execute over Fibre Channel. It specifies the mapping rules of upper layer protocols using the FC levels below. Fibre Channel is equally adept at transporting both network and channel information and allows both protocol types to be concurrently transported over the same physical interface. The following network and channel protocols are currently specified or proposed as FC-4 Layer:

•Small Computer System Interface (SCSI) •Intelligent Peripheral Interface (IPI) •High Performance Parallel Interface (HIPPI) Framing Protocol •Internet Protocol (IP) •ATM Adaptation Layer for computer data (ATM-AAL5) •Link Encapsulation (FC-LE) •Single Byte Command Code Set Mapping (SBCCS) •IEEE 802.2


Rev. 4.21 26


Fibre Channel cabling

Fibre Channel uses two types of cabling:Copper

Optical

FC Cables and TransceiversMedia: Fibre Channel signals can run over both copper and glass fiber media. Even thoughlonger distances can be achieved with glass fiber, it is more expensive. For copper, the following cable types are used: video cable, miniature cable, and shielded twisted pair. The most common by far is shielded twisted pair, using a DB-9 connector. For glass fiber, the choices are: 62.5 micron multi-mode, 50 micron multi-mode, and 9 micron single-mode. Long wave, short wave, and Very Long Distance transceivers can be used. Speed: Fibre Channel offers a very wide range of media speeds.


Rev. 4.21 27


Transceivers and cables

GBIC SFP

SC Connector LC Connector


Rev. 4.21 28


Single-Mode Fiber

• One coherent stream of light to travel single path• Longwave lasers• Single-mode, step-index fiber

Single-Mode

Cladding125

MicronCore

9 Micron Diameter


Rev. 4.21 29


Single-Mode Step-Index Fiber

Step-index fiber does not align the light rays along the axis of the core. Typically, the modal dispersion of a step-index fiber is high; however, with a single-mode fiber and a single mode of light traveling in the core, step-index fiber provides adequate conductive characteristics.Single-mode fiber has the highest bandwidth and lowest loss performance. The core is so small that only a single mode of light can enter it. Therefore, the chromatic and modal dispersion are greatly reduced or eliminated. The information-carrying capabilities of the single-mode fiber are infinite. Single-mode fiber supports speeds of tens of gigabits per second and can carry many gigabit channels simultaneously. Each channel carries a different wavelength of light without any interference. Single-mode fiber is the preferred medium for long-distance telecommunications. It is also useful in networks for inter-building runs and high-speed backbones. Applications for single-mode fiber to the desk are not anticipated.


Rev. 4.21 30


Multimode Fiber

• Multiple streams of light to travel different paths • Most popular for networking

– 50/125– 62.5/125

MultimodeCladding

125Micron

Core

50/62.5 Micron Diameter


Rev. 4.21 31


Multimode Step-Index Fiber

Step-index fiber has a high modal dispersion. The core material (n1) is not graded, it does not focus the light beam to follow the core’s axis (axial transmission). The different light rays leaving the source simultaneously reach the destination at significantly different times, reducing the bandwidth of the fiber and its distance.Step-index fiber:• Is inexpensive• Decreases bandwidth and distance• Has a higher modal dispersion than graded-index fiber• Is seldom used in networking and data communications


Rev. 4.21 32


Multimode Graded-Index Fiber

Grading of the fiber-optic core focuses the light beam closer to the axis of the core. The light rays travel closer to the axis, which reduces travel distance and synchronizes arrival rates.Grading is achieved by varying the chemical composition of the core material (n1).Graded-index fiber:• Is more expensive than step-index fiber. However, manufacturing advances and wide adoption of fiber-optic cabling have significantly reduced its manufacturing costs.• Increases bandwidth and distance.• Has a lower modal dispersion than step-index fiber because it provides more accurate signal transmission.• Is frequently used in networking and data communications (nearly all multimode fibers used in networking have a graded index).


Rev. 4.21 33


Attenuation

Loss of power as a signal travels over a distanceSpecified in decibels per kilometer (dB/km)Is lessened with higher-quality, more expensive,

single mode fibersIs greater with lower quality, less expensive,

multimode fibers

Attentuation can result from:Light absorption caused by material impuritiesLight scattering caused by material impurities or by the defects at the core/cladding

interface, and by the scattering of the molecules of the medium (silica)Macro bends (cable bends beyond the specified radius)Micro bends (cable wrapping or squeezing)Scattering and reflection at cable splices.


Rev. 4.21 34


Dispersion

Dispersion is the degree of scattering of the light beam as the light beam travels along the fiber-optic cable. Types of dispersion:

Scattering — Loss of a light signal caused by microscopic impurities of the materialChromatic dispersion — Loss of light signal caused by different wavelengths traveling at different speedsModal dispersion — Loss of light signal caused by different light rays traveling different path lengths within the fiber


Rev. 4.21 35


Macro Bending

25mm

Macrobending is the physical bending of the fiber cable past the specified radius (25 mm). As the fiber exceeds the specified radius, the light loses some particles and attenuation increases.


Rev. 4.21 36


Micro Bending

Microbend LossMinimum bend radius is the radius of an optical fiber or fiber-optic cable that should not be bent. The minimum bend radius is of particular importance in the handling of fiber-optic cables. It will vary with different cable designs. The manufacturer should specify the minimum radius to which the cable may safely be bent during installation, and for the long termMinimum bend radius for a fiber channel cable should be at least 3 centimeters. If the cabling is bent beyond 3 cm., data loss or corruption is likely to occur. The cabling can crack or break. When the light encounters a break or microbend it scatters.


Rev. 4.21 37


Fibre Channel extension

Reasons for extension:Business continuance in case of site failure

Disaster toleranceGeographically distributed multi-site SAN support

Inter-site file sharingRemote backupCentralized servers or storage

Currently, HP supports the following technologies for Fibre Channel ISL SAN extension:

Fibre Channel Long Distance TechnologiesLong Wave Transceivers

Wavelength Division Multiplexing (WDM)

TCP/IP Data Protocol TechnologiesFibre Channel over Internet Protocol (FCIP)


Rev. 4.21 38


Long Wave transceivers

Long wave GBIC or SFP transceivers are required to go beyond the 500 meter limit for 1 Gbps and the 300 meter limit for 2 Gbps links respectivelyThere are long-wave optical transceivers that are capable of transmitting up to 100 kilometers.Currently HP supports the following long wave transceivers:

10 kilometer GBIC (1GB)100 kilometer GBIC (1GB)10 kilometer SFP (2GB)35 kilometer SFP (2GB)

Long wave transceivers are supported on both M-Series and B-Series product lines. C-Series currently supports only 10 km SFPs.


Rev. 4.21 39


Wavelength division multiplexing

WDM works by combining and transmitting multiple signals simultaneously at different wavelengths on the same fiber. One fiber transports multiple data streams. Key advantage to WDM is that it is protocol- and bit-rate-independent. WDM-based networks can transmit data using the following protocols:

IPATMSONET /SDHEthernet

Bit rates between 100 Mb/s and 2.5 Gb/sWavelength Division Multiplexing devices can be used to extend the distance between two Fibre Channel switchesDevices are transparent to the switches themselves and do not count as an additional hopWavelength Division Multiplexing is supported for both 1 Gbps and 2 Gbps.


Rev. 4.21 40


Fibre Channel over Internet Protocol (FCIP)

Protocol that encapsulates Fibre Channel frames into IP packets and tunnels them through an existing IP network infrastructure to transparently connect two or more SAN fabrics together

FCIP Gateways perform Fibre Channel encapsulation process into IP Packets and reverse that process at the other endFC Switches connect to the FCIP gateways through an E_Port for SAN fabric extension to remote locationsA tunnel connection is set up through the existing IP network routers and switches across LAN/WAN/MAN


Rev. 4.21 41

Learning check

Learning Check1. What is Microbending?…………………………………………………………………………………………….…………………………………………………………………………………………….2. What Fibre Channel layer is responsible for encoding and decoding?…………………………………………………………………………………………….…………………………………………………………………………………………….3. What is the difference between Single-mode and Multimode fiber?…………………………………………………………………………………………….…………………………………………………………………………………………….


Rev. 4.21 42

Introduction to SANs SAN Topologies

Rev. 4.21 1


SAN Topologies

Module 3

HP Restricted


Rev. 4.21 2


Objectives

This unit prepares students to:• Describe various SAN topologies


Rev. 4.21 3


Fibre Channel topologies

Point-to-pointArbitrated LoopSwitched Fabric

Arbitrated Loop

Point-to-Point

Switched Fabric


Rev. 4.21 4


Point-to-point

Is inexpensiveUses full bandwidthConnects two devices only


Rev. 4.21 5


FC-AL

Node BNode BNode ANode A

NL_

Port

0N

L_Po

rt 0

NL_

Port

0

TransmitterTransmitterTransmitter

ReceiverReceiverReceiver

NL_Port 1

NL_Port 1

NL_Port 1ReceiverReceiverReceiver

Node CNode CNode DNode D

NL_

Port

3N

L_Po

rt 3

NL_

Port

3



NL_Port 2

NL_Port 2

NL_Port 2



FC HUB



Rev. 4.21 6


Arbitrated loop hubs

Provide a physical star topology for a loop configurationCompletes connections between transmitters and receivers on a port-by-port basis through mux circuitryFinishes the loop by connecting the transmitter of the last hub port to the receiver of the first Has bypass circuitry at each port – allows the loop to circumvent a disabled or disconnected node

Arbitrated Loop with a hubLike most ring topologies Fibre Channel has hubs as well. Life is made easier when the devices can be connected to a central hub or concentrator. The cabling is easier to deal with, and the hub can usually determine when a device is inserted or removed. Thus, a "bad" device or broken fiber won't keep the whole network down.


Rev. 4.21 7


World-Wide Name (WWN)

A 64-bit unique identifier for each Fibre Channel nodeNot used for routing traffic across networkUsed to preserve identity of a node if its layer 2 or layer 3 address is changed

Identifier formats based on IEEE registration can be found at:http://standards.ieee.org/regauth/oui/tutorials/fibreformat.html

World Wide NamesIn order for Fibre Channel frames to be delivered from source to a destination, there is a need for an addressing scheme.Arbitrated Loops use an 8-bit address, called an AL-PA (Arbitrated Loop Physical Address). Since these are not consecutive numbers, an index or Loop ID is assigned to each AL-PA. The Loop ID is a consecutive number between 0 and 125.Switched Fabrics use a 24-bit address called a Fibre Channel Address. In the switch this address is called the Port ID (PID). These addresses are automatically assigned in the Fabric. The Fibre Channel Address (PID or AL-PA) could be thought of as an “IP address” from the networking world.Each HBA or Fibre Channel interface has a fixed 64-bit World Wide Name (WWN) assigned by the manufacturer, which is regulated by IEEE.


Rev. 4.21 8


WWN of HSG – 80 Controller

WWN identifiers

50 00 - 1F E1 – 00 0B - 00 08

FC-PH WWN format identifier

IEEE Company Identifier.0001FE = Digital Equipment Corporation.

Vendor-specified Identifier


Rev. 4.21 9


Loop initialization

The AL_PA is assigned during the loop initializationA port on the loop assumes the role of loop master to manage thegeneration of a positional map of the loopIf an FL_port is present on the loop, it becomes the temporary loop master; otherwise, an NL_port assumes the roleThe loop master generates and transmits a 128-bit map used to assign the loop addresses

This map is transmitted in a frame from loop port to loop port around the loopEach loop port examines the map to view the unassigned addresses. When an unassigned address is present, the loop port fills in the map to designate that the address has been assigned and forwards the map to the next loop port.

After the AL_PA bit map has returned to the loop master, it contains a complete map of all the ports on the loopThe loop master then transmits the completed frame so that all the ports on the loop can capture the completed positional map.


Rev. 4.21 10


Private loop

A private loop can accommodate up to 126 NL_portsA private loop contains only devices which are not fabric awarePrivate loop devices cannot communicate with fabric devices

HUB


Rev. 4.21 11


Public loop

A public loop can accommodate up to 126 NL_ports and one FL_port The FL_port extends the number of nodes for communication.

HUBHUB

A Public Loop device or node is capable of logging into the fabric and can communicate with other devices in the fabric.Note: Fibre Channel Public Loop devices must communicate through the fabric. Switches cannot connect to one another through the loop. Each public loop device must connect to the fabric through only one FL_port.


Rev. 4.21 12


Port logins - PLOGI

NL_Port login or PLOGIPart of a set of extended link services used to exchange communication parameters and identify devices on the SANPerformed immediately after loop initializationA server attempts to login to all devices on the loop by issuing PLOGI frames addressed to each possible AL_PA addressTargets that accept the PLOGI will return an ACC frame that includes the device WWN, buffer capability, frame size support…


Rev. 4.21 13


Switched Fabric topology

Node BNode B

Node ANode A

N_P

ort 0

N_P

ort 0 TransmitterTransmitter

ReceiverReceiver

F_PortF_Port

ReceiverReceiver

TransmitterTransmitter

F_Po

rtF_

Port


ReceiverReceiver

N_Port 1

N_Port 1


ReceiverReceiver

Node BNode B

Node ANode A

NL_

Port

0N

L_Po

rt 0


ReceiverReceiver

NL_Port 1

NL_Port 1

ReceiverReceiver

Node DNode D

NL_

Port

3N

L_Po

rt 3


ReceiverReceiver

FC HUB


FL_Port FL_Port


ReceiverReceiver

FC_AL

NL_Ports

Fabric

A network of switches in a Fibre Channel environment is referred to as a fabric. Nodes connect into this fabric to access other nodes. A wide-open architecture uses intelligent switches to connect many ports. The Fibre Channel fabric was designed as a generic interface between a node and the physical layer. By adhering to this interface, Fibre Channel nodes can communicate over the fabric with other nodes, without knowing what that node is.A fabric is often referred to as a switch topology. Frames are routed through various switches by having the fabric elements interpret the destination address identifier in a frame as it arrives at each fabric element.Ports on one node can communicate with ports on other nodes connected to the same fabric. With the fabric topology, many connections can be active at the same time. The any-to-any connection service and peer-to-peer communication service provided by a fabric is fundamental to Fibre Channel architecture.


Rev. 4.21 14


Fabric addressing- B series

SS = Switch Domain ID (Switch Number) PT = Port Type

1 = 2nd/3rd Generation Switch (SS = Switch 1 – 239)4 = 2nd Generation Compat. Mode (SS = Switch 0 – 31) – 5 bits0 = 1st Generation (SS = Switch 0 – 31) – 5 bits

P = Switch Port Number. When Core PID = 1, both PT & P comprise the port number. Ex: 26 = Port 38 (2 x 16 + 6)

If FF = 00 then port is F-Port or fabric. If FF = A non-zero value, then port is FL-Port. This is the ALPA for the port.

HP no longer sells 1st Generation switches and has replaced most of them through an exchange program when the 2cd Generation switches came out. There is the possibility there may still be some 1st Generation switches in customer sites.


Rev. 4.21 15


Simple Name Server

For an arbitrated loop, target discovery is performed by port logins to all 126 AL_PA device addressesFabrics use a name server function that resides within each switchDevice drivers may register values for:

24-bit address64-bit port wwn64-bit node wwnClass of service parametersFC-4 protocols supportedPort type

A fabric facilitates device discovery by using a Name Server service in each switch. The name server is a database of registered devices. N_PORTS (or public NL_PORT) register with the Name Server by performing a successful PLOGI to 0xFFFFFC well-known address. The device may register values for some or all ofthe database objects, 64-bit Port Name, 64-bit Node Name, class of service parameters, FC-4 protocols supported, and port type, such as N_Port or NL_Port.For example, the Name Server makes it possible for a file server to begin discovering disk targets by inquiring for a list of all port addresses registered with the switch, or only those ports addresses that reported SCSI-3 support.


Rev. 4.21 16


Translative Mode

Public LoopPublic LoopPublic LoopPublic Loop

Private NodePrivate NodePublic NodesPublic Nodes

Public Loop AddressPublic Loop Address Private Loop AddressPrivate Loop Address

Fabric Assigned AddressFabric Assigned Address

LL LL PPLL LL PPLL LL PP 00 00 PP00 00 PP00 00 PP

NN NN NNNN NN NNNN NN NN

Fabric direct attach NodesFabric direct attach Nodes

N N

NLNL

NL

FL FL

F F

Translative mode allows private loop devices to communicate with fabric-capable devices. This can be accomplished by manipulating the loop identifier of the private loop device’s port address. On a private loop segment, the loop identifier is x0000, indicating that the device is non-fabric. The fabric can proxy a loop identifier and create an entry in the SNS, which would allow fabric-capable devices to access it.Private devices use an address format of “00 00 PP”, where “PP” = the local loop address (AL_PA). This type of address is all that a Private device is capable of receiving or sending (8 bits). Therefore, the Private devices may only communicate with the devices it can see on the local loop.Fabric attached devices use an address format of “NN NN NN”, where:

“NN NN NN” is the address of any Fabric-attached device that has logged into the fabric. This type of address is used by all fabric-attached devices to communicate across the fabric (24 bits). Fabric-attached devices are either directly attached via G_Port or are loop attached via an FL_Port.Public Loop attached devices use an address format of “LL LL PP”, where:

“LL LL” is assigned by the fabric at login time“PP” is the local loop address (AL_PA).

This type of address is simply a fabric assigned address for a device attached to an FL_Port (24 bits). The value of “LL LL” is the same for all Public Loop devices attached to the same FL_Port.


Rev. 4.21 17

Learning check

Learning Check1. What is the difference between a public and private loop?

…………………………………………………………………………………………….…………………………………………………………………………………………….2. Describe the fabric login process.…………………………………………………………………………………………….…………………………………………………………………………………………….…………………………………………………………………………………………….


Rev. 4.21 18

Introduction to SANs Brocade Fibre Channel Addressing

Rev. 4.21 1


Brocade Fibre Channel Addressing

Module 4


Rev. 4.21 2


Objectives

This unit prepares students to:• Describe Brocade Native Addressing Mode• Describe Brocade CORE_PID Addressing Mode• Describe Brocade Extended Edge Addressing Mode


Rev. 4.21 3


Native Brocade Addressing Modes

The format of a 24-bit address in Native Modes

XX1YZZ

XX IS A VALUE BETWEEN 0x1 and 0xEF inclusive (Domain ID 1-239 in decimal.

The 1 means “native” mode.

Y is the port number 0x0 –0xF (0-15 decimal).

ZZ is the AL_PA for a Loop device or 00 for an F-port.

12141C = Domain 18, Port 4, AL_PA 1C

041A00 = Domain 4, Port 10, Fabric Direct Attach


Rev. 4.21 4


CORE_PID Addressing Mode

In Fabric OS v2.6.1/3.1/4.1, there were two PID formats:Native PID - Designed for switches of up to 16 ports

Core PID - Designed for switches of any port countSupported in Fabric OS v2.X, v3.X, and v4.XDefault setting in Fabric OS v3.X and v4.X

All switches in a fabric must be set to the same PID format

• The Native PID format was introduced with the SilkWorm 2000-series switches. This format supports up to 16 ports per switch.• The Core PID format was introduced to support SilkWorm switches with more than 16 ports. This format supports up to 128 ports per switch.


Rev. 4.21 5


CORE_PID Addressing Modes

The format of a 24-bit address in CORE_PID mode

XXSPZZXX IS A VALUE BETWEEN 0x1 and 0xEF inclusive (Domain ID 1-239 in decimal.

The 1 means “native” mode.

X is the logical slot number 0x0 –0x3 (0-3 decimal on 12000, 0-1 decimal on 3900)

P is the physical port in that slot 0x0 to 0xF (0-15 decimal).

ZZ is the AL_PA for a Loop device or 00 for an F-port.

12141C = Domain 18,Slot 1, Port 4, AL_PA 1C

040A00 = Domain 4, Slot 0, Port 10, Fabric Direct Attach


Rev. 4.21 6


Problem - solution

Problem: Adding high-port count switch (SilkWorm 3900/ 12000/ 24000) to a Native PID fabric can be disruptive to some servers

High-port count switches require Core PID format on all switches in the fabricOnce you change the PID format on the existing switches to Core PID, the switch port PIDs change as wellServers that use static PID binding may need to be reconfigured and rebooted – a disruptive event

Solution: Need a new PID format that is compatible with Native PID Format and supports high-port count switches

Result: Existing Native PID format switches continue to present the same PID – no host reboot required!

Since all switches in a fabric must have the same PID format, converting a Native PID format switch to Core PID format changes the switch port PIDs.

For example, the switch port PID for port 1 on domain 8 in Native PID format is 0x081100; in contrast, the switch port PID for port 1 on domain 8 in Core PID format is 0x080100 (note the difference in the third nibble). Any attached device will then change their PID as well, causing problems for those devices (particularly servers) that statically bind system parameters (SCSI ID, etc.) to the device PID.


Rev. 4.21 7


Extended Edge PID

In Fabric OS v2.6.2, 3.1.2, and v4.2, the new Extended Edge PID Format meets this requirement:• Lowest PID = 0x10 (as in Native Mode – not as in Core PID)• Extended Edge PID format is 0xXXYYZZ • XX = domain ID, • YY = port area + 0x10, wrapping at 0x7f• ZZ =AL_PA (as before)

The new Extended Edge PID format starts PID values at 0x10 – the same value as does Native PID format.The maximum YY value in the PID is still 0x7f, as in the Core PID format, so when the port area value is between 112 and 127 (slot 10 in the SilkWorm 24000), the YY values wraps to the unused values 0x00 through 0x0f.Extended Edge PID format replaces Native PID format in Fabric OS v4.2.


Rev. 4.21 8


Extended Edge advantage

The advantage of the Extended Edge PID Format: switches that change from Native PID format to Extended Edge PID format have no change in PIDs!• Easier to add high-port count switches to an existing Native PID

fabric

The real advantage for the Extended Edge PID format is: Native PID switches (like theSilkWorm 3200 in the example above) can be converted to Extended Edge PID format without changing the switch port PID. This make it easier to add high-port count switches (SilkWorm3900, 12000, and 24000) to an existing fabric of lower-port count Native PID switches.


Rev. 4.21 9


Configuring PID

Some Fabric OS license features may be affected by changes in the PID format – particularly those features that require a port area value

Secure Fabric OS - DCC PolicyAdvanced Zoning – domain, port zone member definitionsAdvanced Performance Monitor – SID/DID definitions

There are two ways to change the PID format:Use the configure command:

First, disable the switch with the switchdisable command.Run the configure command.Change the Switch PID Format value to 0 (for Native PID), 1 (for Core PID), or 2 (for Extended Edge PID).Re-enable the switch with the switchenable command.

Download a configuration file with the configdownload commandWhich approach is recommended?

The configure command automatically translates port area values to reflect the new PID format.

Do not change the PID format and the security policy, zoning database, or APM monitors at the same time – the automatic updates may not be completed correctly!

The configdownload command may not automatically translate port area values to reflect the new PID format.

If the downloaded switch configuration includes security policies, zone values, or APM monitors, those area values are not automatically translated

Solution: omit these settings from the configuration file


Rev. 4.21 10

Learning check

Learning Check1. Why is the PID format necessary?

…………………………………………………………………………………………….

…………………………………………………………………………………………….

2. Under what circumstances would you use a format other than the Native PID format?…………………………………………………………………………………………….…………………………………………………………………………………………….


Rev. 4.21 11

Introduction to SANs Configuring a Brocade Switch

Rev. 4.21 1


Configuring a Brocade Switch



Rev. 4.21 2


Objectives

This unit prepares students to:Setup COM port connection to Brocade switchDetermine if the switch has booted successfully after POSTConfigure IP addresses for switches Logon via telnet to Brocade switchUpgrade Brocade switch FirmwareReset the switch to its Factory Default settingsChange the switch configuration parametersSave and restore configurationVerify Brocade switch status using show commands


Rev. 4.21 3


Quick Install SilkWorm 3800 Switches

1.1. Insert Insert SPFsSPFs in the port slots in the port slots 2.2. Connect serial port to a PC with HyperTerminalConnect serial port to a PC with HyperTerminal3.3. Connect power cords to each installed Power supplyConnect power cords to each installed Power supply4.4. Turn on switch power; Wait for the switch to run POST and bootsTurn on switch power; Wait for the switch to run POST and boots……5.5. Configure switch IP address using HyperTerminal (Br2400 only)Configure switch IP address using HyperTerminal (Br2400 only)6.6. Connect FC cables from switch ports to hosts and devicesConnect FC cables from switch ports to hosts and devices

The switch should be now operational and accessible via TelnetThe switch should be now operational and accessible via Telnet

Port#0Port#0

Power supply#1Power supply#1 Power supply#2Power supply#2

LAN PortLAN PortSerial PortSerial Port Port#15Port#15

S A N S w itc h 2 /1 6

IP

0 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 1 51 0 1 0 1

Quick Install SilkWorm 3800 SwitchesPower supplies: Switch power connection is via two switched connectors on the back panel. A green power on indicator light is located above the power switch in the power supply module.Port Cabling: Cables can be connected pre or post power on. Switch will re-establish host connection on cable insertion.


Rev. 4.21 4


Power-On Self Test

POST runs through the following test cycles:Preliminary POST diagnosticsInitialization of operating systemInitialization of hardwareDiagnostic tests on: circuitry, port functionality, memory, parity,

statistics, counters, and serialization.

Each time the switch is powered on, rebooted, or reset, the switch automatically runs a Power-On Self Test (POST). During POST the port status LEDs flash, verifying that the switch is operating properly. POST completes in approximately six minutes, with total boot time approximately seven minutes.If the switch prompt does not display when POST completes, POST was unsuccessful. To determine whether POST completed without errors, verify that all LEDs return to a normal state after POST is complete.


Rev. 4.21 5


Brocade Silkworm 3800 Serial Port

Need A PC with:HyperTerminal UtilityAvailable COM port

9600 Baud8 bitsNo parityOne stop bitNo hardware flow control

Required Cable :HP P/N XHP-000027

SilkWorm 3800 SwitchSilkWorm 3800 Switch

Tip: Any straight-through serial cable (DB9 female to female) with only pins 2, 3 and 5


IP

0 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 1 51 0 1 0 1

Brocade Silkworm 3800 Serial PortSerial Port Connection: The switch includes a serial port used to set the IP address when setting up or reinitializing a switch. It is not used during normal operation. The settings are as follows: 8-bit, No parity, One stop bit, 9600 baud.The switch comes with this default IP configuration:

IP Address: 10.77.77.77SubnetMask: 255.255.255.0

You can access the switch over the LAN; from any computer configured with an IP address from the same subnetwork e.g.:

IP Address: 10.77.77.1SubnetMask: 255.255.255.0

At the switch login prompt, type in:Login: adminPassword: password


Rev. 4.21 6


Setting Initial IP Address on Br3800

For Br3800, use the COM portTo set IP address:

Establish a connection to the shell over the serial portEnter the data to the ipAddrSet commandCopy the IP Address to the label provided on the front panel

To reset factory defaults:Establish a connection to the shell over the serial portEnter the configDefault command


IP

0 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 1 51 0 1 0 1

Setting Initial IP Address on Br3800To set IP Address using the Serial Port:

1. Connect the serial port to a computer using Brocade serial cable.2. Enter the data to the ipAddrSet command

Ethernet IP addressSubnet maskGateway IP addressFibre Channel IP address (if any)Subnetmask (if any)

To reset factory defaults in the event that a user changes a password and forgets it, the password can be reinitialized:

1. Connect the serial port to a computer using Brocade serial cable.2. Establish a connection to the shell.3. Enter this sequence of commands:

– Admin> switchdisable – Admin> configDefault – Admin> switchenable

For more information on commands, see the Fabric OS Manual.


Rev. 4.21 7


Logon

Login capabilitiesAdmin: access all switch commands in the help menuUser: access commands that do not modify the switch state

e.g. show commands

Changing passwords`admin’ login to change passwords for all usersType `passwd’ command; Each username is displayed in sequence allowing the administrator to change each password and usernameEnter a password or name while a user name is displayed to replace the existing password or nameResetting to factory defaults will clear the customer password


Rev. 4.21 8

LogonBased on VxWorks shell, Telnet is the superset of the management choices User needs the following for successful logon:

•Switch Name or IP address•User Name•Password

Only one Telnet session is allowed per switch at a time Login synopsis:

– telnet [Switch_Name | IP_address]– SilkWorm login: [admin | user]– Password: [password]

Survival commands you should knowhelp [command] to display list of available commandsswitchShow to display switch informationnsShow to display SNS informationfabricShow to display fabric informationreboot to reboot the switchfastBoot to reboot without testing memory

Tip: Command to save and restore customer’s configuration. This is recommended before any modification. Proceed as follow:

configUpload ↵ (Immediately `Return’ will force interactive mode)IP Address: IP address of the host to save file to.Username: rootProtocol: <rshd |ftp>Password: (mandatory if Ftp is used)Filename: (e.g. /tmp/my_config)

And to restore the configuration, in case of troubles:ConfigDownload (Immediately `Return’ will force interactive mode)IP Address: IP address of the host to save file to.Username: rootProtocol: <rshd |ftp>Password: (mandatory if Ftp is used)Filename: (e.g. /tmp/my_config)


Rev. 4.21 9


Upgrading the Firmware

firmwareDownloadDownload the firmware file from sourceLogon to the switch as `admin’ via telnetType the command

FirmwareDownload ↵ (Immediate`Return’ will force interactive mode)

Host IP AddressUsername = `root’Protocol = <rshd |ftp>Password (mandatory if Ftp is used)Filename (e.g. v2.1.9f)

fastboot


Rev. 4.21 10

Upgrading Brocade FirmwareNew updates to firmware can be downloaded from a host through an Ethernet port.Host systems should be Unix (supporting rsh daemon) or Windows (NT/9x) using Brocade

supplied utilities.Connect to switch via telnet session and Use firmwareDownload command, then fastboot

command.From hp-ux1. Retrieve the firmware (i.e: v2.0) onto your UNIX system, in root (/) directory. (Since UNIX

comes with rshd and cat daemons, you won’t need to retrieve the rsh.ZIP file)2. Log onto the UNIX system as root and edit two files

a) /etc/hosts to enter the switchName and its IP addressb) edit the/ .rhosts file to enter the switchNamec) Notes: For Solaris system, please also check the / etc/nsswitch.conf file to make sure

the ‘hosts’ lookup table is appropriately set.NOTE: if you log in as a normal user (not root) in step 2, the /.rhosts file is referred to the

user’s home directory .rhosts file.3. From the UNIX system, telnet into the switch and download firmware with the command:firmwareDownload “Your UNIX IP address”, “your UNIX login user’s name”, “/v2.0”

From Windows 9x/NT there are five steps:1. Putting the switch firmware version (i.e:v2.0) onto your Window 9x/NT and the rsh.ZIP file

in the SAME directory (i.e:C\)2. Unzipping the rsh.ZIP program (using Winzip) and you’ll have two programs: rshd.exe and

cat.exe You’d have three files in C:\ (v2.0; cat.exe; rshd.exe)3. Downloading firmware from the Win95/WinNT onto the switch4. From your PC, bring up two MS-DOS windows. In the first window, type rshd.exe and

leave it alone. On the second window, telnet into the switch (login as ‘admin’ with password is ‘password’).

5. After successfully telnet into the switch, issue the command:

FirmwareDownload “Your PC IP address”, “your PC login user’s name”, “C:/v2.0”

Note: that you MUST use the UNIX like slash (/) in the command. When finished, the prompt will return, type ‘reboot’.


Rev. 4.21 11


Factory Default Settings

configDefaultThe configDefault command is used to reset some of the switch functional configuration parameters to their default values. All the parameters are set to defaults except the following:

World Wide NameEthernet MAC addressEthernet IP address and subnetmaskIP gateway addressOEM customizationSNMP configurationZoning configurationQuickloop configurationLicense keysSystem name


Rev. 4.21 12

Factory Default SettingsThe configDefault command is used to reset some of the switch functional configuration parameters to their factory default values. In addition, this command configures the switch to boot from its internal firmware if it has been previously configured to boot from the network. This command may not be executed on enabled switch; you must first disable the switch using the switchDisable command.

Switch: admin> configDefaultCommitting configuration…done.

Because some configuration parameters are cached by the switch, it is recommended that switch be rebooted immediately following the execution of the config Default, otherwise unexpected behavior may result. All configuration parameters are reset to their default values, with the exception of the following:

World Wide NameEthernet MAC addressEthernet IP address and subnetmaskIP gateway addressOEM customizationSNMP configurationZoning configurationQuickloop configurationLicense keysSystem name

Tip: When you come to reset the switch to its factory default configuration you may need to reset as well QuickLoop and Zoning configurations. In such a situation you may need to use the complete following sequence

switchDisableqlDisablecfgClearcfgSaveconfigDefaultswitchEnable fastboot (optional)

Warning: Before resetting the switch to its factory default, make sure you save customer’s configuration to a file on a host, with configUpload command.


Rev. 4.21 13


Configuration Parameters

Switch:admin> configureConfigure…Fabric parameters (yes,y,no,n): [no] ↵Virtual Channel parameters(yes,y,no,n): [no] ↵Arbitrated Loop parameters(yes,y,no,n): [no] ↵System service (yes,y,no,n): [no] ↵


Rev. 4.21 14

Configuration ParametersThe configure command is navigated by entering a series of collapsible top-level menus. Each menu divides up the various switch configuration parameters into logical groupings, which include: fabric parameters, virtual channel parameters, arbitrated loop parameters, and system service parameters. Each top-level menu and its associated sub-menus consist of a text prompt, a list of acceptable values, and the current value (shown in brackets). The current value is used in the absence of an entered value when a carriage return is the only input entered at the prompt. These menus change with firmware revisions. The current firmware may not match this example completely.Detail of configure command:switch:admin> configure

Configure...Fabric parameters (yes, y, no, n): [no] yesDomain: (1..239) [1]BB credit: (1..16) [16]R_A_TOV: (4000..120000) [10000]E_D_TOV: (1000..5000) [2000]Data field size: (256..2112) [2112]Non-SCSI Tachyon Mode: (0..1) [0]Disable Device Probing: (0..1) [0]Unicast-only Operation: (0..1) [0]VC Encoded Address Mode: (0..1) [1]Per-frame Route Priority: (0..1) [0]Virtual Channel parameters (yes, y, no, n): [no] yesVC Link Control: (0..1) [0]VC Class 2: (2..5) [2]VC Class 3: (2..5) [3]VC Multicast: (6..7) [7]VC Priority 2: (2..3) [2]VC Priority 3: (2..3) [2]VC Priority 4: (2..3) [2]VC Priority 5: (2..3) [2]VC Priority 6: (2..3) [3]VC Priority 7: (2..3) [3]Arbitrated Loop parameters (yes, y, no, n): [no] yesSend FAN frames?: (0..1) [1]System services (yes, y, no, n): [no] yesrstatd (on, off): [off] onrusersd (on, off): [off] onNo changes.Disable Translative Mode: (0..1) [1]


Rev. 4.21 15


Using Show Commands

nnssSShhooww DDiissppllaayyss llooccaall NNaammee SSeerrvveerr iinnffoorrmmaattiioonn,, wwhhiicchh iinncclluuddeess iinnffoorrmmaattiioonn aabboouutt ddeevviicceess ccoonnnneecctteedd ttoo tthhiiss sswwiittcchh,, aanndd ccaacchheedd iinnffoorrmmaattiioonn aabboouutt ddeevviicceess ccoonnnneecctteedd ttoo ootthheerr sswwiittcchheess iinn tthhee FFaabbrriicc..

nnssAAllllSShhooww DDiissppllaayyss tthhee ((2244--bbiitt FFiibbrree CChhaannnneell)) ppoorrtt IIDDss ooff aallll ddeevviicceess iinn aallll sswwiittcchheess iinn tthhee FFaabbrriicc.. TThhee nnssAAllllSShhooww ccoommmmaanndd ooppttiioonnaallllyy ttaakkeess aann iinntteeggeerr ppaarraammeetteerr,, tthhee vvaalluuee ooff tthhee FFCC--PPHH ttyyppee.. TThhee ppoossssiibbllee vvaalluueess ffoorr FFCC44TTyyppee aarree::

−− 55 -- FFCC--IIPP

−− 88 -- SSCCSSII--FFCCPP

sswwiittcchhSShhooww DDiissppllaayyss tthhee sswwiittcchh aanndd ppoorrtt ssttaattuuss..

ffaabbrriiccSShhooww DDiissppllaayyss aa lliisstt ooff sswwiittcchheess aanndd mmuullttiiccaasstt aalliiaass ggrroouuppss iinn aa ffaabbrriicc..

qqllSShhooww DDiissppllaayyss tthhee ccuurrrreenntt QQuuiicckkLLoooopp ccoonnffiigguurraattiioonn..


Rev. 4.21 16

Learningcheck

Learning Check1. Describe the process used to change the TCP/IP address of the switch.…………………………………………………………………………………………….…………………………………………………………………………………………….2. What command do you use to update the switch firmware?…………………………………………………………………………………………….3. What command do you use to reset the switch to its factory default settings?…………………………………………………………………………………………….


Rev. 4.21 17

Complete Labs 1-3


Rev. 4.21 18

Introduction to SANs Brocade Simple Name Server

Rev. 4.21 1


BrocadeSimple Name Server

Module 6


Rev. 4.21 2


Objectives

This unit prepares students to:Describe the process of how a device communicates with SNSIdentify the main attributes that are automatically registered with SNSDescribe Brocade’s implementation of SNS


Rev. 4.21 3


Brocade Distributed Simple Name Server at Common Address FFFFFC

Brocade distributed Simple Name ServerSimple Name Server (SNS) is a direct directory service that translates various names, addresses, and attributes related to Fibre Channel objects such as Ports, and Nodes. SNS is specified in the ANSI standard FC-GS 2 Simple Name Server.To the client/end user, a multi-switch fabric just looks like one switch and it is transparent. However in reality, there can be a multiple of switches and whichever switch is local to the host will serve as the Name Server for the fabric for that host. Meaning if the host sends a query to the Name Server, that local Name Server will query the rest of the switches and then respond to the host with all of the fabric information that corresponds to the query that came to it.Name Server Characteristics:

No single point of failureTransparent distribution: Each Distributed Name Server maintains and “owns”

local information and retrieves remote information from other Distributed Name Servers. Server-to-server communication is transparent to the external Name Service clientCaching: A distributed Name Server may cache remote information for a period of

time (~ 15 minutes)Server-to-server protocol (based on FC-CT)Get and Remove RequestsD_ID and S_ID are based on native address Ids


Rev. 4.21 4


SNS: Port and Nodes attributes

Port attributesNative port address (PID)Port name (World Wide Name variant)Class of service supported (2,3,F)FC-4 types (SCSI=8, IP=5)Port type (N,NL)Symbolic name (free-form information)

Node attributesNode name (World Wide Name)IP addressInitial Process AssociatorNode symbolic name

SNS: Port and Nodes attributesThe Symbolic Port_Name and Symbolic Node_Name are free from variables and are not restricted by the Name Service. If a node port (N_Port or NL_Port) registers no value, then the Symbolic Port_Name and Symbolic Node_Name default to a null value.Port Identifier: the 24-bit address assigned by the switch fabricFC-4 Types: Most common are SCSI (8) and IP (5)Port Type: N_Port or NL_PortSymbolic Port/Node_Name: a 256 Character field used by the vendor of the HBA or controller, etc.Initial Process Associator: a 64-bit field that is defined in the FC-GS 2 standard, reserved for future use. This is mainly a placeholder for when a company does decide to use it.


Rev. 4.21 5


1. Fabric Login (FLOGI) creates link from N-port to switch by adding entry to Simple Name Server (SNS) in the switch.

5. Each switch port automatically Logs_Out a link if it loses either light or carrier.

Port Login: PLOGIPLOGI

ACKACK

ACC : AcceptReceives Parameters

ACKACK

HSG PLOGI to KGPSA

Entire n-port login sequence

2. Fabric notifies all N-Ports of new entry in the SNS (State Change Notification).

3. Each N-Port does a PLOGI to SNS to learn all N-Port WWIDs, then disconnects.

4. An N-port does a Process Login (PRLI) through fabric to another N-port to create connections.

Initiator Fabric Recipient

ACC : Accept

ACK

Fabric Login : FLOGIACK

N-port F-port F-port N-port


Rev. 4.21 6


1. Fabric Login (FLOGI) creates link from N-port to switch by adding entry to Simple Name Server (SNS) in the switch.

5. Each switch port automatically Logs_Out a link if it loses either light or carrier.

Port Login: PLOGIPLOGI

ACKACK

ACC : AcceptReceives Parameters

ACKACK

HSG PLOGI to KGPSA

Entire n-port login sequence

2. Fabric notifies all N-Ports of new entry in the SNS (State Change Notification).

3. Each N-Port does a PLOGI to SNS to learn all N-Port WWIDs, then disconnects.

4. An N-port does a Process Login (PRLI) through fabric to another N-port to create connections.

Initiator Fabric Recipient

ACC : Accept

ACK

Fabric Login : FLOGIACK

N-port F-port F-port N-port


Rev. 4.21 7


nsShow

switch06:admin> nsshowThe Local Name Server has 8 entries {Type Pid COS PortName NodeName TTL(sec)N 021300; 3;50:06:0b:00:00:07:d5:00;50:06:0b:00:00:07:d5:01; na

FC4s: FCP Fabric Port Name: 20:03:00:60:69:10:32:0a

NL 0217c9; 3;10:00:00:e0:02:21:ef:7b;10:00:00:e0:02:01:ef:7b; naFC4s: FCP [HP C7200 1330]Fabric Port Name: 20:07:00:60:69:10:32:0a

NL 021d02; 1,2,3;10:00:00:10:83:b8:7e:f7;10:00:00:10:83:b8:7e:f7; naFabric Port Name: 20:0d:00:60:69:10:32:0a

NL 021f7a; 3;50:06:0b:00:00:05:92:33;50:06:0b:00:00:05:92:33; naFC4s: FCP [HP A5236A HP06]Fabric Port Name: 20:0f:00:60:69:10:32:0a

NL 021f8f; 3;21:00:00:20:37:0f:c4:95;20:00:00:20:37:0f:c4:95; naFC4s: FCP [SEAGATE ST39102FC HP03]Fabric Port Name: 20:0f:00:60:69:10:32:0a

NL 021f90; 3;21:00:00:20:37:26:51:79;20:00:00:20:37:26:51:79; naFC4s: FCP [SEAGATE ST39102FC HP03]Fabric Port Name: 20:0f:00:60:69:10:32:0a

NL 021f97; 3;21:00:00:20:37:0f:bc:f1;20:00:00:20:37:0f:bc:f1; naFC4s: FCP [SEAGATE ST39102FC HP03]Fabric Port Name: 20:0f:00:60:69:10:32:0a

NL 021f98; 3;21:00:00:20:37:0f:c5:62;20:00:00:20:37:0f:c5:62; naFC4s: FCP [SEAGATE ST39102FC HP03]Fabric Port Name: 20:0f:00:60:69:10:32:0a

Nsshow – prints the local name server informationType - the port type which can be either N_Port or NL_PortPid -the addressID of the port in hexidecimalCOS - the Class of Service supported by the devicePortName – the Port World Wide NameNodeName – the Node World Wide Name associated with the portTTL - the time-to-live value of the entry. Tis is usually set to n/aFC4s – The FC4 protocols usedFabric Port Name – the WWN of the switchports


Rev. 4.21 8


nsAllShow

switch06:admin> nsallshow

17 Nx_Ports in the Fabric {0110e0 011300 011f8f 011f90 011f97 011f98 021300 0217c9 021d02 021f7a 021f8f 021f90 021f97 021f98}

Switch06 : admin> nsAllShow 5

2 FC-IP Ports in the Fabric

011200 021200

Switch06 : admin> nsShowAll 8

8 FCP Ports in the Fabric

0118e2 0118e4 0118e8 0214e2 0214e4 0214e8

nsAllShow – displays the (24-bit Fibre Channel) port Ids of all devices in all switches of the fabric. The command usually takes a aprameter, the value of the FC4 type5 - FC-IP8 - SCSI-FCP


Rev. 4.21 9


Registered State Change Notification

Fabric Controller @ Fabric Controller @ 0xFFFFFD0xFFFFFD

RSCNRSCN RSCN

RSCN

HBA’s need to be “good citizens” to work correctly with the Name Server. The attributes of a “good citizen” are:•It supports RSCNs (Registered State Change Notification•It queries the Name Server for available ports•It accesses only ports that are defined by the Name Server


Rev. 4.21 10

Worldwide Name Format (WWN)Fibre Channel uses both fixed addresses and assigned addresses within the SAN. Each port is given a 64 bit fixed address as a unique identifier within the fabric. This is known as the World Wide Name. Standards allow for several IEEE formats. In the Brocade Switch:

Bits 63 to 60 determine whether a regular or an extended IEEE0001 identifies IEEE0010 identifies IEEE ExtendedBits 59-48IEEE is all zeroes,IEEE Extended:N_Port ID within the nodeF_Port ID within the fabricBits 47-0IEEE: IEEE addressIEEE Extended:IEEE address for the nodeIEEE address for the fabric

Example of Node WWN format from the SilkWorm: 10:00:00:60:69:00:00:4b

First two bytes 10:00 is the IEEE format. The first four bits are used by the vendor, and the remaining three nibbles “0:00” are all zeroes; they are reserved for future use.Example of Port WWN format from the SilkWorm:

20:04:00:60:69:00:00:55

First two bytes 20:04 are the IEEE Header. The difference between the Port WWN and the Node WWN is that in the Port WWN, the lower three nibbles can be used by the vendors as they wish; for instance, Brocade puts the port number in the three nibbles. “0:04” means this port is port 4.The WWN are assigned to manufacturers.


Rev. 4.21 11


portShowswitch06:admin> portshow 3

portFlags: 0x23805b PRESENT ACTIVE F_PORT G_PORT U_PORT LOGIN NOELP LED ACCEPT

portType: 3.1

portState: 1 Online

portPhys: 6 In_Sync

portScn: 6 F_Port

portRegs: 0x80000000

portData: 0x10f505b0

portId: 021300

portWwn: 20:03:00:60:69:10:32:0a

SNS: Requests and auto-registrationName Service Requests: SNS uses FC-CT, which is the Fibre Channel Common Transport standard. Typical requests that the host will issue are for the port address of FCP/SCSI storage devices. Other requests might be:

•Registering IP address•Registering FC4-Types (IP, SCSI)•Registering a symbolic Port or Node Name

Auto-Registration:SNS performs third-party registration with the Name Server on behalf of an N_Port or NL_Port (public device) after the latter has completed a Fabric Login (FLOGI), the attributes registered are:

•Port address ID,•Port type,•Port name,•Node name,

Classes of serviceSNS probes FCP Fabric-connected devices and registers that information with the Name Server eliminating the need for these devices to explicitly register this information.


Rev. 4.21 12


SNS Common Addresses

x‘FF FFF5’ — Multicast Server. This service acts as the destination port for responses and aggregates them properly in order to send a single response to the multicast originator so as to provide a reliable, acknowledged unidirectional multicast. x‘FF FFF6’ — Clock Synchronization Server. Used for synchronizing fabric-wide real-time clocks.x‘FF FFF7’ — Security Key Distribution Server. Used for securely distributing authenticated private keys for port pairs so that they may securely exchange encrypted data. x‘FF FFF8’ — Alias Server. When alias addresses are implemented, the entity addressed at this address will maintain Alias identifier mappings. x‘FF FFF9’ — Quality-of-Service Facilitator. This service allows reservation of some fraction of the available link bandwidth and/or allows a guarantee of maximum delivery latency. x‘FF FFFA’ — Management Server is an optional entity, which collects and reports information on link usage and quality, errors, and so on. x‘FF FFFB’ — Time Server is an optional entity used to distribute synchronized time values. x‘FF FFFC’ — Directory Server or Name Server. An optional entity contained either within the fabric or at an N_Port that maintains tables correlating N_Port Address Identifiers with N_Port Name Identifiers and possibly many other port characteristics. x‘FF FFFD’ — Fabric Controller. A required entity within the fabric that controls the general operation of the fabric, including fabric initialization, frame routing, generation of link responses, and setup and tear down of dedicated connections. x‘FF FFFE’ — Fabric Login Server is a required entity within the fabric that provides access to the fabric for Fabric Login (FLOGI). This entity assigns, confirms, or reassigns N_Port address identifiers and notifies N_Ports of the operating characteristics of the fabric, if present. x‘FF FFFF’ — Broadcast address. If this optional function is supported, the fabric will route frame with this destination ID to every connected N_Port.


Rev. 4.21 13

Learning check

Learning Check1. What functionality does a SNS server provide?

………………………………………………………………………………………….

………………………………………………………………………………………….

2. How does a device register with the SNS server?

………………………………………………………………………………………….

………………………………………………………………………………………….

3. What information is gathered by the SNS server?

………………………………………………………………………………………….

………………………………………………………………………………………….


Rev. 4.21 14

Introduction to SANs Brocade Zoning

Rev. 4.21 1


Brocade Zoning

Module 7


Rev. 4.21 2

Objectives

This module prepares students to:• Describe zoning concept advantages and limitations• Define the different types of zoning for Brocade switches• Configure a multiple zone fabric• Perform merging of two fabrics with zoning configurations


Rev. 4.21 3

Security Comparisons

Advantages DisadvantagesHost Level(OV-SAM Allocater)

•Mixed heterogeneous devices•Independent of target devices•Ease the management of a storage pool.

•Management software is host/HBA specific and must be present on all hosts in the SAN to be effective.•Someone can plug a host into the SAN that can see and corrupt data. This is particularly vulnerable in a multiple campus situation.

Infrastructure level(Switch zoning)

•Independent of hosts and target devices.•Safe-guard unauthorized hosts to interrupt the SAN

•Granularity is at port and node level or WWN level (Not LUN level).•When connecting switches from different vendors, zoning choices may be limited.•Most switch vendors use WWN zoning when flexibility is required. (i.e. Separating devices on loop into different zones.)

Device Level (Secure Manager XP/VA, Selective Storage Presentation, EVA/MA)

•Best granularity – LUN level•Best safe-guarded – from anywhere

•Device dependent – low end array or JBOD may not support this function.•Administration may become cumbersome for large node counts (e.g. 200 NT servers sharing a LUN for mail database.)•Firmware changes can disturb settings (Secure Manager Only)

Comparisons of different security models

Security ComparisonsData access can be implemented at several different levels within the SAN environment. Each of these has advantages and disadvantages. The level selected will be chosen for the particular needs of the customer’s SAN environment.Host level security offers a single point of management for a large data center. Hosts with many different operating systems can be managed by Open View Storage Area Manager, as clients. However, a host that lacks the software may be unaware of the disk allocations and may access and corrupt storage in the SAN.Switch level security may be more secure than host security, but can be constrained by the topology. Port zoning is very limiting for topologies that have many devices connected through hubs. Switch level zoning cannot separate LUN access for a given device.Device level security is highly secure, but may require time-consuming administration to implement. Not all devices have this function. Firmware changes on these devices may alter the security function, and can impact the availability of the device.


Rev. 4.21 4

Overview – Brocade zoning product

• Licensed product, part of the standard HP bundle• Allows a finer segmentation of Storage Area Networks• Used to setup barriers between different operating environments

– to deploy logical Fabric subsets by creating defined user groups– to create test or maintenance areas that are separate within the

Fabric• Allows the flexibility to manage a SAN to meet different closed user

groups objectives

Overview – Brocade Zoning ProductMicrosoft Windows and HP-UX do not interact well on the same fabric. If the hba’s of the two operating systems see each other, data corruption can occur. Another example for creating zones is to secure devices from each other, such as: payroll, engineering data, corporate finance.You cannot zone down to the LUN level. This is accomplished using Secure Manager or Selective Storage Presentation.


Rev. 4.21 5

Zoning Example

The server in the red zone sees one loop of disks and one tape The server in the red zone sees one loop of disks and one tape The server in the blue zone sees two storage arraysThe server in the blue zone sees two storage arraysThe server in the green zone sees one loop, one array, and one The server in the green zone sees one loop, one array, and one tapetapeNo server sees loop 2No server sees loop 2

Zoning ExampleZones may be configured dynamically. The number of zones and zone members are effectively unlimited. Zones vary in size and shape, depending on the number of Fabric connected devices and device locations. Devices may be members of more than one zone. This is called “over-lapping zones”. In addition, multiple configurations can be created, as an example, for enterprise backup and for normal work access. Zone members see only members in their zones and, therefore, access only one another. A device not included in a zone is not able to access any devices devices.


Rev. 4.21 6

A Hierarchical Structure

Zoning Components

Fabric may have more than one CfgOnly one Cfg can be active

Cfg is a container for zonesZones may overlap

Zone is a container for membersMembers may be Defined with Aliases

Member can beA fabric physical port numberA node or port WWNAn AL_PAAn Alias

Cfg_ICfg_I

Zone_ABCZone_ABC

Member#1Member#1

Member#2Member#2

Member#nMember#n

Zone_XYZZone_XYZ

Cfg _NCfg _N

FabricFabric


Rev. 4.21 7

Zoning enforcement mechanisms

Soft Zoning: Name-Server assistedName Server restricts visibilityAlways available when zoning enabledRelies on `good citizens’ for security (No WWN probing)No reduction in performance

Hard Zoning: Hardware enforcedAvailable when certain rule conditions are met through hardware logic checkingProvides additional security in addition to Soft zoningInhibits illegal access from `bad citizens’

Zone EnforcementSoft zoningSoft zoning is software enforced Brocade zoning. The zoning enforcement is implemented in the firmware, using the entries of the Simple Name Server to determine if the transaction is allowed. The members of the zones must be “good citizens”. A “good citizen” is a member that uses the Name Server, supports RSCN (Remote State Change Notification) and does not circumvent the Name Server for access to other ports.A “bad citizen” is a node that probes the switch, either because of malfunction or malice, to access a device that it should not access. What this means is if there is a server/HBA/Driver that will probe the ports on the switch, that server/HBA/Driver would be able to talk to any device it found because it did not use the Name Server and behave properly.In the Brocade 2x00 Silkworm switches, WWN zoning is software enforced. The term “soft zoning” became used to mean the same thing as World-wide Name zoning. In the Brocade 3x00 Silkworm switches, WWN zoning can be hardware enforced. It is important to separate the enforcement from the format for zoning.Hard zoningHard zoning is hardware enforced zoning. Zoning is enforced by the ASIC. It is not vulnerable to probing by a “bad citizen” node. In the Brocade 2x00 Silkworm switches, port zoning is enforced in the hardware. The term “hard zoning” came to mean the same thing as “port zoning”. With the 3x00 Silkworm switches, WWN zoning is also hard zoning. This terminology is no longer valid.


Rev. 4.21 8

2x00 zoning

Mechanisms• Soft Zoning for WWNs• Hardware Zoning for Domain, Port• Enforced at fabric-level• QuickLoop – Soft Zoning for AL-PAs

Granularity• (domain, port), WWNs, AL-PA (QuickLoop)

Security• Hardware enforcement is very secure• Probing possible when soft zoning

Brocade 2x00 zoningZoning is enabled differently on the 2x00 family and the 3x00 family. On the 2x00 Silkworm switches, WWN zoning is enforced with software, relying on Simple Name Server entries for validation. WWN zoning has been called “soft zoning” because of this implementation. Port zoning is enforced in the ASIC hardware. Port zoning has been called “hard zoning” because of this implementation. However, this naming is no longer correct because of the changes in the 3x00 Silkworm switches. The references to hard and soft zoning must be differentiated from those to port and WWN zoning.Hardware enforced zoning is inherently more secure than software enforced zoning. A node, through malice or malfunction, may succeed in accessing a port outside its zone if it bypasses the Simple Name Server and probes directly for WWNs.


Rev. 4.21 9

2x00 zoning examples

Hardware Zoning (2x00 Silkworm)Port Zoning (Domain, Port) is enforced in hardware.Hardware enforced zoning possible only when no WWN exists in “Effective Configuration”Example:

aliCreate “Host1a”, “1,2”aliCreate “Storage1a”, 1,4”zoneCreate “pZone1”, “Host1a; Storage1a”zoneCreate “pZone2” ,”1,5; 2,4”

Software Zoning (2x00 Silkworm)Software enforced zoning when WWN exists in “Effective Configuration”Example:

aliCreate “Host1b”, “50:00:0b:01:b2:2f:14”aliCreate “Storage1b”, “50:00:0b:01:00:2f:25”zoneCreate “pZone3”, “Host1b; Storage1b”zoneCreate “pZone4”, “50:00:0b:01:b2:2f:14; 50:00:0b:01:00:2f:25”

Mixed configurations are enforced in Soft zoning, as with the following command:zoneCreate “mZone1” “3,4; WWN1”

2x00 Silkworm Zoning ExamplesHard ZoningIn the 2x00 Silkworm switch, hard zoning is used to enforce Port zoning.In the examples shown, the alias for port zoning defines the device associated with the alias name using the domain and port. The alias can then be used as a member when defining a zone (pZone1). However, aliases are not required. A zone can be defined using the domain and port reference (pZone2).Soft ZoningIn the 2x00 Silkworm switch, soft zoning is used to enforce World-wide Name zoning.In the examples shown, the alias for WWN zoning defines the device associated with the alias name using the world-wide name. The alias is then used as a member when defining the zone (pZone3). A fourth zone is shown where the world-wide name is directly entered in the zone definition (pZone4)Mixed ConfigurationsWhere both port and WWN references are used in the configuration definitions, the enforcement will default to software zoning.


Rev. 4.21 10

3x00 zoning

MechanismPort-level zoning is Hardware EnforcedWWN zoning is Hardware EnforcedMixed zones, Fabric Assist zones and Quick Loop zones remain enforced through Name Server (Soft zoning)

GranularitySame as in v2.x

SecurityHardware enforced zoning is very secureProbing is still possible for ports with no hardware enforcement

3x00 Silkworm ZoningThe 3rd Generation ASIC on the 3x00 Silkworm switches can enforce both Port and WWN zoning. Therefore, both Port and WWN zoning are “hard zones”. The term “hard zoning” can no longer refer to port zoning.


Rev. 4.21 11

3x00 zoning examples

The Effective Configuration may contain both “hard” and “soft” zones.Hardware Zoning (3x00 Silkworm)Port Zoning (Domain, Port) or WWN zoning is enforced in hardware. Example:

aliCreate “Host1a”, “1,2”aliCreate “Storage1a”, 1,4”zoneCreate “pZone1”, “Host1a; Storage1a”zoneCreate “pZone2” ,”1,5; 2,4”aliCreate “Host1b”, “50:00:0b:01:b2:2f:14”aliCreate “Storage1b”, “50:00:0b:01:00:2f:25”zoneCreate “pZone3”, “Host1b; Storage1b”zoneCreate “pZone4”, “50:00:0b:01:b2:2f:14; 50:00:0b:01:00:2f:25”

Software Zoning (3x00 Silkworm)Mixed configurations are enforced in Soft zoning, as with the following command:

zoneCreate “mZone1” “3,4; WWN1”


Rev. 4.21 12

If a device is defined by port (D,P) in one zone and by WWN in another, the hardware enforcement at the port will be turned OFF and the zoning control will be controlled by Name Server. This is called ‘soft porting’.

Example:aliCreate “Host1a”, “1,2”aliCreate “Storage1a”, 1,4”zoneCreate “pZone1”, “Host1a; Storage1a”zoneCreate “pZone2” ,”1,5; 2,4”aliCreate “Host1a”, “50:00:0b:01:b2:2f:14”aliCreate “Storage1b”, “50:00:0b:01:00:2f:25”zoneCreate “pZone3”, “Host1b; Storage1b”zoneCreate “pZone4”, “50:00:0b:01:b2:2f:14; 50:00:0b:01:00:2f:25”

Host1a is defined by port zoning in pZone1 and by WWN zoning in pZone3.

Soft Porting

Soft portingIn the example shown, the device identified as “Host1a” is defined using port zoning in Zone1, and defined using its WWN in Zone3. Either definition alone would result in Hard zoning. However, when the device is defined in each zoning type within a single configuration, the switch will not be able to enforce zoning within the ASIC. Soft zoning will be used, instead.


Rev. 4.21 13

Zoning Rules(3x00)

ERROR/WARNING CODESHARDSOFTMIX(warning) - Overlapping SOFT/FA and HARD zones.WWNINPORT – Overlapping hard WWN and PORT zones.FAQLMIX – Overlapping hard WWN or PORT zones with QL or FA zonesDRIVERERR – port-level detected unknown errorNOMORECAM – port-level depleted hardware resourceCHECKBADWWN – WWN probing detected

Error / Warning CodesSome common error codes are shown here. They point to configuration conditions which should be corrected for proper zoning function.

HARDSOFTMIX(warning) - Overlapping SOFT/FA and HARD zones.A device is defined in a soft zone or in a loop (using AL-PA) and in a hard zone. Soft zoning will be used to enforce the zoning for all zones.WWNINPORT – Overlapping hard WWN and PORT zones.A device is configured in a 3x00 Silkworm switch. It is configured using WWN in a WWN zone and using the domain/port in a Port zone. Soft zoning will be used to implement the zoning.FAQLMIX – Overlapping hard WWN or PORT zones with QL or FA zonesA device has been configured in a Fabric Assist or QuickLoop zone using the AL-PA. The same device is defined in another zone using either the WWN or the port. DRIVERERR – port-level detected unknown errorNOMORECAM – port-level depleted hardware resourceCHECKBADWWN – WWN probing detected


Rev. 4.21 14

Port Zoning

HostO

Switch 1

HSG

Orange Zone:1,1;2,11;1,8;1,5;2,15;1,4;2,14 1 11

84 14

HostG

Switch 2

HSG

DLT

Green Zone:2,1;1,11;2,8;2,5;1,15;2,4; 1,141 11

8

414

XP XP

5 15 155

1,5 2,15

1,4 2,14 1,14 2,42,5 1,15

2,8Bridge

DLTDLT

1,8Bridge

DLT

1,1 2,11 2,1 1,11

Port Zoning

Port zoning is defined within the Brocade switch by specifying the switch Domain and physical Port. In the example there are two zones defined: the Orange Zone and the Green Zone. Access is allowed only through the specified port. If the cable to a port is moved to another port, the device will be unavailable. If the port is down or disabled, there will be no device access on that path. This example shows alternate paths in the zones.Port zoning logic is consistent with the HP-UX address and device file structure. Port zoning cannot separate or individually identify zone members of a looplet. All devices on the loop are defined in the zone by the port. Port zoning can be a disadvantage for consolidated storage devices, like the XP family. All the LUNs accessed through the port belong to the zone.


Rev. 4.21 15

World-Wide Name Zoning

Host O

Switch 1

HSG

Orange Zone:O-L0/6;O-L0/7O-DLTS;O-XP1;O-XP2O-FC1, O-FC2

1 118

4 14

Host G

Switch 2

HSG

DLT

Green Zone:G-L0/6;G-L0/7G-DLTS;G-XP1; G-XP2G-FC1; G-FC2

1 118

414

XP XP

5 15 155

B-XP1 B-XP2

B-FC1 B-FC2 G-FC1 G-FC2

G-XP1 G-XP2

G-DLTSBridge

DLTDLT

B-DLTS

Bridge

DLT

G-L0/6 G-L0/7B-L0/6 B-L0/7

B-L0/6: 50:06:0b:00:00:e6:e8

World-wide Name ZoningWWN zoning is defined within the Brocade switch by specifying the node World-Wide Name. In the example there are two zones defined: the Orange Zone and the Green Zone. The 2x00 Silkworm switch uses the Simple Name Server to identify the host and target devices. The 3x00 Silkworm switch uses the ASIC to identify the hosts and targets.WWN zoning has been called “soft zoning” because it is enforced through software on the 2x00 Silkworm switches. This reference is no longer valid for 3x00 Silkworm switches which use hard zoning for WWN and port zones.Access is not limited to a specified port. If the cable to a port is moved to another port, the device will still be available. However, on HP-UX, the target device now has a new devicefile name. This example shows alternate paths in the zones.WWN zoning can separate or individually identify zone members of a looplet. All devices on the loop are defined in the zone by the individual node WWNs. Usually the port WWN is specified.On 3x00 Silkworm switches, there is some performance degradation while WWN zoning is initiated. Performance will increase to normal after initialization.


Rev. 4.21 16

Zoning commands (1 of 4)

• Zoning commands are issued from any switch in a fabric (you must be logged-in to the admin account) to manage zones, zone aliases, and zone configurations.– This is also true when working from the zoning GUI.

• All add, create, delete, and remove commands modify the defined configuration only. – Very important: This has no effect on the effective

configuration until you execute a ‘cfgEnable’ command.


Rev. 4.21 17


• Configuration commands allow you to manipulate fabric configurations:– cfgAdd – Adds a zone to a configuration.– cfgCreate – Creates a zone configuration.– cfgDelete – Deletes a zone configuration.– cfgRemove – Removes a zone from a configuration.– cfgShow – Shows the zone configurations (defined and

effective).• Alias commands allow you to manipulate zone aliases:

– aliAdd – Adds a member to a zone alias.– aliCreate – Creates a zone alias.– aliDelete – Deletes a zone alias.– aliRemove – Removes a member from a zone alias.– aliShow – Shows all defined aliases.


Rev. 4.21 18


• Zone commands allow you to manipulate zones.– zoneAdd – Adds a member to a zone.– zoneCreate – Creates a zone.– zoneDelete – Deletes a zone.– zoneRemove – Removes a member from a zone. – zoneShow – Shows all defined zones.


Rev. 4.21 19


• Management commands allow you to manipulate preexisting configurations.– cfgEnable – Enables a zone configuration.– cfgDisable – Disables a zone configuration (caution).

Note: You should disable the effective configuration by enabling another configuration (for example, cfgEnable ‘new_configuration’).

– cfgSave – Saves all zoning information into flash memory.(to all switches in the fabric)

– cfgShow – Shows all zoning information.– cfgClear – Clears all zone configurations.

• Must be followed by a cfgSave. • If it is your intention to get rid of all zoning fabric-wide, with switch

FW v2.6.0c, this command must be preceded by a cfgDisable command.


Rev. 4.21 20

Zone Management Commands (1 of 5)

CreateConfigurationsaliCreatezoneCreatecfgCreate

Brocade SilkWorm

ConfigurationDefinitions

EnabledConfiguration

FlashMemory

SwitchDomain

1

SDRAM

cfgEngMktZoneEngZoneMkt

Enters configuration information into SDRAM only.


Rev. 4.21 21


Brocade SilkWormConfiguration

DefinitionsEnabled

Configuration

FlashMemory

SwitchDomain

1

SDRAM


cfgEnable “cfgEngMkt”


Flash memory gets updated on a cfgenable


Rev. 4.21 22



DefinitionsEnabled

Configuration

FlashMemory

SwitchDomain

1

SDRAM


cfgDisable


Cfgdisable only disables the effective configuration.


Rev. 4.21 23


Brocade SilkWorm

ConfigurationDefinitions

EnabledConfiguration

FlashMemory

SwitchDomain

1

SDRAM


cfgclear

Cfgclear does not clear the effective (active) configuration.


Rev. 4.21 24



DefinitionsEnabled

Configuration

FlashMemory

SwitchDomain

1

SDRAM


cfgSave


Writes “name”Only to flash

If you have issued a cfgclear and then a cfgsave the switch will now save the cleared SDRAM into flash and everything in the switch will be cleared.


Rev. 4.21 25

Creating a Configuration Example

=> aliCreate => aliCreate ““Alias_NameAlias_Name””,,””member;member;membermember;member;member””=> zoneCreate => zoneCreate ““Zone_NameZone_Name””,,””Alias_Name;1,2; WWNAlias_Name;1,2; WWN””=> cfgCreate => cfgCreate ““cfg_Namecfg_Name””,,””Zone_Name;Zone_NameZone_Name;Zone_Name””=> cfgEnable => cfgEnable ““cfg_Namecfg_Name””=> cfgSave => cfgSave ““cfg_Namecfg_Name””

=> => configUploadconfigUpload ““host_IPhost_IP””,,””useruser””,,””/file_name/file_name””,,””passwordpassword””

Creating a Configuration ExampleThe following sequence of commands creates and enables a configuration called Day_Time, which is made up of two zones, Red_Zone and Blue_Zone…aliCreate “Red_Server”, “10:00:00:00:c9:20:29:22”aliCreate “Blue_Server”, “1,6”aliCreate “Blue_Storage” , “50:00:0b:00:00:07:d0:c8”aliCreate “Red_Storage” , “1,5”zoneCreate “Red_Zone”, “Red_Server; Red_Storage”zoneCreate “Blue_Zone”, “Blue_Server; Blue_Storage”cfgCreate “Day_Time”, “Red_Zone; Blue_Zone”cfgEnable “Day_Time”configUpload …

Alternate forms of the commands:zoneCreate “Red_Zone”

,”10:00:00:00:c9:20:29:22;“50:00:0b:00:00:07:d0:c8”zoneCreate “Blue_Zone” , “1,6;1,5”


Rev. 4.21 26

Changes to the Fabric

Adding a new switch/fabricNot previously had zoning or cfgClear command has been runWhen added, all zone configuration data is copied from the zoned fabric into the new switch/fabric

Joining two switches/fabricIf both fabrics have identical zone configuration data and the same configuration is enabled, fabrics join for one larger fabric If fabrics have different zone configuration data, the ISL is segmented. One switch configuration may become disabled.

Splitting fabricIf an ISL goes down, causing a fabric to split into two separate fabrics, then each new fabric retains the same zone configurationFabric will re-merge when ISL is back up and no zone changes have been madezone changes have been made

Changes to the FabricAdding a new Switch/Fabric:A new switch is a switch that has not previously been connected to a Fabric with ZONING configured or adding a Fabric that has not previously had Zoning configured or, been cleared by using the cfgClear command before connecting it to the Fabric.When a new switch or Fabric is connected to a zoned Fabric, all zone configuration data is immediately copied from the zoned Fabric into the new switch/Fabric. If a zone configuration is enabled in the Fabric, then the same configuration becomes enables in the new switch. After this operation, thecfgShow command displays the same output on all switches in the Fabric, including the new switch.


Rev. 4.21 27

Zoning Example #1

Host A

Switch6

FC10

DLT

0/2/0/0 0/4/0/0

3

715

Host B

Switch7

FC10

DLT

0/2/0/0 0/4/0/0

37

15

8 9

XP XP

0 0

ZoneG: 6,0; 6,3

ZoneG is enabled. Which devices can Host A see? Which devices can Host B see?


Rev. 4.21 28

Zoning Example #2a

Host A

Switch6

FC10

DLT

3

715

Host B

Switch7

FC10

DLT

37

15

8 9

XP XP

0 0

ZoneB: 6,0; 6,3; 6,7; 6, 15

1)No ISL: ZoneB on Domain6. No Zone on Domain7.Which devices can Host A see? Which devices can Host B see?


Rev. 4.21 29

Zoning Example #2b

Host A

Switch6

FC10

DLT

3

715

Host B

Switch7

FC10

DLT

37

15

8 9

XP XP

0 0

ZoneB: 6,0; 6,3; 6,7; 6, 15

After connecting the ISL, which devices can Host A see? Which devices can Host B see?


Rev. 4.21 30

Zoning Example #3a

Host A

Switch6

FC10

DLT

3

715

Host B

Switch7

FC10

DLT

37

15

8 9

XP XP

0 0

ZoneB: 6,0; 6,3 ZoneG: 7,0; 7,3

No ISL: ZoneB on Domain6. ZoneG on Domain7.Which devices can Host A see? Which devices can Host B see ?


Rev. 4.21 31

Zoning Example #3b

Host A

Switch6

FC10

DLT

3

715

Host B

Switch7

FC10

DLT

37

15

8 9

XP XP

0 0

ZoneB: 6,0; 6,3 ZoneG: 7,0; 7,3

After connecting the ISL, which devices can Host A see? Which devices can Host B see?


Rev. 4.21 32

Learning check

Learning Check1. What is the difference between hard and soft zoning?

………………………………………………………………………………………….

………………………………………………………………………………………….

2. Describe the relationship between zone members, zones, and zoning configurations.

………………………………………………………………………………………….

………………………………………………………………………………………….3. What is the process for merging two separate fabrics together as it pertains to

zoning?

………………………………………………………………………………………….

………………………………………………………………………………………….


Rev. 4.21 33

Lab #

Lab title


Rev. 4.21 34

Introdcution to SANs Cascading Brocade Switches

Rev. 4.21 1


Cascading Brocade Switches

Module 8


Rev. 4.21 2


Objectives

This unit prepares students to:• Perform cascading of Brocade switches• Verify dynamic routing in a fabric• Configure static route in a fabric• Verify fabric topology• Verify and Configure Dynamic Load Sharing• Verify and Configure In Order Delivery• Verify and Configure ISL Trunking


Rev. 4.21 3


PATH and ROUTE Terminology

A ROUTE is a map between an input port and an available PATH, to reach the next hop

Domain 1Domain 1

Domain 3Domain 3Domain 2Domain 2

22 88

33 1111

1414

55

77

1313

HostHost

StorageStorage

Cost = 1000Cost = 1000 Cost = 1000Cost = 1000

Cost = 1000Cost = 1000

44

1212

A PATH is a chain of switches from source to destination

Cost = 1000Cost = 1000


Rev. 4.21 4


Brocade Path Selection

• Fully distributed (no single point of failure within the fabric)

• Fast recovery in case of link or switch failure• Traffic load sharing over equivalent paths• Link State Protocol – FSPF (Fibre Channel Shortest Path First)

• Identify switches by DOMAIN_ID• Determines the shortest paths, then maps into Routing

tables

Brocade Path SelectionThis technology provides very fast convergence time in the presence of failures, and loop-free paths at all times. The path selection protocol computes the best path between any two switches in the fabric, and then programs the hardware routing tables.The most important features offered by Brocade’s path selection protocol are:

•Fast computation of new paths in case of failure.•Guaranteed in-order delivery even during topology changes.•Traffic load sharing over equivalent paths.•Negligible bandwidth usage for protocol frames.•High priority protocol frames.

The path selection protocol starts automatically at boot time. No configuration is necessary. All the paths are established at this time, and will not change unless there is a failure, or a new ISL comes online that provides an equal or better path. The protocol is resilient to both hardware and software failures, being able to find an alternate path around the failing link or switch in a very short time.The path selection protocol makes minimal use of the ISL bandwidth, leaving virtually all of it available for user data. In a stable fabric, without topology changes, a SilkWorm switch transmits about 50 bytes every 10 seconds on each ISL.Path selection protocol frames have the highest priority in the fabric.This guarantees that a control frame is not delayed by user data on a congested link, and insures that the protocol can make routing decisions very quickly.


Rev. 4.21 5


Paths and routes

• A path is a chain of switches from a source switch to a destination switch.– In a fabric, every switch has a path to every other switch.

• There is usually more than one path from a source switch to a destination switch.

• Every port on a switch has a route defined to get to every other switch.– There can only be one active route between a source switch

and a destination switch.– The switch dynamically establishes routes based on optimum

paths.


Rev. 4.21 6


Fabric shortest path first (FSPF)

• Switches use a FSPF algorithm to determine the optimal routing path between every two switches in a fabric.

• FSPF has the following benefits:

– Fast computation of new paths in the case of an E-Port failure or SAN topology change.

• The shortest route information (hop path) is then moved into thehardware routing tables.

– Traffic load sharing over equivalent cost metric paths

– High priority FSPF frames for fast fabric reconfiguration times

• FSPF frames used negligible bandwidth.

– Completely flexible for any fabric topology.

– Guaranteed in-order delivery, even during topology changes.

In a stable fabric (one not undergoing topology changes), FSPF transmits less than 100 bytes of information every 10 seconds.


Rev. 4.21 7


Path selection and link costs (1 of 2)

• Path 1 goes directly from domain 1 to domain 3.• Path 2 goes from domain 1 to domain 2, then to domain 3.

– Only the shortest path (path 1) will be programmed into the routing tables.


Rev. 4.21 8


Path selection and link costs (2 of 2)

• The optimum path between two switches is the one with the lowest associated link cost.– The link cost is a cost metric which is inversely proportional to

the bandwidth of the ISL.• The link cost for a 1Gbps link is 1000.• The link cost for a 2Gbps link is only 500.• The linkCost command can be used to view or modify the

associated cost of any ISL. For example: linkCost 7, 500

• The cost of a path between two switches is the sum of the cost of all the traversed ISLs.

– HP does not recommend the use of the linkCost command to alter costs because the switches are designed to automatically optimize SAN performance.


Rev. 4.21 9


Route selection default behavior

Domain 3

Domain 1

Domain 2

13

4

5

26

1

1

78

9

10

11

10001000

1000

1000

1

2

3

B-Series switches follow the FSPF protocol. This protocol assigns a cost metric according to bandwidth. The default value for a 1 Gbps ISL is 1000, with 2 Gbps ISL being 500. The path that is selected from a source domain (switch) to a destination domain (switch) is based on the total least metric cost.

In the route selection diagram above, it is important to note that domain 1 has a total of 3 routes to domain 3. Two of the routes have a total metric cost of 1000 and one route has a total metric cost of 2000. Based on the FSPF protocol, only the routes with the total least metric cost count will be kept in the routing tables. In this case, 2 routes will be kept in domain 1’s routing tables.

Because the 3rd route to domain 3 is not kept in domain 1’s routing table, it will never be selected to route frames to domain3. If this route has very little usage, the total possible bandwidth will not be realized between domain 1 and domain 3.


Rev. 4.21 10


Route selection default behavior

• Command issued from domain 1: linkCost 6, 500

• Command issued from domain 2:linkCost 12, 500

Route selection using linkCost (1 of 2)

Domain 3

Domain 1

Domain 2

13

4

5

26

1

1

78

9

10

11

10001000

500

500

1

2

3

12

The linkCost command is not bi-directional.

4 – 47

In the SAN diagramed above, the link metric cost has been altered with the linkCostcommand to include a 3rd route from domain 1 to domain 3.

The fabric administrator must have a good understanding of existing routing throughout the fabric along with bandwidth utilization. If this is not taken into consideration when setting link costs, a negative performance condition may occur. An example of this would be setting the link cost between domain 1, port 5 to domain 3 to a value of 500. Since the FSPF protocol will only keep entries in the routing table for routes with the least metric cost count, this will reduce the number of routes from domain 1 to domain 3 to one! This may create a negative performance condition, since there is now only one path to domain 3 from domain 1.

By issuing the linkCost command and specifying a port, this will display the metric cost for a single ISL. By issuing the linkCost without a parameter, this will display all ISLs and their link costs.


Rev. 4.21 11


Route selection using linkCost (2 of 2)

• No commands issued from domain 3

Domain 3

Domain 1

Domain 2

13

4

5

26

1

1

78

9

10

11

500500

250

250

1

2

3

12 500

500

4 – 48-49

In the route selection diagram above, the link metric cost has been altered with thelinkCost command to include a third route from domain 1 to domain 3.

By changing the metric cost for an ISL, this may also change the routing tables kept in the fabric and thus influence routing and overall path selection. In the above example, the linkCost command has been invoked from domains 1 and 2. First the linkCost for the ISL from port 6 on domain 1. The second linkCost command has been issued from domain 2 to reduce the metric cost from 1000 to 500 between domain 2, port 12 and domain 3. By reducing the ISL metric cost for these two ISLs, we now have 3 routes from domain 1 to domain 3 with equal total metric costs. All 3 routes will be kept in domain 1’s routing table.


Rev. 4.21 12


Brocade Routing Modes

Dynamic Routing:– Default– Round robin– F16 and Silkworm switches

Static Routing: – Manually configurable– Automatically re-route if ISL goes down– Automatically re-take effect when ISL is back up – Silkworm 2x00/3x00 and higher switches only

Brocade Routing modesThere are two routing modes for Silkworm switches:

Dynamic routing: This is the default routing mode. Only shortest equivalent paths are mapped to routes into routing tables. For instance, if there is a total of 10 paths and only 2 are cost 1000, the rest are cost 2000 or greater, from one source switch to destination switch; then only the shortest, equivalent paths will be put into the routing tables. The routing process will map dynamically each of the shortest paths to a route in a round robin way, according to the demand of input portsStatic routing: may be needed in some rare case. The admin user account should be used to configure manually the route. In case of failure of the ISL used for the static route the switch maps dynamically an available shortest path to the route; dynamic routing is used instead. Later when the failing ISL is up back, the static route is re-established automatically again.Cost per Hop: 1gb = 1000 2gb =500Note: Static routing is available on Silkworm 2x00/3x00 switches only.


Rev. 4.21 13


Dynamic Load Sharing (DLS)

Node2

Node3

Node1

0 1 2 3

4 5 6 7

0 1 2 3

4 5 6 7

Switch 1

Switch 2

Initial Topology

Node3

Node1

0 1 2 3

4 5 6 7

0 1 2 3

4 5 6 7

Switch 1

Switch 2

Change with DLS Set:Switches re-computes the

routes

Node3

Node1

0 1 2 3

4 5 6 7

0 1 2 3

4 5 6 7

Switch 1

Switch 2

Change with DLS Reset:

No change to initial routes

Node2 Down

Node2 Down


Rev. 4.21 14


Load sharing

• When multiple ISLs exist between two switches, and the ISLs have the same link cost:

– Each inport selects one ISL port for all communications to the other switch—a static route.

– The outports are assigned to the inports based on the lowest number of static routes so that the workload is distributed as equally as possible between all available ISLs.

• The routes are configured dynamically and are changed/rerouted every time a utilized E-Port fails, is turned off, or is added between two switches.

– This is called a fabric reroute.

An inport is any switch port that is not an E-port.

An outport is any E-port or ISL.


Rev. 4.21 15


Configuring Static Routes

Domain 1

Domain 3Domain 2

2 8

3 11

14

5

7

13

Host

Storage

ISL#3 ISL#4

ISL#1

4

12ISL#2

uRouteConfig <Input_Port>, <Destination_Domain_ID>, <Output_Port>

uRouteRemove 2, 3

uRouteConfig 2, 3, 12

uRouteRemove <Input_Port>, <Destination_Domain_ID>


Rev. 4.21 16


What is ISL Trunking?

A group of ISLs which functions as a single high speed ISLEnables traffic to be evenly distributed across grouped ISLsPreserves in-order deliverySuch a set of links is called a trunking groupLicensed Option

Fc 16B Fc 16B

Interswitch Link (ISL) trunking enables multiple links between switches to be combined to form single logical Fibre Channel links at aggregate speeds up to 8Gbps. These ‘trunks’simplify network design, optimize bandwidth utilization, and ensure that server-storage performance remains balanced under heavy network load. They result in better utilization of bandwidth at lower overall management cost.Trunking guarantees in-order delivery, which requires hardware support. Frames are striped from an individual sequence across multiple links rather than simply choosing a less loaded ISL out of the group for all the traffic for a given transaction. The link between switches is maintained, even if one of the individual ISLs fails.


Rev. 4.21 17


ISL Trunking

• Aggregate traffic onto fewer logical links• Automatically created when switches are connected• Managed as a single logical 8Gbps ISL• Fault-tolerant – will withstand failure of individual links


Rev. 4.21 18


Before ISL Trunking

DiskB

switch

1

1

1

HostA

HostB

HostC

1.6 [Gb/s]

11

DiskC

11

switch

DiskA

0.4 [Gb/s] 1.6 [Gb/s]

1

<1.6 [Gb/s] 0.4 [Gb/s] <1.6 [Gb/s]

Assume 2Gb/s links.Prior to ISL Trunking, blocking could occur if Host A io and Host C io are routed through the same ISL. Static routing commands can mitigate this situation, but manual tuning is required.One of the primary performance constraints with current SAN implementations is that all fabric switches, regardless of vendor, have had to load share data traffic across multiple ISLs to ensure the in-order delivery that storage subsystems require. For instance, with three servers routed across two ISLs, each would be assigned a route—typically through a round-robin method. At least two servers would be routed across the same ISL, potentially leading to congestion on the ISL, even if the other ISL has available bandwidth.Because devices and applications do not typically sustain or achieve full 100 Mbit/sec performance today, congestion has typically not been an issue. If congestion did become a concern, administrators could intervene by creating routes to prevent the higher performing servers from using the same ISL. However, as scaling and performance requirements continue to grow, more devices have to be routed across the network— increasing the likelihood of congestion. Moreover, it becomes more difficult to create and manage static routes due to the complexity of the routes and network.To help solve this potential scalability issue, Brocade has developed high-performance ISL Trunking.


Rev. 4.21 19


ISL Trunking

DiskB

switch

1

1

1

HostA

HostB

HostC

1.6 [Gb/s]

11

DiskC

11

switch

DiskA

0.4 [Gb/s] 1.6 [Gb/s]

1

1.6 [Gb/s] 1.6 [Gb/s]0.4 [Gb/s]

ISL Trunking automatically ties together as many as four 2 Gbit/sec ISLs to form up to a full 8 Gbit/sec ISL. Rather than having to load share across multiple links (as is the case with all of today’s fabric switches), the links can balance the load across all of the ISLs through trunking.This load balancing helps ensure that there is no congestion in larger networks.The only possibility of congestion with a load-balanced trunk would be if the total traffic across the trunk actually exceeded the entire trunk bandwidth. Moreover, the number of devices routed across the ISLs no longer affects the congestion.This benefit enables administrators to focus on overall network performance rather than worrying about possible congestion due to a chance routing of multiple higher performance devices across a specific ISL.


Rev. 4.21 20


ISL Trunking Commands

islshow, switchshow• Display ISL information

trunkShow• Display information about trunking on a switch

portCfgTrunkport• Configure a port for trunking

switchCfgTrunk• Configure a switch for trunking

trunkDebug• Debug a trunking connection


Rev. 4.21 21


islShow

Aggregate Bandwidth

Islshow gives information about all ISLs.Trunking Groups are identified in the rightmost column. Aggregate bandwidth is displayed. Only the Trunk Master is shown (slave links are not).This screen shot show 3 ISL’s. One 6G Trunk and 2 1 G ISL’s.


Rev. 4.21 22


switchShow

Here we can see the 2 1 G ISL’s on ports 0 and 1. Even though they are on the same ASIC and going to the same switch they are not capable of trunking because they are at 1 G speed. They are most likely ISL’d to a 2x00 switch incapable of trunking feature.Ports 4, 5 and 6 are going to another switch capable of trunking. Port 5 is the trunkmaster, 4 + 6 are the slave ISL’s


Rev. 4.21 23


Trunkable Switch

• Fabric initialization process determines whether an E_Port can become a trunking port

• To guarantee in-order delivery all trunking ports must:

– All ports in trunking group must reside in same ASIC quad

– All ports must run the same 2G link speed– “one way skew value” between all ports in

trunking group must be almost identical. The skew value is related to the cable length.

The ASIC quads are ports 0-3, 4-7, 8-11, 12-15 on a 16 port switch.The ASIC quads are ports 0-3, 4-7, 8-11, 12-15, 16-19, 20-23, 24-27, 28-31 on a 32 port switch.


Rev. 4.21 24


The Deskew counter

• Used to ensure in-order delivery across trunking links• When there is cable difference, the longer cable will have

the bigger deskew• Bigger deskew impact the performance

The Deskew CounterThe algorithm for frame striping across the trunk will delay the transmission of frames in order to account for the longer transmission time through longer cable. When the deskew values become large, trunking will continue to function, but performance will be impacted.


Rev. 4.21 25


Deskew counter

• Minimum deskew is 150 ns• Maximum deskew is 2550 ns• Deskew values are represented in nanoseconds divided by 10• Maximum cable difference is about 400m• Recommend less than 30m difference to avoid any performance impact

Fc 16B Fc 16B

A programmable timeout constant register is available in Bloom to define the time during which a list is bound to a particular transmit port. There are some potential performance advantages for minimizing the link-to-link skew and allowing the the binding time constant to be variable. For example, suppose a number of relatively large frames are available to be ttransmitted from the same queuing list at the same time. At some point, the first frame may begin transmission on one link. Then, even though the first frame is in progress, a second frame may be transmitted across another link in the group after the binding timeout period expires. If the binding time is small, the delay between transmission of multiple frames from a given list across different links may be minimized, allowing the transmit queue to be drained that much more quickly, possibly relieving switch congestion more rapidly.


Rev. 4.21 26


trunkShow

Port 5 is the Trunk Master

Ports as Connected onEach switch

The trunking master implicitly defines the trunking group. All ports with the same master are considered to be part of the same group. Each trunking group includes a single trunking master and from 1 to 3 trunking slave links. The first ISL found in any trunking group is assigned to be the trunking master, also known as the principal ISL. After the trunking group is fully established, all data packets intended for transmission across the trunk are dynamically distributed at the frame level across the ISLs in the trunking group, while preserving in-order delivery.


Rev. 4.21 27


portCfgTrunkPort

Portcfgtrunkport is used to turn on or off trunking configuration for one port. The last parameter indicates on or off ( 1 for on and 0 for off).


Rev. 4.21 28


switchCfgTrunk

Enables or Disables trunking at the switch level.


Rev. 4.21 29


trunkDebug

This command reports one of the following messages based on the trunking property of the specified two ports:

• Switch doesn't support trunking• Trunking license required• port<port_id> is not E port• port<port_id> trunking disabled• port<port_id> speed is not 2G• port<port_id> and port<port_id> are not on same quad• port<port_id> and port<port_id> connects to different switches• port<port_id> is not Trunk port due to: E port being disabled,or trunking may be disabled at remote port• port<port_id> and port<port_id> can't trunk, please check link length to make sure difference is less than 400m


Rev. 4.21 30


portCfgShow

Port CfgShow displays the entire switch port configuration on one screen.


Rev. 4.21 31


Dynamic Load Sharing (DLS)

• Trunking handles the dynamic load sharing (DLS) at frame level when traffic is running

• FSPF handles DLS at port login time and based on number of ports

• When trunking slave port goes offline, there is minimal impact on the user traffic

• When trunking master port goes offline, re-route is done by FSPF. This can take several seconds


Rev. 4.21 32


FSPF For Trunking

• Trunking group is considered as ONE fat ISL identified by the master link• Load assigned to trunking group is based on the total bandwidth of the links in the group


Rev. 4.21 33


Trunking Summary

• Trunking combines 2 to 4 ISLs to in effect create “One fat ISL”• The new 3rd Generation chip allows traffic to be evenly distributed across ISLs and preserves in-order delivery• Trunking is only available on 2G switches • Trunking can join up to 4 ISLs to create an 8 Gbps ISL trunk• There are many useful commands to determine trunking status and performance


Rev. 4.21 34


Routing Algorithm

Route with minimum hops is taken

1

3

5

6

1 4

3

2

4

6

1 hop route

2 hop routeA

B

C

QL + Fabric

4 1

3

Q: What is a hop?A: When a FC frame is transferred from one switch to another is called hop.

Q: When is the routing table built?A: During switch boot time or if a new ISL is established.


Rev. 4.21 35


Routing Example 1.1

Load balancing across both 1 hop routes.

1

3

5

6

4

1 4

1

3

2

4

6switch A

switch B

switch C

QL + Fabric3

Q: Which Route will be taken for IOs from the hosts to the storage?A: The fabric will always select the route with the shortest path to reach the destination domain. In the example above, all IOs from both hosts will be routed by switch A through port 5 to switch C / port 2. Only in case of a link failure, the IOs will be rerouted through switch B. Once the failed link between switch A and C comes up again, the IOs will be rerouted to the shorter path again.

Q: How is the shortest path determined?A: The shortest path between a source and a destination port of the fabric is the route with the minimum number of hops (switch to switch).


Rev. 4.21 36


Routing Example 1.2

Both paths will choose the remaining 1 hop route.

1

3

5

6

4

1 4

1

3

2

4

6switch A

switch B

switch C

QL + Fabric3


Rev. 4.21 37


Routing Example 1.3

Now, 2 hop route became the shortest to destination.

1

3

5

6

4

1 4

1

3

2

4

6switch A

switch B

switch C

QL + Fabric3


Rev. 4.21 38


Routing Example 2.1

All paths run over shortest routes.

1

3

5

6

4

1

3

4

1

3

2

4

6switch A

switch B

switch C

QL + Fabric


Rev. 4.21 39


Routing Example 2.2

One path is forced to take a 2 hop route.

1

3

5

6

4

1

3

4

1

3

2

4

6switch A

switch B

switch C

QL + Fabric


Rev. 4.21 40


In Order Delivery (IOD)

Reset (Default)• If an E_Port goes away, the switch takes about 650ms to

detect and run routing protocol and the new route will take effect

Set• When set, the routing code enforces a hold-down period of

entire fabric (equal to E_D_TOV) after an old route is deleted, before installing a new route to the same destination domain

• When IOD parameter is set, frames are either delivered in-order or dropped


Rev. 4.21 41


In order delivery (IOD)

•iodSet – This command enforces in-order delivery of frames when a fabric topology change takes place.

•iodReset – This command allows out-of-order delivery of frames during a fabric topology change.

•iodShow – Indicates whether or not IOD is set.

• When a configDefault command is issued, switches will not have iodSet.

– The HP default is to have iodSet. Therefore, it must be set manually after the issuance of a configDefault command.

– hp maintains a default switch parameter settings location at thefollowing URL:• http://stgwrks.americas.cpqcorp.net/dsgga/switchsettings/switchsettings.h

tm


Rev. 4.21 42


Fabric data flow


Rev. 4.21 43


Fabric Timing Values

Important Rule:

Don’t change any of thedefault timing values on theswitch.

Otherwise: In order delivery can*not* be guaranteed !

Note: Currently most operating systems rely on in-order-package-delivery from the IO system. Changing the fabric timing values may cause out-of-order-package-delivery in a meshed environment. The SW2800 routing algorithm will ensure in order package delivery in a meshed environment even in case of topology changes, when using the default timing values.

Remember: Out-of-order-package-delivery will cause most current OS versions to crash!


Rev. 4.21 44


uRouteShow

switch38:admin> uRouteShowLocal Domain ID: 38In Port Domain Out Port Metric Hops Flags Next (Dom, Port)----------------------------------------------------------------------------

2 29 7 1500 2 D 39,739 7 500 1 D 39,7

5 29 13 1500 2 D 39,1339 14 500 1 D 39,14

In addition to the uRouteShow command, there is a uRouteConfig command. The syntax is: uRouteConfig <INPUT-PORT>, <DESTINATION DOMAIN ID>, <OUTPUT PORT>. Example: uRouteConfig 5, 4, 13.

The uRouteConfig command allows you to manually configure the routing tables. This command can have negative performance consequences and should therefore be used with great caution or under the direction of hp switch engineering.

If the OUTPUT PORT fails, the route is treated as a regular route and is allocated to a different path if one is available. When the OUTPUT PORT resumes functioning, the port is rerouted back to the static route.

If a PORT has a static route, the flags field in uRouteShow is set to S instead of D. Note that this does not affect the flags field in the topologyShow command which will always indicate dynamic paths (D).

To remove a route use the uRouteRemove command:ExampleuRouteRemove <INPUT-PORT>, <DESTINATION DOMAIN ID>


Rev. 4.21 45


uRouteShow


2 29 7 1500 2 D 39,739 7 500 1 D 39,7

5 29 13 1500 2 D 39,1339 14 500 1 D 39,14

In addition to the uRouteShow command, there is a uRouteConfig command. The syntax is: uRouteConfig <INPUT-PORT>, <DESTINATION DOMAIN ID>, <OUTPUT PORT>. Example: uRouteConfig 5, 4, 13.

The uRouteConfig command allows you to manually configure the routing tables. This command can have negative performance consequences and should therefore be used with great caution or under the direction of hp switch engineering.

If the OUTPUT PORT fails, the route is treated as a regular route and is allocated to a different path if one is available. When the OUTPUT PORT resumes functioning, the port is rerouted back to the static route.

If a PORT has a static route, the flags field in uRouteShow is set to S instead of D. Note that this does not affect the flags field in the topologyShow command which will always indicate dynamic paths (D).

To remove a route use the uRouteRemove command:ExampleuRouteRemove <INPUT-PORT>, <DESTINATION DOMAIN ID>


Rev. 4.21 46


topologyShow

switch38:admin> topologyShow3 domains in the fabric; Local Domain ID: 38Domain Metric Hops Out Port In Ports Flags Bandwidth Name-----------------------------------------------------------------------29 1500 2 7 0x00000004 D 2 (Gbs) "switch29"

13 0x00000020 D 2 (Gbs) 14 0x00000000 D 2 (Gbs)

39 500 1 13 0x00000000 D 2 (Gbs) "Switch39"7 0x00000004 D 2 (Gbs) 14 0x00000020 D 2 (Gbs)


Rev. 4.21 47


uRouteShow and topologyShow


4 38 1 1500 2 D 39,1239 2 1000 1 D 39,10

switch29:admin> topologyShow3 domains in the fabric; Local Domain ID: 29Domain Metric Hops Out Port In Ports Flags Name-----------------------------------------------------------------38 1500 2 1 0x00000010 D "switch38"

2 0x00000000 D

39 1000 1 2 0x00000010 D "Switch39"1 0x00000000 D


Rev. 4.21 48




4 38 1 1500 2 D 39,1239 2 1000 1 D 39,10

switch29:admin> topologyShow3 domains in the fabric; Local Domain ID: 29Domain Metric Hops Out Port In Ports Flags Name-----------------------------------------------------------------38 1500 2 1 0x00000010 D "switch38"

2 0x00000000 D

39 1000 1 2 0x00000010 D "Switch39"1 0x00000000 D


Rev. 4.21 49



Switch39:admin> uRouteShowLocal Domain ID: 39In Port Domain Out Port Metric Hops Flags Next (Dom, Port)----------------------------------------------------------------------------

2 29 10 1000 1 D 29,238 7 500 1 D 38,7

7 29 12 1000 1 D 29,110 38 13 500 1 D 38,1312 38 14 500 1 D 38,1413 29 10 1000 1 D 29,214 29 12 1000 1 D 29,1

Switch39:admin> topologyShow3 domains in the fabric; Local Domain ID: 39Domain Metric Hops Out Port In Ports Flags Bandwidth Name-----------------------------------------------------------------------29 1000 1 10 0x00002004 D 1 (Gbs) "switch29"

12 0x00004080 D 1 (Gbs)

38 500 1 7 0x00000004 D 2 (Gbs) "switch38"13 0x00000400 D 2 (Gbs) 14 0x00001000 D 2 (Gbs)


Rev. 4.21 50


Cascading HUBs

InexpensiveMore flexible than SCSI

Latency & throughput decreases on large loops

NL

NL

NL

NLNL

NL

Q: How many HUBs can I cascade?A: Currently HP supports 2 cascaded HUBs.

Q: How many links can I have between the two HUBs?A: Exactly 1 link.

Q: Would additional links between both HUBs increase performance?A: No.


Rev. 4.21 51


May block transmission ordetect NL_Port removal

NL_Port

Optional ActiveControl

TXRX

or Normal

Bypassed

Always Listen

Port bypass circuit

Port Bypass CircuitA bypass circuit is a simple device. The signal coming to the NL_Port is sent to the port for processing and also into the bypass circuit proper. The port always listens. The transmit portion is where decisions are made.The circuit can either propagate the incoming signal or transmit from the NL_Port, but not both. The circuit can sense that an NL_Port has been removed and will transition to propagate the incoming signal.A control function permits either the NL_Port itself or a hub function to bypass the port even when it is present. An NL_Port can remove itself or be removed by an external agent.Hubs provide a fault tolerance and fault isolation ability to a loop. Normally, loops break whenever any element on the loop is removed or powered down. Hubs allow a port to be bypassed so that a failure at any NL_Port or node does not bring down the entire loop.


Rev. 4.21 52


Cascading Switches [1 hop]

Virtually one big switchExcellent performance

More Expensive

E E

Q: What is a ‘hop’?A: When a FC frame is transferred from one switch to another is called a hop. The above config shows a 1 hop setup.

Q: How many links between switch (ISL) are supported?A: Currently 8 ISLs are supported between any of two switches.


Rev. 4.21 53


Meshing Switches [>1 hop]

Virtually one big switchExcellent scalability

Increasing complexityNeeds careful design

E

E E

E

EE

Q: How many hops are currently supported?A: The Br3800 is capable of operating in a 7 hop configuration. Please check for current supported configurations in the ESBU Configuration Matrix

http://essd-tm.boi.hp.com/essdatc/config_matrix.htm

Q: How many links between switch (ISL) are supported?A: On the SW3800 all 16 ports can be used as ISLs.


Rev. 4.21 54


How to increase fabric fan out ?

Combines best of bothReasonable price

consider Hubs

N

NL

N

NL

F F

NL

Q: Why can’t there be 2 FL_Ports on a Loop?A: An FL_Port has a dedicated AL_PA = ‘00’h on the Loop. Two or more FL_Ports would cause a address conflict during the LIP. Only one FL_Port will be able to gain AL_PA(00h), all other FL_Port would go into a non participation mode.


Rev. 4.21 55


What would happen here?

NL

NL

Communication within the loop still possible.

portDisableNL

Q: What happens if a FL_Port is being disabled?A: The disabled FL_Port will go into bypass mode. Therefore the connected loop will be able to continue to communicate locally. Any node within the the loop will not be able to communicate with any device outside the loop. Also any node outside the loop will not be able to communicate with devices inside the loop.

Note: The Loop will become a private loop until the FL_Port is being re-enabled.


Rev. 4.21 56


Connecting two switches to a hub

FL

No, only 1 FL_port per loop allowed.

F

F

F

FFL

Q: Can there be 2 FL_Ports connected to a Loop?A: No. An FL_Port has a dedicated AL_PA = ‘00’h on a Loop. Two or more FL_Ports would cause an address conflict during the LIP. Only one FL_Port will be able to gain AL_PA(00h), all other FL_Port would go into a non participation mode.


Rev. 4.21 57


Cascading two switches through a hub

No, ISLs can’t go through HUBs.

EE

Q: Can I cascade two switch through a Hub?A: No. E_Ports run a point to point protocol whereas the Hub always runs arbitrated loop protocol.


Rev. 4.21 58


Switched fabric vs. loop

100 MB/sec per port200 MB/sec per port full dplx

F

F

F

F

100 MB/sec total

Non blockingParallel Media

BlockingShared Media


Rev. 4.21 59


Link utilization

Switch offersBuffer to Buffer credit

HUB has no buffers

Keeps links streaming!16 credits per port

Much bandwidth unused No Buffers

rep

req

sw

req req req req

req

Q: What is the FC cable latency?A: Although speed of light may be fast but it becomes noticeable slow when using Gbit rates as the FC does. For example on a 10km long wave FC cable multiple FC frames can be on transit

512 byte frame -> 10 frames on transit1024 byte frame -> 5 frames on transit2048 byte frame -> 2.5 frames on transit

Note - The Buffer-To-Buffer credit on each switch port makes it possible to use the FC link pipeline effect. This reduces the overall latency if multiple frames are being transmitted.


Rev. 4.21 60


Balance inter-switch links

Don’t bottleneck the ISLs

QL + Fabric

Q: How many ISL’s can a switch have?A: Maximum throughput between two switches can be achieved having 8 ISLs. All 16 ports on a switch could be used as ISLs (E-ports) on the SW3800 if the switch happens to be an interconnecting switch in a meshed configuration.

Q: Does the switch automatically load balance across ISLs?A: Yes. But it will only load balance across ISLs which have an equivalent path.

Q: Adding additional ISLs, will it change the HW paths on the host?A: No. Adding ISLs is transparent to the hosts.


Rev. 4.21 61


HP standard SAN fabric topologies

Cascaded fabrics

Ring fabrics

Meshed fabrics

Backbone fabrics

Skinny Tree

Fat Tree

Backbone using Core/Edge

switches


Rev. 4.21 62


Cascaded fabric topology

A cascaded fabric SAN is a set of switches connected together, by one or more ISLs, in a tree-like arrangement.Cascaded fabric designs are well suited to environments with local data access patterns. In these cases, I/O requests from a servers attached to a given switch are made most often to storage systems that are attached to the same switch. Groups of servers and their storage systems can be connected to the same switch to provide the highest level of I/O performance.Cascading provides a means to scale the SAN for additional connectivity of servers and storage, and allows for centralized management and backup, while maintaining the high I/O performance of local access. Cascaded designs can also be used for centralized or distributed access; however, traffic patterns should be well understood and should be factored into the design to ensure that there are an adequate number of ISLs to meet performance requirements. Using more than one ISL between switches in a cascade also provides redundant paths between a given pair of switches in the fabric. HP highly recommends that cascaded designs be implemented with a minimum of two ISL connections on each switch, either as a pair of ISLs between the same two switches or by connecting every switch to at least two other switches in the fabric.Features of a cascaded fabric:

Accommodates diverse geographic conditionsScales easily for additional connectivitySupports shared backupSupports shared managementOptimal for local accessMultiple subscribers share 100 MB/sec or 200 MB/sec2 to 28 switches, up to 7 hopsScales easily for additional connectivityServers and storage typically on the same switchHP highly recommends a minimum of 2 ISL connections on each switch


Rev. 4.21 63


Ring fabric topology

A ring fabric is a continuous ring of switches connected together into a single fabric. Each switch is connected to adjacent switches, with the last switch in the ring connected back to the first. This arrangement of switches provides almost the same level of fabric resiliency as the mesh design, with full fabric connectivity and at least two internal fabric paths or routes.If you use fewer than 12-switches in a ring constructed using B-Series Product Line switches, you can add additional switches to the outside of the ring. These satellite switches provide additional user ports with only a slight reduction in fabric availability. For example, 11 satellite switches can be connected to a 11-switch ring. This results in a 22-switch fabric and maintains the overall seven hop limit.Features of a ring fabric topology:

• Modular design• Simple and non-disruptive scaling• Supports shared backup• Supports centralized management• Optimizes local access needs• Multiple Subscribers share 200 MB/sec or 400 MB/sec• 2 to 15 switch ring• Up to 7 hops• 2 to 11 switches in ring• Up to 11 satellite switches (one outside switch is cascaded from each of the ring switches.)• Max. 7 hops in either direction• Multiple ISLs for resiliency


Rev. 4.21 64


Meshed fabrics

In a meshed fabric design, all of the switches are interconnected so there are at least two paths or routes from any one switch to any other switch in the fabric. This type of connectivity provides fabric resiliency. If a single ISL or ISL port interface fails, the fabric can automatically re-route data through an alternate path. The new route can even pass through additional switches in the fabric.As switches are added to a meshed topology, the number of ISLs required to maintain full connectivity between any switch and any other switch becomes excessive. This reduces the number of user ports in comparison to the total number of ports, which is a measure of the connection efficiency of the fabric.Meshed fabrics are well suited to applications where data access is a mix of local and distributed. The full connectivity (or high connectivity, in the case of modified meshes) supports many-to-many access, while at the same time allowing localized access toindividual switches, servers and storage.Features of a meshed fabric:

• 2 to 28 switches, up to 7 hops• Can be configured for “many to many” or local access, or a mix• Provides protection against link and switch port failures• Scales easily for additional connectivity• Optimal distributed access is inherent in the design• Centralized or “Many to One” Traffic Patterns


Rev. 4.21 65


Backbone fabrics

A backbone fabric has one or more Fibre Channel switches primarily dedicated to connecting to other switches within the fabric. The backbone switches provide high bandwidth and redundant connectivity to the other switches. This type of implementation offers the best "many-to-many" connectivity.Backbone fabrics are well suited for implementations where the primary requirement is for full network “many-to-many” connectivity with high performance. They are the most conservative design approach in cases where the I/O traffic patterns are unknown or varying. They are also the best design to choose if you plan to implement SAN-wide storage pooling and sharing, and for environments that use storage virtualization.


Rev. 4.21 66


Advantages of backbone fabrics

More efficient port expansion than a full mesh fabric design • All edge switches are only two

hops apartFull "many to many" connectivity with evenly distributed bandwidth and redundant connectivity• Improved bandwidth with multiple

parallel ISLsOffer maximum flexibility for implementing mixed access types: • Local, Distributed, Centralized


Rev. 4.21 67


Fat tree and skinny tree designs

Backbone fabrics can be classified as fat tree or skinny tree

Defined by the number of ISLs used to connect the backbone switches to the edge switches

Skinny trees use fewer switches to supply the same number of user ports* < 50% of the ports are ISLsFat trees have more ISL

connections which provide higher cross sectional Bandwidth

*>= 50% of the ports are ISLs


Rev. 4.21 68


Fat vs. skinny example

Six 16 port switches, two backbone, four edge

Skinny = 64 user ports

Fat = 32 user ports

Skinny tree Fat Tree


Rev. 4.21 69


HA considerations

Have everything redundant!

Otherwise you are not HA. It’s assimple as it sounds.

QL + Fabric


Rev. 4.21 70


HA Recommendations

14

14

14

14

14

14

14

14

Non HAconfiguration

HAconfiguration

NoteIt is important to mention, for full HA behavior two completely independent Sans must be established. That means, no Inter-Switch-Link (ISL) between both fabrics must be installed. The reason for this strict rule is, that meshed switch configurations form one virtual fabric. There are some theoretical (rare) scenarios, where a fault switch may flood the fabric with packages causing massive congestion on the links.


Rev. 4.21 71


1

2

3 4

5

67

8

In a meshed environmentmake sure the max numberof hops is not exceeded!

Remember a failing ISL may causere-routing.

Watch the Max Hops!


Rev. 4.21 72

Learningcheck

Learning Check1. What is a Path? What is a Route?

…………………………………………………………………………………….

…………………………………………………………………………………….2. What are the two types of Brocade routing?

…………………………………………………………………………………….3. Describe the benefits of ISL trunking.

…………………………………………………………………………………….

…………………………………………………………………………………….4. If you wanted higher cross-sectional bandwidth, would you use a Fat-tree or Skinny-tree topology? Why?

…………………………………………………………………………………….

…………………………………………………………………………………….


Rev. 4.21 73

Complete Labs 4-8


Rev. 4.21 74

Introduction to SANs Storage Presentation to Operating Systems

Rev. 4.21 1


Storage Presentation to Operating Systems

Module 9


Rev. 4.21 2


Objectives

After completing this module, the student will be able to:• Verify device connectivity for OVMS, TRU64, HP-UX, and Windows OS’s


Rev. 4.21 3


Verifying adapter connectivity

OpenVMS• analyze /system

Tru64 Unix• emxmgr

HP-UX:• ioscan

Windows:• Device manager• LPUTILNT


Rev. 4.21 4


OpenVMS connectivity

– Issue the following command to display system configuration• analyze /system

sda> clue config (output below)

TR Adapter ADP Hose Bus BusArrayEntry Node CSR Vec/IRQ Port Slot Device Name / HW-Id-- ----------- ----------------- ---- ----------------------- ---- ---------------------- ---- ---- ---------------------------1 KA2208 FFFFFFFF.81457840 0 BUSLESS_SYSTEM2 PCI FFFFFFFF.81457C80 1 PCI

FFFFFFFF.814580A0 38 FFFFFFFF.85B6F800 FC PKA: 7 Symbios 895FFFFFFFF.81458110 48 FFFFFFFF.85B72800 40 9 PPB5

3 PCI FFFFFFFF.814584C0 1 PCI FFFFFFFF.814587F8 220 FFFFFFFF.85B76000 DC EIA: 4 DE602-AA i82558 100BaseTXFFFFFFFF.81458830 228 FFFFFFFF.85B78800 D8 EIB: 5 DE602-AA i82558 100BaseTX

4 PCI FFFFFFFF.81458EC0 0 PCI FFFFFFFF.81459230 28 FFFFFFFF.85B7E800 40 5 MFPCIFFFFFFFF.81459268 30 FFFFFFFF.85BA5000 8C 6 MFPCIFFFFFFFF.814592A0 38 FFFFFFFF.85BB1800 BC GZA: 7 ELSA GLoria Synergy (3Dlabs

P2A)FFFFFFFF.814592D8 40 FFFFFFFF.85BB4000 AC FGA: 8 KGPSA-CA (Emulex LP8000)FFFFFFFF.81459310 48 FFFFFFFF.85BB6800 9C FGB: 9 KGPSA-CA (Emulex LP8000)


Rev. 4.21 5


The emxmgr utility can be utilized to gather extensive information about your KGPSAs:#emxmgr -t emx2emx2 state information:Link : connection is UPPoint to PointFabric attachedFC DID 0x210413 (tells us the HBA is plugged into switch 1 port 4)Link is SCSI bus 3 (this adapter was assigned scsi bus 3)SCSI target id 7portname is 1000-0000-C921-07C4nodename is 1000-0000-C921-07C4N_Port at FC DID 0x210013 - SCSI tgt id 5 :portname 5000-1FE1-0001-8932 (HBA driver assigned SCSI ID 5 to the bottom right controller port – assuming MB)nodename 5000-1FE1-0001-8930Present, Logged in, FCP Target, FCP Logged in,N_Port at FC DID 0xfffffc - SCSI tgt id -1 :portname 20FC-0060-6900-5A1Bnodename 1000-0060-6900-5A1BPresent, Logged in, Directory Server,N_Port at FC DID 0xfffffe - SCSI tgt id -1 :portname 2004-0060-6900-5A1Bnodename 1000-0060-6900-5A1BPresent, Logged in, F_PORT,

Tru64 Unix connectivity

The output on this slide is enlarged below and descriptions are provided on the next page.

#emxmgr -t emx2emx2 state information:Link : connection is UPPoint to PointFabric attachedFC DID 0x210413 (tells us the HBA is plugged into switch 1 port 4)Link is SCSI bus 3 (this adapter was assigned scsi bus 3)SCSI target id 7portname is 1000-0000-C921-07C4nodename is 1000-0000-C921-07C4N_Port at FC DID 0x210013 - SCSI tgt id 5 :portname 5000-1FE1-0001-8932 (HBA driver assigned SCSI ID 5 to the bottom right controller port –

assuming MB)nodename 5000-1FE1-0001-8930Present, Logged in, FCP Target, FCP Logged in,N_Port at FC DID 0xfffffc - SCSI tgt id -1 :portname 20FC-0060-6900-5A1Bnodename 1000-0060-6900-5A1BPresent, Logged in, Directory Server,N_Port at FC DID 0xfffffe - SCSI tgt id -1 :portname 2004-0060-6900-5A1Bnodename 1000-0060-6900-5A1BPresent, Logged in, F_PORT,


Rev. 4.21 6

1. Emx1 link statusThe connection is a point-to-point fabric (switch) connection, and the link is up. The adapter is on SCSI bus 3 at SCSI ID 7. Both the port name and node name of the adapter (WWN) are provided. The Fibre Channel DID number is the physical Fibre Channel address being used by the N_Port.

2. List of all other Fibre Channel devices on this SCSI bus, with their SCSI ID, port name, node name, physical Fibre Channel address, and other items such as:

– Present — The adapter indicates that this N_Port is present on the fabric.– Logged in — The adapter and remote N_Port have exchanged initialization

parameters and have an open channel for communications (nonprotocol-specific communications).

– FCP target — This N_Port acts as a SCSI target device (it receives SCSI commands).

– FCP Logged in — The adapter and remote N_Port have exchanged FCP-specific initialization parameters and have an open channel for communications (Fibre Channel protocol-specific communications).

– Logged Out — The adapter and remote N_Port do not have an open channel for communication.

– FCP Initiator — The remote N_Port acts as a SCSI initiator device (it sends SCSI commands).

– FCP Suspended — The driver has invoked a temporary suspension on SCSI traffic to the N_Port while it resolves a change in connectivity.

– F_PORT — The fabric connection (F_Port) allowing the adapter to send FibreChannel traffic into the fabric.

– Directory Server — The N_Port is the FC entity queried to determine who is present on the Fibre Channel fabric.

3. A target ID of -1 (or -2) shows up for remote Fibre Channel devices that do not communicate using Fibre Channel protocol, the directory server, and F_Port.


Rev. 4.21 7


HP-UX connectivity

• Issue the following command to initiate HP-UX to search for installed HBA’s:# ioscan –f

– Available switches• -f (full output)• -n (show device node information)• -k (scans kernel instead of I/O path – this means you will not see

any new devices)• -C (scan devices in a specific class, ex.-C disk)• -d td (restrict output to only ‘td’ devices)

ioscanIoscan scans system hardware, usable I/O system devices, or kernel I/O system data structures as appropriate, and lists the results. For each hardware module on the system, ioscan displays by default the hardware path to the hardware module, the class of the hardware module, and a brief description. By default, ioscan scans the system and lists all reportable hardware found. The types of hardware reported include processors, memory, interface cards and I/O devices. Scanning the hardware may cause drivers to be unbound and others bound in their place in order to match actual system hardware. Entities that cannot be scanned are not listed.


Rev. 4.21 8


Verify HBA installation

>>>ioscan -f

Class I H/W Path Driver S/W State H/W Type Description

fc 3 1/10/0/0 fcd CLAIMED INTERFACE HP 2Gb Dual Port

fc 4 1/10/0/1 fcd CLAIMED INTERFACE HP 2Gb Dual Port

If the ioscan output is similar to the following, HP-UX detected the adapter, but the drivers are not properly loaded.


fc 0 8.0 fcT1 UNCLAIMED UNKNOWN

HP Fibre Channel Mass Storage installation is verified if the ioscan output lists all mass storage devices attached to the adapter. Verify that all devices you attached to the Fibre Channel adapter are listed in the ioscan output.


Rev. 4.21 9


HP-UX ioscan utility output descriptions (1 of 4)Class I H/W Path Driver S/W State H/W Type Description==================================================================disk 333 0/2/0/0.38.5.255.0.0.1 sdisk CLAIMED DEVICE DEC HSG80

/dev/dsk/c95t0d1 /dev/rdsk/c95t0d1

Field Description

Class Device category such as disk, tape, or printer

I Instance number associated with the device or card. It is a unique number assigned to a card or device within a class. If no driver is available for the hardware component or an error occurs binding the driver, the kernel will not assign an instance number and a (-1), is listed.


Rev. 4.21 10


HP-UX ioscan utilityoutput descriptions (2 of 4)

Field DescriptionH/W path A decimal numerical string of hardware components, notated sequentially from

the bus address to the deviceHBA path.domain id.port.255.bus.target.lun• HBA Path – Internal hardware path to the bus converter and HBA• Domain ID – Domain ID of the switch used to get to this storage• Port – Port of the switch used to get to this storage• 255 – Always set to 255 for fabric connected array controller• Bus –Dependent on addressing mode(PDF,LU,VSA)• Target –Dependent on addressing mode(PDF,LU,VSA)• LUN – LUN number of device as seen by HP-UX

Class I H/W Path Driver S/W State H/W Type Description==================================================================disk 333 0/2/0/0.38.5.255.0.0.1 sdisk CLAIMED DEVICE DEC HSG80


0/2/0/0 38.5.255.0.0.1 would indicate Switch Domain = 38Port on Switch = 5HP-UX sees every array controller as 255. Followed by a second line of information detailing the devices on the array :0/2/0/0 38.5.0.0.0.1 would indicate FABRIC DIRECT ATTACHED LUN 10/2/0/0 38.5.255.1.7.1 would indicate PERIPHERAL DEVICE ADDRESSING AL_PA 17h LOOP-ID 119d


Rev. 4.21 11


HP-UX ioscan utility output descriptions (3 of 4)

Field DescriptionSoftware state

The result of software bindingCLAIMED – software bound successfullyUNCLAIMED – No associated software foundDIFF_HW – Software found does not match the associated softwareNO_HW – Hardware at this address is no longer respondingERROR – Hardware at this address is responding but is in an error stateSCAN – Node locked, try again later




Rev. 4.21 12


HP-UX ioscan utility output descriptions (4 of 4)

Field DescriptionHardware type

Entity identifier for the hardware componentUNKNOWN – No hardware associated or the type of hardware is unknownPROCESSOR – Hardware component is a processorMEMORY – Hardware component is memoryBUS_NEXUS – Hardware component is bus converter or bus adapterINTERFACE – Hardware component is an interface cardDEVICE – Hardware component is a device bus type. Bus type associated with the node.

Description Description of the device




Rev. 4.21 13


Windows connectivity

The Windows driver is the device driver installation file (called an INF file) needed by the Windows Operating System. The storage controller needs no driver in reality, as the HBA is in the Host server. But this file lets the Windows Device Manager register the controller's Controller LUN as a "System" device, so that the Device Manager thereafter will not consider the controller LUN to be an unknown or "newly discovered" device with every reboot. Using this file, a User only has to "identify the controller to the Device Manager" once. To install (register) the controller, use the included INF file. A controller LUN must also set up on the controller so that Windows can "discover" it.When the controller FC link is up, the user can either reboot the PC, or run the "Scan for new Hardware" function of the Windows Device Manager. Either action should cause the HBA to issue a SCSI Inquiry command, to which the controller replies with its ASCII Inquiry string. Initially, the Windows Hardware Wizard will use this string to refer to the controller. After this discovery interaction occurs, the Hardware Wizard will prompt the user to install a device driver. The user should then select the Wizard's "Search for a suitable driver" option, and specify the folder containing the controller INF file. The Hardware Wizard scans all the INF files in the specified folder, and selects the first INF file it finds with a device entry containing a matching hardware ID string. It then copies the selected INF file, renaming it to "oem<#>.inf," where the '#' is some integer, and places the copy into the "C:\WINNT\inf" folder. It "compiles" the INF file to a ".PNF" file with the same root filename, and uses its controller model entry information to install -- or register in the controller's case -- the newly discovered device.


Rev. 4.21 14


LPUTILNT

Note: Use this utility to view settings only

The LightPulse Utility/NT, LPUTILNT, is a component of the Emulex HBA driver kit and is installed during the HBA driver installation. Use this utility to update firmware, BIOS, view registry parameters, perform persistent binding operations on selected targets, and obtain specific information about all Emulex HBAs installed in the server.The data display lists all available device driver parameters, along with the current, minimum, maximum, and default values. Parameters that have their value specified in the system registry are denoted with either a G or an L in the left-most column of the screen. The G indicates that the value is set in the global registry entry, which applies to all HBAsthat do not have a local registry entry. The L indicates that the value is set in a registry entry specific to the selected HBA, which overrides the value settings in the global entry.


Rev. 4.21 15


OpenVMS device discovery

• Issue the following command to have OpenVMS search for newly added virtual disks:$mc sysman io auto/log

• Issue the following commands to initialize and mount a virtual disk:$init $1$dga1002: test$mount $1$dga1002: test

• Issue the following command to display storage devices– show device d

• Devices are named using the following convention:$1$DGAidentifier where D is a disk device, G is a fibre channel

device


Rev. 4.21 16


OpenVMS multipath

• Multipath capability is integrated into O/S• Allows failover between device paths• Mount verify invoked when path is switched• Failover between controllers takes 1-15 seconds• Failover between controller ports instantaneous• Multipath set is created when two or more paths to a single

device are discovered• Each path name is constructed from the Fibre Channel port

name and the controller port WWIDExample

PGA0.5000-1FE1-0000-9631 or

PGB0.5000-1FE1-0000-9633

If a device is mounted as foreign the device will not failover to an alternate path.


Rev. 4.21 17


OpenVMS multipath discovery

• When VMS discovers a disk with a valid identifier it creates a normal disk UCB.

• When it finds the second path to the same disk it converts the unit into a multipath device.– VMS only uses one active path to a device at a time.

• As additional paths are discovered these paths are added to the multipath device.

• Disks that have only had one path do not show up with a SHOW DEVICE/MULTIPATH command.


Rev. 4.21 18


Viewing device details in OpenVMS

• SHOW DEVICE/FULL devname– Shows all valid paths to device

$ sho dev/full dga1001

Disk $1$DGA1001: (VMS24), device type HSG80, is online, file-oriented device,shareable, available to cluster, device has multiple I/O paths, errorlogging is enabled.

Error count 0 Operations completed 184Owner process "" Owner UIC [SYSTEM]Owner process ID 00000000 Dev Prot S:RWPL,O:RWPL,G:R,WReference count 0 Default buffer size 512Allocation class 1

I/O paths to device 4Path PGA0.5000-1FE1-000B-B283 (VMS24), primary path, current path.Error count 0 Operations completed 46

Path PGA0.5000-1FE1-000B-B282 (VMS24).Error count 0 Operations completed 46

Path PGB0.5000-1FE1-000B-B281 (VMS24).Error count 0 Operations completed 46

Path PGB0.5000-1FE1-000B-B284 (VMS24).Error count 0 Operations completed 46


Rev. 4.21 19


ioscan device discovery

• Issue the following command to initiate HP-UX to search for newly added virtual disks:# ioscan –fnCdisk

– Switches• -f (full output)• -n (show device node information)• -k (scans kernel instead of I/O path – this means you will not see

any new devices)• -C (scan devices in a specific class, ex.-C disk)• -d td (restrict output to only ‘td’ devices)

ioscanIoscan scans system hardware, usable I/O system devices, or kernel I/O system data structures as appropriate, and lists the results. For each hardware module on the system, ioscan displays by default the hardware path to the hardware module, the class of the hardware module, and a brief description. By default, ioscan scans the system and lists all reportable hardware found. The types of hardware reported include processors, memory, interface cards and I/O devices. Scanning the hardware may cause drivers to be unbound and others bound in their place in order to match actual system hardware. Entities that cannot be scanned are not listed.


Rev. 4.21 20


HP-UX device naming

HP-UX device naming• cxtydz

x = Adapter numbert = SCSI target IDd = SCSI LUN number

• Block device /dev/dsk/cxtydz

• Raw device/dev/rdsk/cxtydz

Devices can be classified in two categories, raw and block. A raw or character-mode device, such as a line printer, transfers data in an unbuffered stream and uses a character device special file. Block devices, as the name implies, transfer data in blocks by means of the system's normal buffering mechanism. Block devices use block device special files and may have a character device interface too.

When creating device special files, it is recommended that the following standard naming convention be used:

/dev/prefix/devspec[options]

prefix indicates the subdirectory for the device class (for example, rdsk for raw device special files for disks, dsk for block device special files for disks, rmt for raw tape devices).

devspec indicates hardware path information and is typically in the format c#t#d# as follows:

c# Instance number assigned by the operating system to the interface card. There is no direct correlation between instance number andphysical slot number.

t# Target address on a remote bus (for example, SCSI or HP-IB address).

d# Device unit number at the target address (for example, SCSI LUN).

options Further qualifiers, such as disk section s# (for backward compatibility), tape density selection for a tape device, or surface specification for magneto-optical media.


Rev. 4.21 21


HP-UX ioscan utility output

# ioscan -fnCdisk | moreClass I H/W Path Driver S/W State H/W Type Description======================================================================disk 1 0/0/1/1.15.0 sdisk CLAIMED DEVICE COMPAQ BD018122C0

/dev/dsk/c1t15d0 /dev/rdsk/c1t15d0disk 333 0/2/0/0.38.5.255.0.0.1 sdisk CLAIMED DEVICE DEC HSG80





This output was derived from a Hewlett Packard 9000 series A400 system.


Rev. 4.21 22


Device discovery for Tru64 UNIX V5.x (1 of 2)• When Tru64 UNIX V5.x boots, it probes the hardware

starting with bus 0, target 0.– All devices found are assigned (in order) the next available

disk instance number.• All instance numbers are maintained in a hardware

database so if the device’s BTL changes, the instance number is not affected.– Each virtual disk is assigned a unique hardware identifier

(HWID).• When Tru64 UNIX V5.x starts, any device which no longer

has any valid paths to it are still referenced in the hardware database– These devices show up with all paths showing ‘stale’, but their

device special files are retained.


Rev. 4.21 23


Device discovery for Tru64 UNIX V5.x(2 of 2)• Issue the following command to have Tru64 UNIX scan for

any newly added devices: # /sbin/hwmgr -scan scsi

– Newly discovered disks are assigned the next available disk instance number.

– Devices that have been removed (no valid paths) are marked offline/stale, but their device special files are retained.

• Issue the following command to create the device special files for any new devices:# dsfmgr –K


Rev. 4.21 24


Device naming for Tru64 UNIX V5.x (1 of 2)

• Tru64 UNIX V5.x uses the following naming convention that greatly expands the number of devices possible.

– Device special files for disks start with “dsk” ‘x’ ‘y’.– Device special files for CCLs start with “scp” ‘x’ ‘y’

Where: ‘x’ is the disk instance‘y’ is the partition

• Raw and block special device files are put in different directories.

– Block device special files are located in /dev/disk. – Raw device special files are located in /dev/rdisk.– Example: /dev/disk/dsk56c

• The BTL of the device cannot be determined just by viewing the device special file.


Rev. 4.21 25


• To determine the BTL of a device, use the hwmgr utility:# hwmgr -show scsi -full -id 62

SCSI DEVICE DEVICE DRIVER NUM DEVICE FIRSTHWID: DEVICEID HOSTNAME TYPE SUBTYPE OWNER PATH FILE VALID

PATH-------------------------------------------------------------------------

62: 8 stgaxp12 disk none 2 4 dsk56 [5/0/4]WWID:01000010:6000-1fe1-0003-8610-0009-9471-2165-0687

BUS TARGET LUN PATH STATE--------------------------------------------------4 0 4 valid4 1 4 valid5 0 4 valid5 1 4 valid

• Using Tru64 V5.x, the device special file name remains the same even if the device’s BTL is changed.

– Why? = The device is recognized by its 128 bit WWID, not by its BTL location.

Device naming for Tru64 UNIX V5.x (2 of 2)


Rev. 4.21 26


Windows disk manager

After completing the installation of both the HBA driver and the solutions software (requires reboot), the server sees the storage presented to it.Open disk manager, and the server scans for all disks.When new disks are discovered, disk manager starts the New Disk Signature wizard. This wizard writes a signature and by default promotes the disk to a dynamic disk. To write a signature, right-click on the disk.


Rev. 4.21 27

Learning check

Learning Check1. What utility do you use to scan for storage devices in HP-UX?

…………………………………………………………………………………….

…………………………………………………………………………………….

2. What command do you use to search for new storage units in OVMS?…………………………………………………………………………………….…………………………………………………………………………………….


Rev. 4.21 28

Introduction to SANs Secure Path

Rev. 4.21 1


Secure Path

Module 10


Rev. 4.21 2


Objectives

After completing this module, the student will be able to:• State the function of Secure Path components• Describe how profiles are created and implemented• Discuss device states• Describe path definitions• Explain the installation process• Identify common problems/troubleshooting


Rev. 4.21 3


Definition

Secure Path is:A multi-path component used in high availabilityand fault tolerant solutions. • Maintains connections to storage• Monitors I/O paths and alerts on significant events• Assists in load balancing

Used in…• Stand Alone Configurations• Cluster configurations• Disaster Tolerant Solutions• SAN configurations

HP Secure Path provides a fault-tolerant solution for continuous availability of RAID storage systems. It allows multi-controller HP StorageWorks RAID Array systems to be cabled on independent interconnects, using multiple host bus adapters in single-server or clustered environments.When combined with the inherent fault-tolerant features of the RAID Array, Secure Path eliminates single points of failure, such as disk drives, controllers, interconnect cables, hubs, switches, and host adapters in the storage system. Secure Path also automatically detects path failures and reroutes I/O to a functioning, alternate path.


Rev. 4.21 4


Features

• High availability: Manages up to 32 paths per virtual disk, allowing for no-single-point-of-failure in a SAN • Increased performance: Monitors and reroutes data with dynamic load balancing for bestuse of storage • Simple: Allows policies to be set with web-based manager and alerts you if changes occur • Manageable: Enables simple administration for Windows hosts with Secure Path Manager web GUI • Integrated: Offers multi-path support for a variety of storage arrays • Flexible: Supports storage consolidation, allowing large RAID sets to be shared among Windows hosts

Note: Features vary depending on operating system and storage system• differences in storage capabilities• differences in OS capabilities

There is an effort underway to equalize all features across platforms.


Rev. 4.21 5


Support

Operating systems that require Secure PathLinuxHP-UXIBM AIXMicrosoft WindowsNovell NetWareSUN Solaris

Note: Support for storage subsystems and operating systems continually changes. For an updated list of supported OS’s and storage devices, please see:http://h18006.www1.hp.com/products/sanworks/secure-path/index.html


Rev. 4.21 6


Secure Path for Microsoft Windows

Secure Path for Microsoft Windows


Rev. 4.21 7


Components

Drivers installed by Secure Path:Raidisk.syshpap.sysrdfil.sys

Software components:Hs_ServiceSecure Path AgentSPM CLI (spmgr)SPMSPEMPersistent Reservation Clear utilitySecure Path and Auto Path registry editing utilities

Microsoft Secure Path Drivers• Raidisk.sys is the Secure Path multipathing driver for Enterprise Virtual Array, Enterprise/Modular Storage RAID Array, and Modular SAN Array 1000. It integrates multipathed hardware to the Windows operating system, provides path management, failure detection, dynamic Load Balancing, and the reporting of path state information to the Secure Path Manager.• hpap.sys is the Secure Path multipathing driver that provides path failover and Load Balancing capabilities for VA and XP disk arrays.•rdfil.sys is a Windows filter driver that provides support for Secure Path with Microsoft clustered servers (MSCS). It also facillitates driver binding for EVA 3000 systems.


Rev. 4.21 8


Secure Path process

Secure Path processPath management is implemented with Secure Path Manager. Secure Path Manager is a client/server application that continuously monitors a multiple path storage environment and automatically updates the status of the displayed configuration. Secure Path Manager indicates which path is currently servicing each configured storage unit and displays the mode and state information for all available paths.Secure Path Manager communicates with the RaiDisk drivers through the Secure Path Agent. The Secure Path Agent runs as a Windows NT service, which is installed with RaiDisk on the host server. The Secure Path Agent communicates with the Secure Path Manager through the TCP/IP protocol and the Winsock API. To minimize local area network traffic, display information is relayed from the Secure Path Agent to the Secure Path Manager only when the RaiDisk driver detects configuration changes. The Secure Path Agent also notifies Secure Path clients when RaiDisk performs path failover or auto-failback operations.Each Secure Path software component uses the Windows event log facility to post error and informational messages as required.


Rev. 4.21 9


Profiles

ProfilesSecure Path provides the capability of managing large configurations through a single instance of Secure Path Manager. However, there are certain practical limits on the configuration size that can be displayed and managed in a single window. Secure Path Manager uses the concept of a “profile” to express this working configuration limit.Each storage system must be configured and connected in multiple-bus failover mode. The host systems may be standalone servers or grouped into clusters, as long as all servers in the profile have access to all of the storage systems listed in that profile.Secure Path provides the capability of managing large configurations through a single instance of Secure Path Manager. However, there are certain practical limits on the configuration size that can be displayed and managed in a single window. Secure Path Manager uses the concept of a “profile” to express this working configuration limit.Each storage system must be configured and connected in multiple-bus failover mode. The host systems may be standalone servers or grouped into clusters, as long as all servers in the profile have access to all of the storage systems listed in that profile.Secure Path Manager supports the creation of multiple profiles stored as separate files in the same directory in which it resides. .


Rev. 4.21 10


Configuration limits

• Secure Path Manager uses the profile concept for a working configuration limit. The Secure Path Manager limits are:

– A maximum of 128 servers (host systems)

– Connected to and sharing up to 128 storage systems

• Must be configured for multibus failover mode


Rev. 4.21 11


Load balancing

Dual Controller Storage subsystem

Controller A

Controller B

Active

Active

Server

Active

Active

FC SwitchFC

Switch

4321

1

1

2

2

LUN1 is online to Controller A LUN2 is online to Controller B

LUN2LUN1

HBA1=A2

HBA2=B1

HBA3=A1

HBA4=B2

LUN1 round robin between HBA1 & HBA3

LUN2 round robin between HBA2 & HBA4

With load distribution enabled, multiple paths between a host and a specific storageset can be used for parallel I/O. I/O intended for units connected to a specified controller is alternately dispatched through the set of appropriate paths, spreading the load across all ports to maximize performance. Load distribution requires a Fibre Channel SAN configuration that has at least four unique paths from the host node to the storage subsystem. While this can be accomplished using several different physical configurations, maximum performance potential is achieved when all four ports of the RAID storage subsystem are utilized. With load distribution enabled, the Secure Path driver marks all paths to the owning controller as Preferred by default. This is true when a host boots, when Secure Path fails over a storagesetfrom one controller to the other, or if a user manually moves a selected storageset between controllers using Secure Path Manager. The user can also modify the operational mode of individual paths to discrete storagesets.


Rev. 4.21 12


Device states

A device can exist in the following states:Degraded — At least one or more paths are failed to the storage unit.Working — The Secure Path device can be accessed on at least one path.Failed — Unable to communicate with the unit. This can indicate no available path or a failed device.


Rev. 4.21 13


Moving LUNs to other controller


Rev. 4.21 14


General installation tips

• Verify proper version of the operating system and that all necessary patches are installed.

• Ensure that storage is configured for multibus failover– If changing from transparent mode – follow the instructions

completely for setting this mode – including restarting the controllers.

• Configure storage/SAN in advance of the Secure Path installation.

• Be sure that you have installed the supported version of the Platform Kit before installing Secure Path.

• If you are uninstalling Secure Path, be sure to remove it beforeyou remove the Platform Kit.– Stop SP Agent, then uninstall SP

• If you cannot get a response from SP Manager, be sure that the agent is running.


Rev. 4.21 15


Windows installation tips

• To migrate your existing client profiles to Secure Path Manager V4.0. (for each V3.x client system):– Insert the Secure Path Manager V4.0 CD.– Choose Start > Run > AutoMenu.exe.– Run the Profile Migration Utility as described in the Secure

Path Manager Version 4.0 Installation Guide.• Verify that the new Secure Path Manager is working:

– At a client system, install the Java 2 Runtime Environment,1.3.1 or higher, if it does not already exist on thesystem.

– Browse to the host you upgraded. When you enter the URL, it must contain the host name appended with the :2301 suffix• For example, for an appliance with the system name

SWMAL1B435, enter:http://swmal1b435:2301


Rev. 4.21 16


Troubleshooting (1 of 3)

Secure Path Element Manager/Agent considerations• Add the NetBIOS host name, or fully qualified domain name

(FQDN) of each system that will run Secure Path Element Manager (SPEM) to the Agent's list of authorized clients. – Use the Agent Configuration utility, and set the password in

the Password Dialog box. • SPEM IP address must resolve to the host name entered in

the agent’s authorized client list


Rev. 4.21 17



• If the SPEM host system uses DHCP to obtain an IP address, the address resolution may fail due to non-dynamic DNS or WINS not being integrated with DNS.


Rev. 4.21 18



• Make sure that you use the same name type, either NetBIOS or FQDN, during Secure Path client login that you have entered in the agent's database.

• Make sure that the names you use can be mapped to the associated IP address using one of the following methods:– HOSTS file — Static text file with either NetBIOS or FQDN

mapped to IP– Windows Internet Naming Service — WINS with a NetBIOS

name– Domain Name System — DNS with an FQDN

• In cluster configurations, make sure that the password you choose is identical for all agents in the cluster.


Rev. 4.21 19


Secure Path for HP-UX


Rev. 4.21 20


Secure Path 3.0C HP-UX description

Secure Path V3.0C is a high availability software product which eliminates single points-of-failure such as:• HSX controllers• SAN switches• Fibre Channel cables• Host adapters


Rev. 4.21 21


Secure Path V3.0C support limits


Rev. 4.21 22


Secure Path 3.0C HP-UX switch support

• For complete switch support information refer to:http://h18000.www1.hp.com/products/storageworks/san/documentation.html

• SAN Director – 64-port 1-Gb 254512-B21 (DS-DMGGD-AA)

• These switches need their port speeds set correctly, and must not be set to auto-negotiate.


Rev. 4.21 23


Secure Path 3.0C HP-UX components(1 of 2)• swsp driver

– Failover driver is a pseudo-HBA driver– Present multiple paths as a single device to host SCSI driver

• hsx driver– Provides paths from HBA driver to swsp driver– Specifies commands to migrate a LUN

• spmgr– Utility used to monitor and manage devices, paths– /sbin/spmgr


Rev. 4.21 24


Secure Path 3.0C HP-UX components(2 of 2)• spagent

– Daemon that communicates with the multipath drivers– Path change notification through email– Not required to be running for full failover functionality– Must be running for spmgr commands to operate– /sbin/spagent

• spinit– Used to start/stop the spagent (/sbin/spagent)

• Start/sbin/init.d/spinit start

• Stop/sbin/init.d/spinit stop


Rev. 4.21 25


Secure Path 3.0C HP-UX driver model

The diagram on this slide is enlarged below.


Rev. 4.21 26


Secure Path 3.0C HP-UX IOSCAN (1 of 2)

# ioscan -fnCdisk


===========================================================================

disk 1 0/0/1/1.15.0 sdisk CLAIMED DEVICE COMPAQ BD018122C0


disk 301 0/2/0/0.38.5.0.0.0.1 sdisk CLAIMED DEVICE DEC HSG80








The output below depicts a single HSG80 LUN with four paths to it. This output was taken prior to Secure Path being installed.


Rev. 4.21 27


Secure Path 3.0C HP-UX IOSCAN (2 of 2)

Class I H/W Path Driver S/W State H/W Type Description======================================================================disk 1 0/0/1/1.15.0 sdisk CLAIMED DEVICE COMPAQ BD018122C0

/dev/dsk/c1t15d0 /dev/rdsk/c1t15d0disk 4 255/255/0/0.0 sdisk CLAIMED DEVICE HSG80


The output below depicts the same HSG80 LUN as described in the previous slide after Secure Path has been installed.


Rev. 4.21 28


Secure Path 3.0C HP-UX SCSI-2 CCL

• HP recommends that D0 is left unassigned if the CCL is enabled or that the CCL be left disabled in SCSI-2 mode of the HSG80s.

• In large ioscan outputs, one way to isolate Secure Path HSG80 LUNs is by using the following command:

#ioscan -fnCdisk | grep 255


Rev. 4.21 29


Secure Path 3.0C HP-UX spmgr commands(1 of 3)spmgr • add – add a new device• delete – remove a device• (un)alias – assign/remove an alias to an object• display – list secure path information• log – control logging• notify – manage email and event logging• quiesce – to remove an adapter or path• restart – return adapter or path to active or available• restore – restore one or move devices to preferred path• select – select a path for I/O• set – enable or disable special driver functionality


Rev. 4.21 30


Secure Path 3.0C HP-UX spmgr commands(2 of 3)# spmgr display -a[v] [adapter]

-c[v] [controller_serial_number]-d[v] [device]-p [path_instance]-r[v] [WWNN]-u (unattached units)

(no argument) example next page

v=verbose flag


Rev. 4.21 31


Secure Path 3.0C HP-UX spmgr commands(3 of 3)# spmgr displayServer: hp.mydomain.com Report Created: Thu, Sep 13 16:11:50 2001Command: spmgr display= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =Storage: 5000-1FE1-0010-5B00Load Balance: Off Auto-restore: OffPath Verify: On Verify Interval: 30HBAs: td0 td1Controller: ZG10505167, Operational

ZG10506981, OperationalDevices: c16t0d0 c16t0d1TGT/LUN Device WWLUN_ID H/W_Path #_Paths0/ 0 c16t0d0 6000-1FE1-0010-5B00-0009-1050-6981-0013 4

255/255/0/.0.0 Controller Path_Instance HBA Preferred? Path_StatusZG10505167 no

c4t0d0 td0 no Standbyc10t0d0 td1 no Standby

Controller Path_Instance HBA Preferred? Path_StatusZG10506981 YES

c5t0d0 td0 no Activec11t0d0 td1 no Available

The output on this slide is enlarged below.

Server: hp.mydomain.com Report Created: Thu, Sep 13 16:11:50 2001Command: spmgr display= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =Storage: 5000-1FE1-0010-5B00Load Balance: Off Auto-restore: OffPath Verify: On Verify Interval: 30HBAs: td0 td1Controller: ZG10505167, Operational

ZG10506981, OperationalDevices: c16t0d0 c16t0d1TGT/LUN Device WWLUN_ID H/W_Path #_Paths0/ 0 c16t0d0 6000-1FE1-0010-5B00-0009-1050-6981-0013 4

255/255/0/.0.0

Controller Path_Instance HBA Preferred? Path_StatusZG10505167 no

c4t0d0 td0 no Standbyc10t0d0 td1 no Standby

Controller Path_Instance HBA Preferred? Path_Status


Rev. 4.21 32


Secure Path 3.0C HP-UX dynamic LUNs

Secure Path 3.0C supports dynamic addition of LUNs

To add storage after installation:1. Present the units or virtual disks to the host system.2. Run ioscan -fnCdisk to add and attach the new storage.3. Run insf -e to install and assign device files.


Rev. 4.21 33


Secure Path 3.0C HP-UX troubleshooting(1 of 3)• Ensure hardware installation meets specs in reference and

release notes.• Ensure you are running a supported version of the

operating system with proper patches installed.• Ensure the storage subsystem has been set to multibus.• Completely configure all storage units before installing

Secure Path.• Ensure the server’s network is configured prior to

installation of Secure Path.


Rev. 4.21 34


Secure Path 3.0C HP-UX troubleshooting(2 of 3)• Stopping spagent using spinit stop and then starting

spagent using spinit start results in stderr messages being printed in that session. To keep messages from being printed, start spagent in a new session and then exit that session.


Rev. 4.21 35


Secure Path 3.0C HP-UX troubleshooting(3 of 3)• The Event Monitoring Service (EMS) logs erroneous

HSG80 LUN errors. At the time of these EMS notifications, the HSG80 devices do not have operating problems, and you can ignore the messages.– See the Secure Path V3.0C release notes for how to disable

HSG80 hardware monitoring (optional), or how to disable EVA hardware monitoring (required).

• Setting the Preferred path/mode to Path A –Failover/failback or Path B – Failover/failback is not supported with Secure Path on the EVA.

• On a server reboot, it does not matter if Auto-Restore is on or off, or if paths are preferred or not. The Active path comes up on the last path probed and not necessarily the preferred path. An spmgr restore all command must be issued to restore all active paths to their preferred paths.


Rev. 4.21 36


The Secure Path Manager

Using Secure Path ManagerSecure Path Manager (SPM) is an application that monitors and manages a Secure Path environment. SPM functions include:

• Displaying specific state information about RAID Array systems and I/O paths.

• Setting various properties and modes associated with a managed storage profile, such as determining path verification, dynamic load balancing, and auto failback.

• Automatically detecting and indicating path status.

• Moving RAID Array storagesets across controller pairs to facilitate static load balancing.


Rev. 4.21 37


Launching Secure Path Manager

Launching Secure Path ManagerYou can launch Secure Path Manager in either a standalone or Storage Management Appliance

environment.

Standalone Environment

To log into Secure Path Manager in a standalone environment:

• Enter the following address in your web browser:

http://yourhost:2301

• Access the Web management account:

• Choose Login Account and type your system access information into the Name and Password fields.

• Click Secure Path Management to Launch SPM.

Storage Management Appliance Environment

To log into Secure Path Manager in a Storage Management Appliance environment:

1. Log in to Open SAN Manager.

2. Expand the Applications folder in the Navigation pane.

3. Expand the Networking folder in the Navigation pane.

4. Click the SANworks Secure Path icon.

5. Click Launch in the Content pane.


Rev. 4.21 38


Profiles

ProfilesProfiles define a set of agent hosts and parameters that relate to a common storage system or group of storage systems. A profile includes:

• Agent Host Names• Password• Polling Interval • View type (for example: is the operating system Windows NT or Novell NetWare?)

Profile PasswordsWhen you set up a Secure Path profile, you must enter a Secure Path password. This password must match the password that was assigned to the Secure Path Agents when they were installed. SPM uses this password to establish a secure network connection with all Secure Path hosts included in the profile. For storage profiles that include more than one host, be sure to assign the same password to the profile that was assigned to each of the agents.When you save the profile, you can also save the password. If you save the password, you are not prompted to reenter the password each time you launch the profile. If the passwords do not match, the host displays a question mark icon and the communication status displays as validation failed.


Rev. 4.21 39


Profile views

Profile by SubsystemChoose View > Profile by SubsystemAll subsystems are displayed at the top level of the Navigation pane. Each system is displayed only once. The subsystem has a 64-bit value subsystem ID.Profile by HostChoose View > Profile By HostWhen selected, all hosts are displayed at the top level of the Navigation pane. All storage systems attached to a host are displayed beneath it. A storage system may display under multiple hosts.


Rev. 4.21 40


LUN views

LUNs byView LUNs by selections are displayed in the Navigation pane. Choose View > LUNs by and select any of the following:

• Default View - Displays the default view based on host type

• Disk Number - Displays the disk numbers assigned by the operating system

• Drive Letter - Displays the drive letters and mount points assigned by the operating system

• LUN UUID - Displays the LUN UUID

• Bus, Target, LUN - Displays the Bus number, Target ID and LUN ID

• Volume Label - Displays the system-supplied label

• Set Default Views - Sets the default view to be used by a specific host type

• Show Default Views - Displays the defined default views

• Note: Defaults are indicated by color:

• Black- Generated by the system

• Blue - Generated by the user

• Red - Unrecognized operating system. The view defaults to LUN UUID.


Rev. 4.21 41


Content pane information

Content paneThe Content pane contains detailed information about the controller, host, LUN, and storage subsystem components. The information displays in the Content pane when you click any of the Navigation pane icons or on the Host Selection buttons shown below the Quick Reference menu bar.


Rev. 4.21 42


Secure Path properties


Rev. 4.21 43


Path Operation

Path OperationSingle Host or Cluster Environment

Preferred Path rules:

• You can only add preferred paths to the LUNs on the same controller.• You may transfer a preferred path to another controller by moving a LUN or a group of LUNs to that controller.• You can take any alternate path offline.• You cannot take the last preferred active path offline.• When you return a path to the online state, it comes back in the last state it was in.

Load Balancing rules:

• Use one path to set up Load Balancing and then add preferred paths.• All preferred available paths become preferred active paths.• The last preferred path cannot be taken offline.

Cluster Only Paths• In a cluster environment, the owning host in the cluster has the preferred active path. • The Preferred available paths are on other non-owning cluster hosts.• If host ownership changes through a cluster-administrated move or a failover event, the preferred available path becomes the preferred active path.


Rev. 4.21 44


Alternate Web Access

Alternate Web AccessSecure Path Manager uses an Element Manager to communicate with Secure Path agents. You can configure a second Secure Path Element Manager (SPEM) on an alternate server for fault-tolerance. SPM allows failover to the alternate SPEM if communication is lost with the primary one. With Alternate Web Access enabled, whenever you save a profile in SPM, that new or updated profile is automatically copied to the alternate web server you have defined.Important: You must install the Secure Path Element Manager on an alternate server for fault-tolerant SPM operation.


Rev. 4.21 45

Learning check

Learning Check1. What components make up a Secure Path solution?

…………………………………………………………………………………….

…………………………………………………………………………………….

2. What is the purpose of a profile?…………………………………………………………………………………….

3. What is the maximum number of servers that a Secure Path Manager can manage?…………………………………………………………………………………….

4. What command do you use for Secure Path Manager for HP-UX?…………………………………………………………………………………….


Rev. 4.21 46

Complete Labs 9-12


Rev. 4.21 47

Introduction to SANs SAN Troubleshooting

Rev. 4.21 1


SAN Troubleshooting

Module 11


Rev. 4.21 2

Objectives

After completing this module, the student will be able to:• Understand how to trace SAN problems• Gather troubleshooting data• Use switch commands to gather information


Rev. 4.21 3

About Troubleshooting

IP

1 6 1 7 1 8 1 9 2 0 2 1 2 2 2 3 2 4 2 5 2 6 2 7 2 8 2 9 3 0 3 1

1 2 1 3 1 4 1 58 9 1 0 1 14 5 6 70 1 2 3


Troubleshooting should begin at the center of the SAN — the fabric. Because switches are located between the hosts and storage devices, and have visibility into both sides of the storage network; starting with them can help narrow the search path. After eliminating the possibility of a fault within the fabric, see if the problem is on the storage side or the host side, and continue a more detailed diagnosis from there. Using this approach can quickly pinpoint and isolate problems.For example, if a host cannot see a storage device, run a switch command to see if the storage device is logically connected to the switch. If not, focus first on the storage side. Use storage diagnostic tools to better understand why it is not visible to the switch. Once the storage can be seen from the switch, if the host still cannot see the storage device, then there is still a problem between the host and the switch.


Rev. 4.21 4

Tracing problems

What has changed?• Has anything new been added/removed/reconfigured?• When was it working last?

Analyze physical system setup• Devices attached to switch ports • Diagram of system configuration

Gather external switch information• Is the power LED on?• List the LED colors by port


Rev. 4.21 5

Analysis

Review log/tracesportLogShow/DumpsupportShowAnalyzer Traces

Hardware or

Software?

Hardware Software

Run diagnostics as necessary

Based on analysis of initial information:


Rev. 4.21 6

Common problem areas


Rev. 4.21 7

Troubleshooting tools


Rev. 4.21 8

Gather troubleshooting data

Whenever there is a potential problem with the behavior between the switch and a connected device, the following information should be collected:

• Finisar, Xyratec trace (FC Analyzer)-if available

• System configuration information• Switch - Type of switch, Switch serial

number, Firmware version, IP address• Host Bus Adapter (HBA) - Driver

version, Firmware version, Registry settings, Configuration load data

• Storage device - Firmware version (controller and disk), LUN configuration, Software applications on host and storage

• Switch trace files using supportShowcommand

• Routing outputs using uRouteShowcommand

• Front panel errors• Port log dump analysis

FC configuration• Which ports are used?• Which type of GBICs?• Chassis type and ID• Diagram of setup• Tests that have been run and results• Data patterns on traces• Document failure mode• Is failure easily reproducible?

Switch topology• Describe the network configuration• Is the switch cascaded?• How many switches are in the

configuration?• Are there parallel fabrics?• Dual host, storage connections?


Rev. 4.21 9

SupportShow command

SupportShow prints the switch’s information for debugging purposes and executes the following commands in the order shown:

• version• uptime• tempShow• psShow• licenseShow• diagShow• errDump• switchShow• portFlagsShow• portErrShow• mqShow• portSemShow• portShow

• portRegShow• portRouteShow• FabricShow• TopologyShow• qlShow• nsShow• nsAllShow• cfgShow• configShow• faultShow• traceShow• portLogDump


Rev. 4.21 10

errShow command

The following information is displayed for the errShow command:• Two errors which been detected (in this example)• The task ID and task name that incurred the error• The error type, date and time, the error level, and description• If there is more than one occurrence of an error type, the number

of occurrences is shown in brackets following the date and timestamp


Rev. 4.21 11

switchShow command

Switch DiagnosticsThe switch status can be either Healthy/OK, Marginal/Warning, or Down. The overall status of a switch is determined by the status of several individual components within the switch.Enter the switchshow command at the command line. This command displays the following information for a switch:

• switchname - Displays the switch name.• switchtype - Displays the switch model and firmware version numbers.• switchstate - Displays the switch state: Online, Offline, Testing, or Faulty.• switchrole - Displays the switch role: Principal, Subordinate, or Disabled.• switchdomain - Displays the switch Domain ID.• switchid - Displays the embedded port D_ID of the switch.• switchwwn - Displays the switch World Wide Name.• switchbeacon - Displays the switch beaconing state: either ON or OFF.

The switchshow command also displays the following information for ports on the specified switch:

• Module type - The GBIC type if a GBIC is present.• Port speed - The speed of the Port (1G, 2G, N1, N2, or AN). The speed can be fixed, negotiated, or auto negotiated.• Port state - The port status.• Comment - Displays information about the port. This section may be blank or display WWN for F_port or E_port, Trunking state, upstream or downstream status.


Rev. 4.21 12

portshow

Displaying Software Statistics for a PortSoftware statistics for a port include information such as port state, number of interrupts, number of link failures, number of loss of synchronization warnings, and number of loss of signal warnings.


Rev. 4.21 13

portstatsshow

Displaying Hardware Statistics for a PortHardware statistics for a port include information such as the number of frames received, the number of frames sent, the number of encoding errors received, and the number of class 2 and 3 frames received.


Rev. 4.21 14

Hardware diagnostics

Hardware monitoring commands:fanshow

psshow

tempshow


Rev. 4.21 15

Helpful Commands 1 of 3

Error Log errDump Displays the error log without page breaks.

Switch Offline switchDisable Sets the switch to offline state necessary to run certain switch diagnostics.

Memory Test ramTest Checks CPU RAM memory. Run offline or online.

Port Register Test portRegTest Checks that the registers and static memory in each ASIC can be successfully accessed.Run offline.

Central Memory Test centralMemoryTest Checks that the central memory in each ASIC can be successfully accessed. Run offline.


Rev. 4.21 16


Control Message Interface (CMI) Conn Test

cmiTest Verifies that control messages can be sent from ASIC to ASIC. Run offline.

Content Addressable Memory (CAM) Test

camTest Verifies CAM functionality. Run offline.

Error Log errDump Displays error log without page breaks.

Port Loopback Test portLoopbackTest Checks all switch main board\ hardware. Frames transmitted are looped back and received. Run offline


Rev. 4.21 17


Cross Port Test crossPortTest Checks all switch paths. Run offline or online.

Spin Silk Test spinSilk Checks all switch paths. Run offline.

SRAM Data Retention Test

sramRetentionTest Verifies that data written into ASIC memories is retained. Runs offline.

CMem Data Retention Test

CmemRetentionTest Verifies that data written into ASIC SRAMs is retained. Runs offline.


Rev. 4.21 18

Interpreting LED Activity

Note: Other switch models have similar capabilities

Interpreting LED ActivitySAN Switch 2/16 status is determined through LED activity. The LEDs will flash green, yellow, or orange while the switch is booting and while POST or other diagnostic tests are running. This is normal, and does not indicate a problem.Front Panel LEDsThe front panel includes the following LEDs:

•One Switch Power LED on the front panel•One Port Status LED above and to the left of each of the 16 ports•One Port Speed LED above and to the left of each of the 16 port


Rev. 4.21 19


Rev. 4.21 20


Rev. 4.21 21

Rear Panel LEDs

Rear Panel LEDsThe rear panel includes the following LEDs:

•One power supply LED for each power supply•One Port Readiness LED


Rev. 4.21 22

Learning check

Learning Check1. When gathering troubleshooting data, what specific tools or commands can you use?

……………………………………………………………………………………………………………………………………………………………………………………

2. What command do you use to gather information about the switch temperatures?……………………………………………………………………………………………………………………………………………………………………………………

3. If you had connectivity issues within your SAN, what would you check first?……………………………………………………………………………………………………………………………………………………………………………………


Rev. 4.21 23

Introduction to SANs Fibre Channel Switches

Rev. 4.21 1


Fibre Channel Switches

Module 12


Rev. 4.21 2


Supported Fibre Channel switches

Enterprise

Mid-range

Entry level

B-Series M-Series C-Series

core 2/64

switch 2/32

switch 2/8 EL

switch 2/16

director 2/64

director 2/140

edge 2/32

edge 2/24

MDS 9509

MDS-9216

HP branded Cisco branded

MSA SAN Switch 2/8

SAN director 128

edge 2/12

MDS 9120

MDS 9140

MDS 9506

Supported switches as of May, 2004.

B-series — Manufactured by BrocadeM-series — Manufactured by McDataC-series — Manufactured by Cisco


Rev. 4.21 3


Brocade SilkWorm 3800

StorageWorks SAN Switch 2/16 by Compaq DS-DSGGD-AA (240602-B21) A7340A StorageWorks SAN Switch 2/16 by Compaq DS-DSGGD-BA (287055-B21)(Replacement for end-of-life DS-DSGGD-AA)HP StorageWorks SAN Switch 2/16 B-Series Family DS-DSGGD-CA (322118-B21) (Replacement for end-of-life DS-DSGGD-AA and DS-DSGGD-BA)HP StorageWorks SAN Switch 2/16 PowerPak B-Series FamilyDS-DSGGD-CB (322119-B21)

System Architecture - 16 2-Gbit/second non-blocking Fibre Channel portsNumber of Fibre Channel Ports - 16 universal SFP (Small Form factor Pluggable) portsPort Types - FL_Port, F_Port, and E_Port; self-discovery based on switch type (U_Port)Data Traffic Types - Unicast, multicast (256 groups), and broadcastPerformance - 2.125 Gbit/sec line speed, full duplexMaximum Frame Size - 2112 bytes per frameFabric Latency - Less than 2 microseconds with no contention, cut-through routingData Transmission Range - Up to 300m (~975 ft) for short-wavelength optical link at 2 GbpsTopology - Fabric Crossbar SwitchFabric Services

•Simple Name Server, Registered State Change Notification (RSN), Alias Server (multicast), Zoning, WEB TOOLS, Optional QuickLoop, Fabric Watch, Extended Fabrics, Remote Switch

Supported Software•Telnet, SNMP, WEBTOOLS, Zoning, Fabric Watch, Extended Fabrics, Remote Switch

Management Access - 10/100 Ethernet (RJ-45), serial portDiagnostics - POST and embedded online/offline diagnostics


Rev. 4.21 4


McDATA Sphereon 3216

HP StorageWorks Edge Switch 2/16DS-DMGGE-BB (286811-B21) A7284A

System Architecture - 16 2-Gbit/second non-blocking Fibre Channel connectionsNumber of Fibre Channel Ports - 16Port Types - G_portMedia Type - Hot plug, industry-standard LC Small Form FactorPort Speed - 1.0625 – 2.125 Gb/sec, full duplexSwitch Bandwidth - 64 Gbps end-to-endMaximum Frame Size - 2112 bytes per frameFabric Latency - Less than 2 microseconds averageTopology - Fabric Crossbar SwitchFabric Services - Simple Name Server, In-order delivery (Class 2, 3), Management Server (optional), Broadcast, Name Server Zoning Management Options - HP StorageWorks HA-Fabric Manager, Command Line Interface, Embedded Web Server, SNMP Management Access - In-band, Ethernet (10/100 Mbps) Availability Features - Hot plug power supplies and fans, Hot plug optics, Hot load firmware upgrades, Call-home, email, Maintenance port (DSUB), Thermal protection, Unit, port, FRU beaconing, System error LED, FRU failed LED Diagnostics - Power-on self test (POST), On-line port, CTP, SBAR, Internal and external loopback, On-line system health


Rev. 4.21 5



HP StorageWorks Edge Switch 2/32 M-Series FamilyDS-DMGGE-BC (286810-B21) A7283A

System Architecture - 32 2-Gbit/second non-blocking Fibre Channel connectionsNumber of Fibre Channel Ports - 32Port Types - G_portMedia Type - Hot plug, industry-standard LC Small Form FactorPort Speed - 1.0625 – 2.125 Gb/sec, full duplexSwitch Bandwidth - 128 Gbps end-to-endMaximum Frame Size - 2112 bytes per frameFabric Latency - Less than 2 microseconds averageTopology - Fabric Crossbar SwitchFabric Services - Simple Name Server, In-order delivery (Class 2, 3), Management Server (optional), Broadcast, Name Server Zoning Management Options - HP StorageWorks HA-Fabric Manager, Command Line Interface, Embedded Web Server, SNMP Management Access - In-band, Ethernet (10/100 Mbps) Availability Features - Hot plug power supplies and fans, Hot plug optics, Hot load firmware upgrades, Call-home, email, Maintenance port (DSUB), Thermal protection, Unit, port, FRU beaconing, System error LED, FRU failed LED Diagnostics - Power-on self test (POST), On-line port, CTP, SBAR, Internal and externalloopback, On-line system health


Rev. 4.21 6



HP StorageWorks Edge Switch 2/24 M-Series FamilyDS-DMGGE-BD (316095-B21)

System Architecture - 24 2-Gbit/second non-blocking Fibre Channel connectionsNumber of Fibre Channel Ports - 24 universal SFP portsPort Speed - 1.0625 Gb/s, full duplex, 2.125 Gb/s, full duplex Switch Latency - Less than 1 microsecond averageAggregate Bandwidth - 96 Gb/sMedia Type - Hot-plug, industry standard LC Small Form Factor (SFP)Fabric Services –SNS, In order delivery (Class 2,3), Management Server (optional), Broadcast, Name server zoning Management Access - In-band over Fibre Channel, 10/100Mb/s Ethernet (RJ-45) Diagnostics - Power on self-test (POST), On-line port, Internal & external loopback, On-line system health


Rev. 4.21 7


Cisco MDS9120

Cisco MDS 9120 Multilayer Fabric Switch C-Series SwitchCisco MDS 9120 (346700-B21)

Maximum Number of Devices/Ports• 20 Fibre Channel ports (9120) • 1 10/100 Mb Ethernet port Port Speed - 1 or 2 Gb/sec auto-sensing /Fibre ChannelDiagnostics and Troubleshooting Tools• Fibre Channel ping and trace route • SPAN • Protocol analysis and decoding • Zone and VSAN merge analysis • Integrated Call Home capability.


Rev. 4.21 8


Cisco MDS9216

Cisco MDS 9216 Multilayer Fabric SwitchCisco MDS 9216 (332315-B21)

Port Speed - 1 or 2 Gigabit/sec auto-sensing /FC portPorts per Chassis -•16 to 48 Auto-Sensing 2 / 1-Gb Auto-Sensing FibreChannel ports•Upto 8 1-Gb Ethernet ports (user configurable foriSCSI or FCIP)Port Types -•1 / 2 Gigabit FC ports•10/100 Mb Ethernet port (management)•RS-232 RJ-45 console port•DB-9 COM portSupported Protocols -•Fibre Channel (auto-sensing 1 / 2 Gigabit)•iSCSI•FCIPDiagnostics and Troubleshooting Tools -•SPAN•FC Traceroute•FC Ping•Call Home•Embedded Fibre Channel protocol analyzer•Zone and VSAN merge analysis•System health monitoringIP Storage Services Module Capable - Yes


Rev. 4.21 9

Complete Lab 13


Rev. 4.21 10

Documents

Basic San