Brocade ISL Trunking

1

Exploring Brocade ISL Trunking

SOLUTIONware ----Rev1 Tech Note Last modified: January 2002

This document provides an in-depth examination of trunking and how trunking impacts the design, implementation, and maintenance of a SAN.

Category: SOLUTIONware Technical Note Sub-category: SAN Design Key words: ISL trunking, SAN Design

53-0000263-01

Page 2 of 26

Table of Contents

Table of Contents..........................................................................................................................................................2 Introduction ..................................................................................................................................................................3 Trunking Behavior .......................................................................................................................................................3

Virtual Channels......................................................................................................................................................6 High Availability and Dynamic Bandwidth Allocation ...............................................................................................7

Adding An ISL.........................................................................................................................................................7 An ISL Failure or Removal ....................................................................................................................................8

Performance................................................................................................................................................................12 Monitoring Performance.......................................................................................................................................12 ISL Performance....................................................................................................................................................12 Trunking Maximizes Performance and Minimizes The SAN Management Effort .........................................13

How Does Trunking Impact SAN Design..................................................................................................................15 Tip: Place Trunking-Capable Switches Adjacent To Each other .....................................................................15 Tip: Place Trunking-Capable Switches In The Core .........................................................................................15 Tip: Leave Open Ports On The Quad..................................................................................................................16 Tip: If Two Trunkable Switches Are Connected By More than One ISL, Use Trunking...............................16

Trunking Requirements..............................................................................................................................................17 ISL Trunking Administration ....................................................................................................................................19 Summary .....................................................................................................................................................................21 Appendix A: Terms and Definitions..........................................................................................................................22 Appendix B: Helpful Trunking Commands..............................................................................................................24 Copyright.....................................................................................................................................................................25 Evaluation Form.........................................................................................................................................................26

53-0000263-01

Page 3 of 26

Introduction

This document provides an in-depth examination of trunking and how trunking impacts the design, implementation, and maintenance of a SAN. Currently the SilkWorm 3800 and 12000 switches support trunking.

Trunking is a feature that enables traffic to be optimally shared across available inter-switch links (ISLs) while preserving in-order delivery. A trunk group logically joins two, three, or four ISLs into one logical ISL. Use of trunking can minimize or eliminate congestion in the SAN because trunking optimizes ISL utilization. Additionally, the use of trunking minimizes the effort of managing a SAN since ISLs are now managed as a group instead of individually.

Trunking can also increase availability. As long as at least one ISL link remains, I/O continues if an ISL failure occurs -- albeit at a lower bandwidth. It is also possible to dynamically increase bandwidth by adding ISLs to a trunk -- without impacting I/O -- to enable up to 8Gb/s of bandwidth over a single logical link. For reference, a convenient list of trunking definitions and terms is provided at the end of this document in Appendix A.

Trunking Behavior

Due to the implementation of some Fibre Channel protocol devices, frame traffic between a source device and a destination device must be delivered in-order within an exchange. Out of order and in order delivery is shown in Figure 1. The host is sending a SCSI sequence in the order 0, 1, 2, 3. The in order delivery for the storage to receive this sequence is 0, 1, 2, 3 and any other order, such as 2, 3, 0, 1 is considered out of order.

Figure 1 In order and Out of Order Delivery

3 2 1 0

In OrderDelivery

3 2 1 0

SAN

2 3 0 1

Out OfOrder

Delivery

53-0000263-01

Page 4 of 26

This restriction forces the current Brocade frame routing method to fix a routing path within a fabric. As a consequence, certain traffic patterns may become unevenly distributed, leaving some paths congested and other paths underutilized. This situation is depicted in Figure 2. Hosts A and B require an aggregate of 300 MB/s of bandwidth while host C requires 10 MB/s of bandwidth (Note: The SilkWorm 3800 switch supports 2 Gb/s technology). Fabric Shortest Path First (FSPF) establishes routes in a round robin fashion while preserving in-order delivery. In Figure 2, hosts A & B, which require 150 MB/s each of bandwidth will share the left-hand ISL, while host C will utilize the right-hand ISL. Note that the aggregate bandwidth required is 310 MB/s. Hosts A & B will experience congestion, as only 200 MB/s is available from the utilized ISL and the bandwidth requirement is 300 MB/s. Host C will experience no congestion, as there is sufficient bandwidth to support the 10 MB/s requirement.

The two ISLs provide an aggregate bandwidth of 400 MB/s. If the routing initialization happened in a different order, there would not be any congestion. However, even if the route assignments were different, say host A and host C on one ISL and host B on the other ISL, it is still possible to encounter congestion due to load variations and fluctuations. For example, a tape backup might create enough of a spike in demand to cause congestion on an ISL that would otherwise be avoided if trunking were utilized.

It is possible to deterministically setup the routes of a fabric with static routes by using the uRouteConfig command and prevent the scenario depicted in Figure 2 from happening; however, doing so require some effort and doing so for a large SAN with numerous ISLs can become a very complex task. With trunking it is not necessary to setup static routes or manage individual ISLs to achieve optimal performance.

53-0000263-01

Page 5 of 26

Figure 2 Uneven distribution of traffic without trunking

h o s t A1 5 0 M B /s

h o s t B1 5 0 M B /s

h o s t C1 0 M B /s

h o s t Ah o s t B

h o s t C

A g g re g a te re q u ire d b a n d w id th o fh o s t A a n d h o s t B is 3 0 0 M B /s .A v a ila b le b a n d w id th ( le ft-h a n dIS L ) is 2 0 0 M B /s .

B a n d w id th D e fe c it = 1 0 0 M B /s

H o s t C o n ly re q u ire s 1 0 M B /s o fb a n d w id th . A v a ila b le b a n d w id th( r ig h t-h a n d IS L ) is 2 0 0 M B /s

B a n d w id th S u rp lu s = 1 9 0 M B /s

Figure 3 depicts a similar scenario as shown in the Figure 2 configuration. Notice that all variables are the same, except that trunking is utilized. The trunk is a logical aggregation of the two 200 MB/s ISLs into one logical 400 MB/s trunk. This means that we have gone from an unbalanced load situation with a deficit of 100 MB/s to a surplus bandwidth of 90MB/s.

Note that trunking allows the grouping of up to four 2 Gb/s ISLs into a maximum of one logical 8 Gb/s trunk. It is also possible to divide the four ISLs in a quad between two trunk groups, with two 2-ISL trunks (4 Gb/s each) – each trunk connected to a different switch. As shown in Figure 3, the trunk provides sufficient bandwidth for all traffic streams. Trunking is dynamic and responds in real-time to performance issues.

53-0000263-01

Page 6 of 26

Figure 3 Performance and management benefits of trunking

h o s t A1 5 0 M B /s

h o s t B1 5 0 M B /s

h o s t C1 0 M B /s

A g g re g a te re q u ire d b a n d w id th o f h o s t A , h o s t B , & h o s t Cis 3 1 0 M B /s . A v a ila b le b a n d w id th is 4 0 0 M B /s .

B a n d w id th S u rp lu s = 9 0 M B /s

h o s t Ah o s t Bh o s t C

h o s t Ah o s t Bh o s t C

2 IS L T ru n k = 4 0 0 M B /s

Virtual Channels Brocade Virtual Channels is a unique technology that logically partitions physical bandwidth and prioritizes traffic within inter-switch links (ISLs). It is essential for enabling large fabrics by insuring the reliable operation of the network, improving the efficiency and performance of switch-to-switch communication, simplifying capacity planning and fabric design, and reducing the total cost of SAN ownership.

Brocade Virtual Channels has individual circuits for eight different channels with four different priority levels. Critical traffic for management communication and link control receive dedicated channels and the highest priorities. Data traffic can be sent across multiple channels with different priority levels.

53-0000263-01

Page 7 of 26

Brocade Virtual Channels is integral to Brocade ISL Trunking and is built into the switch ASIC, which is a part of every Brocade switch product. The underlying technology uses an anti-starvation algorithm that insures that all Brocade switches are non-blocking and it is used as the foundation for Brocade Extended Fabrics to communicate over long distances. The virtual channel technology logically partitions bandwidth within a single ISL and ISL Trunking logically aggregates bandwidth across up to four different ISLs. Together, they provide maximum network efficiency and they enable Brocade switches to scale cost effectively into large high performance reliable storage area networks.

High Availability and Dynamic Bandwidth Allocation

When adding ISLs to an existing trunk, there is no disruption of I/O and I/O is immediately spread across the existing ISLs and the new ISL. With trunking, the deletion or failure of an ISL to an existing trunk can impact the I/O traversing that trunk.

Adding An ISL Figure 4 depicts the configuration utilized to quantify the impact of an ISL addition to a trunk Essentially, the hosts and storage communicate via a 4 Gb/s trunk, which consists of two 2 Gb/s ISLs.

Figure 4 Configuration Utilized To Quantify Dynamic Bandwidth Allocation

3800

38003800

3800

4 x SilkWorm 3800 w/ v3.0.1 FabricOS6 x Sun Hosts w/ Solaris 5.8Emulex PCI LP8000 (1 Gb/s) & LP9000 (2Gb/s) w/ 4.2 driversClariion 4500 w/ 2 controllers4 x JMR JBODs w/ (8) Seagate ST336605FC disks

Sun Hosts CLARiiON4500 JMR JBODs

I/O

53-0000263-01

Page 8 of 26

In Figure 5, the portPerfShow command is executed while an I/O load is issued between hosts and storage. Notice how the aggregate bandwidth utilized, which is approximately 400 MB/s, now distributes over the additional ISL. Furthermore, the I/O consumption expands from 400 MB/s to 420 MB/s to utilize the additional bandwidth enabled by the ISL addition to the trunk. No interruption to I/O was noticeable.

The test involved the use of the UNIX dd utility issuing a maximal I/O load from the six Sun hosts to the Clariion and JMR storage. Ports 4 and 5 are the existing 4 Gb/s trunk. Port 6 was then enabled. Once port 6 was enabled, it immediately joined the trunk and I/O was immediately distributed.

Figure 5 I/O Immediately Spreads To The New ISL When An ISL Is Added To A trunk

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

--------------------------------------------------------------------------------

0 0 0 0 201m 201m 0 0 55m 91m 93m 55m 22m 22m 30m 30m

2.2k 2.0k 0 0 198m 199m 0 0 55m 91m 92m 55m 22m 21m 28m 29m

0 0 0 0 201m 201m 0 0 55m 91m 92m 55m 22m 22m 31m 31m

112 0 112 0 200m 201m 0 0 55m 91m 91m 55m 21m 22m 31m 31m

0 0 0 0 154m 154m 109m 0 55m 91m 94m 55m 30m 28m 30m 31m

0 0 0 0 140m 140m 140m 0 55m 91m 93m 55m 31m 31m 30m 30m

0 0 0 0 139m 139m 139m 0 55m 91m 93m 55m 30m 30m 30m 30m

0 0 0 0 140m 139m 139m 0 55m 92m 92m 55m 31m 30m 30m 31m

2 ISL Trunk 3rd ISL

Point where 3rd ISL is added

An ISL Failure or Removal When an ISL that is not a trunk master is removed or fails, there is minimal or no disruption of I/O and the I/O will be distributed over the remaining ISLs that form the trunk. When an ISL that is a trunk master is removed or fails, there will be a pause in I/O associated with that trunk as the I/O is re-routed and distributed over the remaining ISLs that form the trunk.

It is possible that upon failure or removal of the ISL that some frames were in flight and that these frames will be lost. When this happens, the behavior and impact to I/O is dependent upon the host, application, storage, and HBA utilized. The recovery period for lost frames needs to be accounted for before I/O can be expected to fully recover. If the master link happens to be an upstream or downstream port (see Appendix A: Terms and Definitions for more detail), then the time for a fabric reconfiguration also needs to be included.

53-0000263-01

Page 9 of 26

Figure 6 depicts the configuration utilized to quantify the impact of an ISL failure within a trunk. The test involves performing I/O between the host and the storage with the dd UNIX utility. Once the I/O begins, a trunk port is disabled (portDisable command) and the period until I/O resumes is measured. Specifically the portPerfShow command is monitored for a pause or decrease in I/O, at which point, the timer starts. Once I/O resumes, the timer stops.

When a trunking slave is disabled, there was no pause in I/O. The I/O rate decreases slightly and after approximately three seconds, the I/O rate returns to steady state. The same test was executed and instead of disabling a trunking slave, a trunking master was disabled. For this test, the average pause in I/O was twenty seconds.

Figure 6 Configuration Utilized To Quantify The Impact Of An ISL Failure Within A Trunk Upon I/O

3800

38003800

3800

4 x SilkWorm 3800 w/ v3.0.1 FabricOS6 x Sun Hosts w/ Solaris 5.8Emulex PCI LP8000 (1 Gb/s) & LP9000(2 Gb/s) w/ 4.2 driversClariion 4500 w/ 2 controllers4 x JMR JBODs w/ (8) Seagate ST336605FC disks


I/O

53-0000263-01

Page 10 of 26

Figure 7 displays a capture of portPerfShow data from a switch while I/O was running and a trunk slave ISL was disabled (portDisable). Note that I/O stops for port 6, which was the port that was disabled. The flow of data essentially continues with minimal impact; however, there is a pause while ports 12 and 14 get rerouted across the remaining ISLs that form the trunk.

Figure 7 PortPerfShow Data During A Trunk Slave Failure

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

--------------------------------------------------------------------------------

0 0 0 0 0 145m 144m 145m 30m 91m 93m 93m 27m 27m 35m 35m

0 0 0 0 0 143m 142m 143m 30m 91m 93m 93m 25m 25m 34m 35m

0 0 0 0 0 144m 143m 144m 30m 91m 91m 93m 27m 27m 35m 35m

0 0 0 0 0 162m 0 162m 29m 89m 91m 91m 15m 27m 19m 40m

0 0 0 0 0 185m 0 185m 26m 88m 90m 90m 0 26m 0 48m

0 0 0 0 0 186m 0 186m 29m 87m 90m 89m 0 27m 0 48m

0 0 0 0 0 193m 0 192m 27m 87m 89m 89m 8.0m 25m 14m 42m

0 0 0 0 0 202m 0 202m 28m 87m 87m 89m 21m 19m 34m 36m

0 0 0 0 0 207m 0 206m 29m 89m 91m 90m 19m 20m 35m 35m

0 0 36 76 0 209m 0 209m 28m 91m 92m 91m 21m 21m 35m 35m

0 0 0 0 0 207m 0 207m 25m 91m 93m 92m 21m 20m 34m 35m

Slave failure

Trunk

53-0000263-01

Page 11 of 26

The portPerfShow data shown in Figure 8 is taken during the failure of a trunk master ISL. Once the trunk master (port 5) fails by disabling the trunk master, I/O pauses for approximately twenty seconds before it resumes.

Figure 8 PortPerfShow Data During A Trunking Master Failure

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

--------------------------------------------------------------------------------

0 0 0 0 0 142m 141m 141m 30m 88m 90m 91m 27m 27m 35m 35m

0 0 0 0 0 143m 143m 143m 31m 89m 92m 90m 27m 27m 35m 35m

0 0 0 0 0 142m 141m 141m 30m 89m 92m 91m 25m 25m 34m 35m

0 0 0 0 0 0 94m 95m 20m 61m 63m 61m 18m 18m 21m 21m

680 40 936 220 0 0 824 1.1k 0 0 0 0 0 0 0 0

133k 133k 275k 259k 0 0 0 0 0 267k 267k 267k 0 0 84 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

data deleted to preserve space – essentially a 20 second pause when no I/O is active

0 0 0 0 0 0 1.4m 1.4m 2.8m 6.6k 6.6k 6.6k 1.1k 832 832 832

0 0 0 0 0 0 25m 25m 51m 13k 13k 13k 1.6k 1.6k 1.6k 1.6k

0 0 0 0 0 0 35m 35m 51m 18m 0 0 0 0 0 0

0 0 0 0 0 0 191m 190m 50m 91m 62m 85m 19m 17m 26m 28m

0 0 0 0 0 0 206m 206m 29m 90m 91m 91m 19m 19m 34m 36m

0 0 0 0 0 0 206m 206m 30m 88m 91m 91m 20m 20m 35m 35m

0 0 0 0 0 0 205m 205m 30m 88m 90m 90m 20m 20m 35m 35m

Master failure

Trunk

I/O Resumes

53-0000263-01

Page 12 of 26

Performance

The combination of 2 Gb/s technology and the capability of ISL trunking enable SilkWorm switches to deliver impressive performance. With duplex 2 Gb/s ISLs, the potential bandwidth is really 400 MB/s – one stream achieving 200 MB/s in one direction and another stream achieving 200 MB/s in the opposite direction. To realize the value of trunking, it is only necessary to have an I/O load that is capable of saturating a single ISL. Once this saturation occurs, it is possible to starve other devices – even if additional ISLs are available between the communicating devices. As discussed earlier in this document and as is detailed later in this section, trunking eliminates this problem.

Monitoring Performance Port statistics may not be the same across all participating ISLs within a trunk. The level of uniformity depends on the payload variations at the frame level. While optimal bandwidth utilization is the goal of the Brocade ISL Trunking feature, traffic might not be distributed exactly the same across all ISL links within a trunk.

ISL Performance In Figure 9, if Host A writes to Storage 1 at 200 MB/s and Host B reads from Storage 2 at 200 MB/s it is possible to generate 400 MB/s of I/O on a single ISL. This performance capability equates to 1600 MB/s (16Gb/s) of potential bandwidth for a four ISL trunk. Achieving such performance in a real life situation is unlikely. To do so requires perfectly balanced I/O and devices capable of generating sufficient levels of I/O.

Figure 9 2 Gb/s Duplex Performance

38003800

200 MB/s (write)

200 MB/s (read)

Host A

Host B Storage 2

Storage 1

all device connections and ISL are 2 Gb/s

53-0000263-01

Page 13 of 26

In a lab setting it is possible to approach maximal ISL bandwidth. Given the configuration shown in Figure 10 and using the dd UNIX utility, it was possible to generate over 350 MB/s on a single ISL. The limiting variable in this test was that there was not enough storage to generate a 400 MB/s I/O load.

Figure 10 Configuration Used To Achieve Maximal ISL Bandwidth

38003800


Sun Hosts CLRiiON4500 JMR JBODs

I/O

portPerfShow 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 -------------------------------------------------------------------------------- 0 0 0 0 355m 0 0 0 31m 43m 93m 94m 22m 22m 23m 23m 0 0 0 0 354m 0 0 0 31m 43m 94m 93m 22m 22m 23m 22m 0 0 0 0 349m 0 0 0 31m 43m 93m 93m 22m 22m 20m 20m

ISL

Trunking Maximizes Performance and Minimizes The SAN Management Effort In Figure 11, an identical read oriented I/O load is generated in the configuration with the dd UNIX utility. The load is primarily read oriented, with a small percentage of writes. The load is run with and without trunking. Ports 4, 5, and 6 are utilized for ISLs, while ports 8 through 15 are used for storage.

Due to the way in which FSPF established routes, the ISL on port 5 is operating at maximum bandwidth (208 MB/s) in the test where no trunking is utilized. Should another device attach to the switch and be assigned a route over port 5, that device would encounter congestion and it is likely that intervention would be necessary to manage the performance conflict. It is also possible that the devices using port 5 were experiencing congestion. In addition, notice how the loads in the test run without trunking enabled are unbalanced.

53-0000263-01

Page 14 of 26

In the test run with trunking enabled, the load is evenly distributed and there exists surplus bandwidth within the trunk to accommodate additional devices. It is not necessary to intervene to manage the addition of devices since the resources are optimally utilized.

Figure 11 Fabric Performance With and Without Trunking

3800 3800



I/O

portPerfShow 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 -------------------------------------------------------------------------------- 0 0 0 0 101m 208m 89m 0 55m 76m 76m 90m 28m 27m 23m 23m 112 0 112 0 103m 208m 94m 0 55m 76m 76m 94m 28m 28m 23m 23m 0 0 0 0 100m 208m 93m 0 55m 76m 76m 93m 26m 28m 22m 22m

Without Trunking

portPerfShow 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 -------------------------------------------------------------------------------- 0 0 0 0 132m 132m 132m 0 55m 90m 93m 55m 27m 27m 23m 23m 0 0 0 0 132m 132m 131m 0 55m 90m 92m 55m 27m 27m 23m 23m 0 0 0 0 131m 132m 131m 0 55m 91m 93m 55m 26m 27m 22m 22m

With Trunking

3 ISLs

6 Gb/s trunk

53-0000263-01

Page 15 of 26

How Does Trunking Impact SAN Design

Trunking optimizes the utilization of ISLs and reduces the SAN administration effort. These benefits enhance the utility of ISLs and so enhance the utility of designs that make use of ISLs. Given two SANs of equivalent functionality, the SAN that is easier to manage has the advantage. Instead of monitoring multiple ISLs, a single trunk is now monitored.

The high performance capabilities of a trunking-capable switch also makes these types of switches ideal for placement in the Core of a Core/Edge fabric (refer to the Brocade SAN Design Guide, publication number 53-0000231-xx for a detailed discussion regarding SAN Design). Because the trunk is efficient, it is less likely that congestion will be experienced and it is possible that fewer ISL are required in the fabric topology, yielding additional ports for attaching SAN devices.

Tip: Place Trunking-Capable Switches Adjacent To Each other When designing a SAN with trunking-capable switches or introducing trunkable switches to an existing SAN, it is important to place these switches adjacent to each other when working with any topology, including Core/Edge, Mesh, Cascade, and Ring topologies. Doing so enables the ability to trunk.

Tip: Place Trunking-Capable Switches In The Core When working with a Core/Edge topology, it is recommended to place the trunkable switches in the Core. As new trunking-capable switches are added to a Core/Edge topology, it will allow the added switches to connect with trunking-capable switches. Migrating existing SilkWorm switches not capable of trunking from the core to the edge and inserting trunking-capable switches into the core is an effective strategy, as shown in Figure 12. Then as new trunking switches are added, these switches can harness the power of trunking. This strategy also preserves the investment in switches that are not capable of trunking.

53-0000263-01

Page 16 of 26

Figure 12 Place trunking-capable switches in the core and redeploy the former core to the edge

Tip: Leave Open Ports On The Quad When connecting two switches in a fabric with one, two or three ISLs, consider leaving open the other ports on the quad for future trunk growth and fill in the remaining open ports with SAN devices first.

Tip: If Two Trunkable Switches Are Connected By More than One ISL, Use Trunking When connecting two switches with two or more ISLs consider utilizing trunking to achieve optimal performance and minimize the management effort.

53-0000263-01

Page 17 of 26

Trunking Requirements

For trunking to be enabled several requirements must first be met. If a switch is licensed and meets these requirements, it is only necessary to connect up the ISLs to form a trunk. The following requirements are necessary to trunk ISLs:

1. The switch platform must support trunking. Current switches that support trunking are the SilkWorm 3800, and 12000. Trunking is not supported on the SilkWorm 2000 series or SilkWorm 1000 series (Fabric OS 1.x or 2.x) switches. It is possible, supported, and encouraged to connect switches that are not capable of trunking to switches that are capable of trunking.

2. The two switches that are being trunked together must be licensed for trunking. To check whether or not a switch is licensed for trunking, use the licenseShow command.

3. The ports that form a trunk must reside in the same contiguous four-port groups, as shown in Figure 14. For example: 0-3, 4-7, 8-11, 12-15. Note, that the SilkWorm 3800 and 12000 quads are color coded to delineate a quad. It is supported to have two trunk groups per quad, meaning that two 2-ISL trunks (4 Gb/s each) are resident on the same quad and each trunk then connects to a different switch. The ports on both switches being trunked must be configured as E_Ports and these ports must run at 2Gb/s. Note that trunking is enabled by default in v3.x and v4.x Fabric OS if there exists a trunking license. The default state of 2 Gb/s switch ports is to auto-negotiate, so nothing is required to enable 2 Gb/s connectivity as the default is to connect at 2 Gb/s if the other device is 2 Gb/s capable. Also the default state for SilkWorm 2000, 3000, and 12000 series switch ports is to automatically configure as an E-port when attached to another Brocade switch. If there is some question as to a port’s suitability for trunking, use the command portCfgShow to verify the port configuration.

4. The difference in cable length between all ports in a trunking group must be less be less than 400 meters, and should be kept to 30 meters or less within a trunk to ensure optimal performance and bandwidth utilization.

53-0000263-01

Page 18 of 26

Figure 13 Port groupings that form a Quad For SilkWorm 12000

QuadsEach SilkWorm 12000 switch assembly blade consists of 16-ports, grouped into four quads

Figure 14 Port groupings that form a Quad For SilkWorm 3800

Quads

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

53-0000263-01

Page 19 of 26

ISL Trunking Administration

Once an ISL Trunking license has been purchased and installed ISL Trunking can be administered either through telnet commands or through Brocade Web Tools. Combined together, both methods provide excellent ease of use and an appropriate level of control. For convenience, a list of helpful trunking commands has been included as a reference in Appendix B of this document.

Brocade Web Tools provides an intuitive graphical interface to enable and review ISL Trunking settings. The ISL Trunking screens are accessed through a web browser using the Admin View of Web Tools. Figure 15 shows the Trunk Information Tab that lists the trunk groups in the switch, the Master Ports for the trunk groups, and the Member Ports for the trunk groups. Figure 16 shows the Port Setting Tab that displays which ports in the switch have ISL Trunking enabled. All ports have ISL Trunking enabled by default. ISL Trunking can be enabled or disabled by simply clicking the check box.

Figure 15 Web Tools Trunk Information Tab

53-0000263-01

Page 20 of 26

Figure 16 Web Tools Port Settings Tab Shows Which Ports Have Trunking Enabled

53-0000263-01

Page 21 of 26

Summary

A trunk logically joins two, three, or four ISLs into one logical ISL with a capacity of up to 8 Gb/s while preserving in-order delivery. In supported products, Brocade 2 Gb/s switches also have the ability to create up to two trunk groups per quad.

Use of trunking can minimize or eliminate congestion in the SAN because trunking optimizes ISL utilization. Since ISLs are now managed as a group instead of individually, the use of trunking minimizes the effort of managing a SAN. Trunking also increases high availability. Virtual Channels are integral to Brocade ISL Trunking and are essential for enabling large fabrics by insuring the reliable operation of the network, and improving the efficiency and performance of switch-to-switch communication.

To trunk, it is necessary to use switches that are licensed for trunking and support trunking, such as the SilkWorm 3800 and 12000. Ports that form a trunk must reside in a quad, run at 2 Gb/s, and be cabled together with cables that ideally differ in length by less than 30 meters. It is possible to dynamically increase bandwidth in a trunk and not impact ongoing I/O by simply adding an ISL to the trunk. The failure of an ISL within a trunk will result in some impact to I/O; however, as long as one ISL remains communication between the switches will continue. An ISL is capable of supporting up to 400 MB/s when operating full duplex. A four ISL trunk is potentially capable of 1600 MB/s when operating full duplex.

When designing and implementing SANs, place trunking capable switches adjacent to each other and in the core of a core/edge fabric. Fill up the open ports of a quad last so as to leave open these ports for trunk expansion. If two switches are connected by more than one ISL, consider the use of trunking to achieve optimal performance and minimize than SAN management effort.

53-0000263-01

Page 22 of 26

Appendix A: Terms and Definitions

Several terms, software functionalities, and definitions relative to trunking and ISLs are defined and explained below.

• Deskew Counter: This value is a metric used to identify the differences in cabling distances between trunking links within a trunking group. This metric is used to ensure in-order delivery across trunking links within the same trunking group. The higher the deskew counter value, the longer the cable lengths. If using the same cable lengths to connect two switches, expect to see the deskew counters for each ISL in the trunk to have very similar values.

• FSPF: Fabric Shortest Path First protocol. The FSPF protocol was developed by Brocade and subsequently adopted by the Fibre Channel standards community for allowing switches to discover the fabric topology and route frames correctly. It is the industry standard routing protocol for Fibre Channel networks.

• Trunking Group: Trunking uses a simple algorithm to optimally distribute frames across a set of available paths that link two adjacent switches. Such a set of links is called trunking group (see Figure 17). It is possible to have two trunk groups per quad. This means having two 2-isl trunks, with each trunk connected to a different switch.

• Trunking Ports: Ports that belong to a trunking group are called trunking ports (see Figure 17).

• Trunking Links: The components that comprise a trunk are termed trunking links. A link consists of the two trunking ports and the connecting components: SFP and cable. A trunking link is analogous to an ISL.

• Trunking Master: Only one of the trunking ports is used to set up all routing paths for the entire trunking group. This port is called trunking master port (see Figure 17). Note that if a master trunking link goes offline or fails, the remaining slave links will also go offline and regroup with a new master if there are at least two remaining active trunking ports remaining. The failure of any master port impacts the traffic that traverses this trunk since a regrouping of the trunks slaves must occur and because some in-flight data may be lost.

• Trunking Slave: Non-master trunking ports from a trunking group are termed trunking slaves. The failure of any slave port has minimal impact on the trunk or the traffic the traverses this trunk.

• Upstream/Downstream Designator: The terms upstream and downstream designate that the particular position of a switch in reference to the principal switch in the fabric. These paths are used in the process for assigning switch domain IDs. In Figure 17, notice that switch “edge1” is the principal switch and all “stream” designators are downstream of switch edge1. For switch core1, the path to the principal switch is upstream through port 0. The principal switch has no upstream ports.

53-0000263-01

Page 23 of 26

Figure 17 Trunking and ISL Elements

edge6edge5edge4edge3 edge7edge1 edge2

core2core1

edge1:admin> switchshowswitchName: edge1switchType: 2.4switchState: OnlineswitchRole: PrincipalswitchDomain: 2switchId: fffc02switchWwn: 10:00:00:60:69:10:8d:fdswitchBeacon: OFFport 0: sw Online E-Port 10:00:00:60:69:50:03:de "core1" (downstream)port 1: sw Online E-Port 10:00:00:60:69:50:03:a2 "core2" (downstream)port 2: sw No_Lightport 3: cu No_Lightport 4: sw No_Lightport 5: sw No_Lightport 6: -- No_Moduleport 7: -- No_Moduleport 8: id No_Lightport 9: -- No_Moduleport 10: id No_Lightport 11: -- No_Moduleport 12: id No_Lightport 13: sw No_Lightport 14: -- No_Moduleport 15: sw No_Light

core1:admin> switchshowswitchName: core1switchType: 9.1switchState: OnlineswitchRole: SubordinateswitchDomain: 8switchId: fffc08switchWwn: 10:00:00:60:69:50:03:deswitchBeacon: OFFport 0: id 1G Online E-Port 10:00:00:60:69:10:8d:fd "edge1" (upstream)port 1: id 1G Online E-Port 10:00:00:60:69:11:f9:f7 "edge2"port 2: id 1G Online E-Port 10:00:00:60:69:10:9b:52 "edge3"port 3: id 1G Online E-Port 10:00:00:60:69:12:f9:8c "edge4"port 4: id 1G Online E-Port 10:00:00:60:69:12:f9:8c "edge4"port 5: id 1G Online E-Port 10:00:00:60:69:10:9b:5b "edge5"port 6: id 2G No_Lightport 7: id 2G No_Lightport 8: id 2G Online E-Port 10:00:00:60:69:50:03:b3 "edge7" (Trunk master)port 9: id 2G Online E-Port (Trunk port, master is port #8)port 10: id 2G Online E-Port (Trunk port, master is port #8)port 11: id 2G No_Lightport 12: id 2G Online E-Port 10:00:00:60:69:50:02:d0 "edge6" (Trunk master)port 13: id 2G Online E-Port (Trunk port, master is port #12)port 14: id 2G Online E-Port (Trunk port, master is port #12)port 15: id 2G Online E-Port (Trunk port, master is port #12)

3800

2800

1 ISL2 ISL Trunk3 ISL Trunk4 ISL Trunk

Principal

} Trunking Group

} Trunking Group

Trunk Master

53-0000263-01

Page 24 of 26

Appendix B: Helpful Trunking Commands

An annotated list of trunking related commands follow to provide a quick reference. For detail on a particular command, please reference the manuals or “help” page.

trunkShow: Use this to display detailed trunking information. Helpful for identifying deskew constant. This command is new in V3.0 Fabric OS.

switchShow: Use this command to check status of switch ports and trunking. This command is available in all versions of Fabric OS.

portCfgTrunkport: Use this command to individually configure a port for trunking. This command is new in V3.0 Fabric OS.

switchCfgTrunk: Use this command to enable/disable an entire switch for trunking. This command is new in V3.0 Fabric OS. Trunking is enabled by default in the V3.x and V4.x Fabric OS releases.

islShow: Use this command to display ISL status. Identifies aggregate trunk bandwidth. This command is new in V3.0 Fabric OS.

trunkDebug: Use this command to assist with troubleshooting a trunk. Supply two local port numbers. This command is new in V3.0 Fabric OS.

portCfgShow: Use this command to view the configuration status of all ports. An improperly configured port cannot join a trunk group. For example, a port could be configured as a non-E_Port, which would prevent that port from joining into the trunk group. This command is new in V3.0 Fabric OS.

53-0000263-01

Page 25 of 26

Copyright

IMPORTANT NOTICE This document is the property of Brocade. It is intended solely as an aid for installing and configuring Storage Area Networks constructed with Brocade switches. This document does not provide a warranty to any Brocade software, equipment, or service, nor does it imply product availability. Brocade is not responsible for the use of this document and does not guarantee the results of its use. Brocade does not warrant or guarantee that anyone will be able to recreate or achieve the results described in this document. The installation and configuration described in this document made use of third party software and hardware. Brocade does not make any warranties or guarantees concerning such third party software and hardware.

2002, Brocade Communications Systems, Incorporated.

ALL RIGHTS RESERVED. Part number: 53-0000263-01

Brocade, SilkWorm, SilkWorm Express, and the Brocade logo are trademarks or registered trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries.

All other brands, products, or service names are or may be trademarks or service marks of, and are used to identify, products or services of their respective owners.

NOTICE: THIS DOCUMENT IS FOR INFORMATIONAL PURPOSES ONLY AND DOES NOT SET FORTH ANY WARRANTY, EXPRESS OR IMPLIED, CONCERNING ANY EQUIPMENT, EQUIPMENT FEATURE, OR SERVICE OFFERED OR TO BE OFFERED BY BROCADE. BROCADE RESERVES THE RIGHT TO MAKE CHANGES TO THIS DOCUMENT AT ANY TIME, WITHOUT NOTICE, AND ASSUMES NO RESPONSIBILITY FOR ITS USE. THIS INFORMATIONAL DOCUMENT DESCRIBES FEATURES THAT MAY NOT BE CURRENTLY AVAILABLE. CONTACT A BROCADE SALES OFFICE FOR INFORMATION ON FEATURE AND PRODUCT AVAILABILITY.

Export of technical data contained in this document may require an export license from the United States Government. Brocade Communications Systems, Incorporated 1745 Technology Drive San Jose, CA 95110

53-0000263-01

Page 26 of 26

Evaluation Form

Your feedback is valued by Brocade. In particular, we are interested in understanding how the SOLUTIONware report helped make a difference in a SAN task, problem, or decision. Please FAX this form back to: FAX (408) 392-5200.

SOLUTIONware Title: Exploring Brocade ISL Trunking

Document number: 53-0000263-01

Review:

What other subjects would you like to see SOLUTIONware reports on?

Please rate your overall satisfaction: ( )very satisfied ( )satisfied ( )not satisfied

Please identify yourself as belonging to one of these groups:

( ) End-User Customer ( ) Fabric Integrator ( ) Master Reseller ( ) Partner ( ) Brocade Employee ( ) Brocade OEM ( )Other

Documents

Brocade ISL Trunking