Brocade Fabric Technology with the Pure Storage FlashArray

Supporting Fabric OS 8.1.0 and 8.2.0

VALIDATION TEST REPORT

Brocade Fabric Technology with the Pure Storage FlashArray//m20 Validation Test Report

53-1005025-0216 March 2018

Copyright © 2018 Brocade Communications Systems LLC. All Rights Reserved.

Brocade and the stylized B logo are among the trademarks of Brocade Communications Systems LLC. Broadcom, the pulse logo, and Connectingeverything are among the trademarks of Broadcom. The term "Broadcom" refers to Broadcom Limited and/or its subsidiaries.

Brocade, a Broadcom Limited Company, reserves the right to make changes without further notice to any products or data herein to improve reliability,function, or design. Information furnished by Brocade is believed to be accurate and reliable. However, Brocade does not assume any liability arising out ofthe application or use of this information, nor the application or use of any product or circuit described herein, neither does it convey any license under itspatent rights nor the rights of others.

The product described by this document may contain open source software covered by the GNU General Public License or other open source licenseagreements. To find out which open source software is included in Brocade products, view the licensing terms applicable to the open source software, andobtain a copy of the programming source code, please visit https://www.broadcom.com/support/fibre-channel-networking/tools/oscd.

Brocade Fabric Technology with the Pure Storage FlashArray//m20 Validation Test Report2 53-1005025-02

https://www.broadcom.com/support/fibre-channel-networking/tools/oscd

ContentsPreface...................................................................................................................................................................................................................................5

Document History......................................................................................................................................................................................................................................5Overview........................................................................................................................................................................................................................................................5Purpose of This Document....................................................................................................................................................................................................................5Audience........................................................................................................................................................................................................................................................ 5Objectives......................................................................................................................................................................................................................................................5Related Documents...................................................................................................................................................................................................................................6About Brocade............................................................................................................................................................................................................................................ 6About Pure Storage...................................................................................................................................................................................................................................6

Configure DUT and Test Equipment ..............................................................................................................................................................................7Task 1. Brocade FC Fabric Configuration........................................................................................................................................................................................7Task 2. Pure Storage Array Configuration.................................................................................................................................................................................... 12Task 3. Host Configuration................................................................................................................................................................................................................. 15

Test Report.........................................................................................................................................................................................................................21What's New in This Report .................................................................................................................................................................................................................21Test History................................................................................................................................................................................................................................................21Test Plan Overview.................................................................................................................................................................................................................................21Scope........................................................................................................................................................................................................................................................... 21Test Configuration...................................................................................................................................................................................................................................22DUT Descriptions................................................................................................................................................................................................................................... 22

Test Cases.......................................................................................................................................................................................................................... 251. Fabric Initialization—Base Functionality................................................................................................................................................................................... 26

1.1—Storage Device – Physical and Logical Login with Speed Negotiation....................................................................................................... 261.2—Zoning and LUN Mapping..............................................................................................................................................................................................271.3—Storage Device Fabric I/O Integrity.............................................................................................................................................................................271.4—Storage Device Multipath Configuration – Path Integrity...................................................................................................................................28

2. Fabric – Advanced Functionality ................................................................................................................................................................................................292.1—Storage Device Bottleneck Detection Using MAPS FPI – with Congested Host................................................................................... 292.2—Storage Device Bottleneck Detection Using MAPS FPI – with Congested Fabric................................................................................ 302.3—Storage Device – IO Insight Latency Monitoring with MAPS Alerts.............................................................................................................312.4—Storage Device – QoS Integrity with QoS Zone Based Traffic Prioritization.............................................................................................322.5—Storage Device – QoS Integrity with CS_CTL Based Traffic Prioritization.................................................................................................332.6—Storage Device — FC Protocol Jammer Test Suite..............................................................................................................................................36

3. Stress and Error Recovery with Device Multipath................................................................................................................................................................ 373.1—Storage Device Fabric I/O Integrity - Congested Fabric....................................................................................................................................373.2—Storage Device Nameserver Integrity – Device Recovery with Port Toggle..............................................................................................383.3—Storage Device Nameserver Integrity – Device Recovery with Device Relocation.................................................................................393.4—Storage Device – Device Recovery with Device Port Toggle – Extended Run.........................................................................................403.5—Storage Device Recovery - ISL Port Toggle...........................................................................................................................................................413.6—Storage Device Recovery – ISL Port Toggle (Entire Switch).............................................................................................................................413.7—Storage Device Recovery – Switch Offline.............................................................................................................................................................. 423.8—Storage Device Recovery – Switch Firmware Download...................................................................................................................................43

4. Storage Device – Fibre Channel Routing (FCR) Internetworking Tests...................................................................................................................... 444.1—Storage Device Internetworking Validation with FC Host.................................................................................................................................. 444.2—Storage Device Internetworking Validation with FCoE Test Using VDX FlexPort................................................................................... 45

Brocade Fabric Technology with the Pure Storage FlashArray//m20 Validation Test Report53-1005025-02 3

4.3—Storage Device Edge Recovery after FCR Disruptions......................................................................................................................................464.4—Storage Device Backbone Recovery after FCR Disruptions............................................................................................................................47

5. Optional/Additional Tests............................................................................................................................................................................................................... 485.1—Storage Device Firmware Update................................................................................................................................................................................485.2—Performance I/O Testing with Varying Workload Dimensions........................................................................................................................495.3—VMware IOAnalyzer Workload Test.............................................................................................................................................................................50

Test Conclusions...............................................................................................................................................................................................................53


Preface• Document History................................................................................................................................................................................................ 5• Overview...................................................................................................................................................................................................................5• Purpose of This Document.............................................................................................................................................................................. 5• Audience...................................................................................................................................................................................................................5• Objectives................................................................................................................................................................................................................ 5• Related Documents............................................................................................................................................................................................. 6• About Brocade.......................................................................................................................................................................................................6• About Pure Storage............................................................................................................................................................................................. 6

Document HistoryDate Part Number Description

April 7, 2017 53-1005025-01 Initial release supporting Fabric OS 8.1.0.

March 16, 2018 53-1005025-02 Supporting Fabric OS 8.2.0.

OverviewThe Storage Fabric Ready (SFR) program is a comprehensive testing and configuration initiative to provide Fibre Channel SANinteroperability with flash storage. This program provides testing of multiple fabrics, heterogeneous servers, and HBAs in large port-count Brocade environments. Testing covers future Brocade OS versions and vendor's software releases.

The SFR qualification program will help verify seamless interoperability and optimum performance with solid state storage, software-defined storage and hyper-converged systems in Brocade storage fabrics.

Purpose of This DocumentThis document provides the validation of Brocade fabric technology with the Pure M20 storage array, using multiple switch platforms,HBAs, and server operating systems. This validation shows that the M20 interoperates properly within Brocade Gen 5 and Gen 6 FibreChannel fabrics while supporting the performance and low latency associated with solid state storage.

AudienceThe content in this document is written for a technical audience, including solution architects, system engineers, and technicaldevelopment representatives.

Objectives• Test the Pure //m20 array with Brocade FC fabrics, in single and routed configurations for different stress and error recovery

scenarios, to validate the interoperability and integration of the array with Brocade FC fabrics.


• Validate the performance of FC fabric in a solid state storage environment for high throughput and low latency applications.

Related Documents• Brocade Fabric OS Administration Guide

• Brocade Monitoring and Alerting Policy Suite Configuration Guide

• Brocade Flow Vision Configuration Guide

• Brocade Fabric OS Command Reference Guide

• SAN Design and Best Practices

• SAN Fabric Administration Best Practices Guide

• Emulex ExpressLane Configuration

• Emulex OneCommand Manager User Manual

• QLogic QLE2672 Adapter User Guide

• QLogic BR-1860 Adapter Administrator’s Guide

About BrocadeBrocade, a Broadcom Limited company, provides innovative storage networking solutions for data center, campus, and service providernetworks, helping to reduce complexity and cost while enabling virtualization and cloud computing to increase business agility. To helpensure a complete solution, Brocade partners with world-class IT companies and provides comprehensive education, support, andprofessional services offerings (www.broadcom.com).

About Pure StoragePure Storage has a simple mission: to enable the widespread adoption of flash in the enterprise data center. We're a team of some of thebrightest minds in storage, flash memory and related technical industries. Founded on principles of teamwork and knowledge sharing, wefocus on producing an exceptional result while we transform the landscape of the enterprise storage market (and have some fun alongthe way).

Related Documents


http://www.brocade.com/en/backend-content/pdf-page.html?/content/dam/secure-external/product-guides/brocade-fabric-os/8-1-0/fos-81x-adminguide.pdf

http://www.brocade.com/en/backend-content/pdf-page.html?/content/dam/secure-external/product-guides/brocade-fabric-os/8-1-0/fos-81x-maps.pdf

http://www.brocade.com/en/backend-content/pdf-page.html?/content/dam/secure-external/product-guides/brocade-fabric-os/8-1-0/fos-81x-flow.pdf

http://www.brocade.com/en/backend-content/pdf-page.html?/content/dam/secure-external/product-guides/brocade-fabric-os/8-1-0/fos-81x-commandref.pdf

http://www.brocade.com/downloads/documents/best_practice_guides/san-design-best-practices.pdf

https://www.brocade.com/content/dam/common/documents/content-types/product-design-guide/san-admin-best-practices-bp.pdf

https://docs.broadcom.com/docs/12356123

http://www.broadcom.com

Configure DUT and Test Equipment• Task 1. Brocade FC Fabric Configuration.................................................................................................................................................. 7• Task 2. Pure Storage Array Configuration...............................................................................................................................................12• Task 3. Host Configuration............................................................................................................................................................................15

Task 1. Brocade FC Fabric ConfigurationConfiguration settings of the Brocade switches in the test bed are covered here.

1. Zoning is configured per host, and the array ports are placed in a zone alias for convenience.

> alicreate pure_ali, “52:4a:93:75:8c:54:50:00; 52:4a:93:75:8c:54:50:01; 52:4a:93:75:8c:54:50:10; 52:4a:93:75:8c:54:50:11”

> alishow pure_ali alias: pure_ali 52:4a:93:75:8c:54:50:00; 52:4a:93:75:8c:54:50:01; 52:4a:93:75:8c:54:50:10; 52:4a:93:75:8c:54:50:11

> zonecreate ssd067172_pure, “10:00:8c:7c:ff:1f:7b:00; 10:00:8c:7c:ff:1f:7b:01; pure_ali”

> zoneshow ssd067172_pure zone: ssd067172_pure 10:00:8c:7c:ff:1f:7b:00; 10:00:8c:7c:ff:1f:7b:01; pure_ali


2. Configure MAPS on switches (requires Fabric Vision License). This will enable reporting of latency and congestion alerts oneach switch, as well as a number of switch and fabric health metrics. For more information on configuring and using MAPS, seethe Brocade Monitoring and Alerting Policy Suite Configuration Guide.

Enable the desired MAPS policy using any of the available default policies or create a custom policy. For this test the defaultAggressive Policy is used. This provides the most sensitive threshold levels for detection of latency and congestion.

< ========== > > mapsconfig --enablemaps -policy dflt_aggressive_policy< ========== >

Define the reporting actions MAPS will take. In this test, errors discovered by MAPS will generate a RASLog entry and sendemail to the configured recipients.

< ========== >> mapsconfig --actions raslog,email< ========== >

The Fabric Performance Impact (FPI) category within MAPS monitors the current condition of the latency seen on F_Ports overdifferent time windows and uses that to determine the performance impact to the fabric and network.

To examine the effective MAPS configuration, use this command:

> mapsconfig –showConfigured Notifications: RASLOG,EMAILMail Recipient: [email protected],[email protected] Monitoring: EnabledPaused members :===============PORT :CIRCUIT :SFP :

Use this command to view the summary of the events or rules triggered, and the objects on which the rules were triggered overa specified period of time.

> mapsdb –show1 Dashboard Information:=======================DB start time: Thu Feb 23 07:02:51 2017Active policy: ssr_aggressive_policyConfigured Notifications: RASLOG,EMAIL,SW_CRITICAL,SW_MARGINALFenced Ports : NoneDecommissioned Ports : None

Task 1. Brocade FC Fabric Configuration


3. Configure Flow Monitoring with I/O Insight feature on the Brocade G620 Gen 6 32Gb switches in the fabric.

The I/O Insight feature supported on the Gen 6 32 Gb hardware allows us to monitor the flow latency statistics at the SCSI I/Oexchange level. The monitoring can be configured at an IT (Initiator-Target) flow level on pizza-box type switches and at an ITL(Initiator-Target-LUN) flow level on chassis based switches.

NOTERequires "Fabric Vision" and "IO Insight" license.

a) Create a "Flow Monitor" flow at the source or destination device port on the G620 switch.

root> flow --create ios_pure_1 -fea mon -dstdev df1400 -egrport 20Monitor feature(s) have been activated.root> flow --create ios_pure_2 -fea mon -dstdev df0400 -egrport 4Monitor feature(s) have been activated.root> flow --show ios_pure_1=============================================================================Name : ios_pure_1 Features: mon(Activated) noConfig: OffDefinition: EgrPort(20),DstDev(0xdf1400)

b) Import the created flows into MAPS.

root> mapsconfig --import ios_pure_1root> mapsconfig --import ios_pure_2root> logicalgroup –show--------------------------------------------------------------------------------------------Group Name |Predefined |Type |Member Count |Members--------------------------------------------------------------------------------------------ios_pure_1 |No |Flow |1 |Monitored Flowios_pure_2 |No |Flow |1 |Monitored Flow

c) Create MAPS rules to monitor the desired SCSI I/O latency statistics and add it to a custom MAPS policy.

root> mapspolicy --clone dflt_moderate_policy -name ios_mod_policy

root> mapsrule --create ios_pure_4k_rd_status -group ios_pure_1 -monitor RD_STATUS_TIME_LT_8K -timebase min -op ge -value 1 -action email,raslog -policy ios_mod_policyroot> mapspolicy --show ios_mod_policyPolicy Name: ios_mod_policyRule Name |Condition |Actions |ios_pure_4k_rd_status |ios_pure_1(RD_STATUS_TIME_LT_8K/min>=1) |email,raslog |

d) Enable the MAPS policy.

root> mapspolicy --enable ios_mod_policyroot>root> mapsdb --show

1 Dashboard Information:========================

DB start time: Thu May 19 09:27:24 2016Active policy: ios_mod_policyConfigured Notifications: RASLOG,EMAIL,SW_CRITICAL,SW_MARGINALFenced Ports : NoneDecommissioned Ports : NoneFenced circuits : N/AQuarantined Ports : NoneTop Zoned PIDs <pid(it-flows)>: 0xdf2000(30) 0xdf2100(30) 0xdf0400(28) 0xdf1400(28) 0xdf1b00(22)



4. Configure FlexPorts on VDX 6740 switches for Fibre Channel support (FCoE Base License is required).

These ports are used as uplinks from VCS fabric to the Fibre Channel fabric in this test. For detailed information on configuringVDX 6740 switches for Fibre Channel support, see the Brocade Network OS Layer 2 Switching Configuration Guide.

Example of configuring a Fibre Channel port on a Brocade VDX 6740 switch:

# conf t(config)# hardware(config-hardware)# flexport 112/0/38(config-flexport-112/0/38)# type fibre-channel

FlexPort speeds are configured in connector-groups. In this test bed example, the speed HighMixed is chosen. This will allowthe Fibre Channel FlexPort to operate at 16Gb speed, which is optimal because the uplink is attaching to a 16Gb Fibre ChannelSAN. Also, not all of the ports in this connector group are configured as Fibre Channel ports; some are still configured asEthernet ports, and HighMixed setting will support the Ethernet speeds on these ports as well. For detailed information onconfiguring VDX 6740 connector group speeds. see the Brocade Network OS Layer 2 Switching Configuration Guide.

• LowMixed - 2/4/8G Fibre Channel, and Ethernet speeds (default)

• HighMixed - 16G Fibre Channel, and Ethernet speeds

• FibreChannel - 2/4/8/16G Fibre Channel

Use this command to set the connector-group speed to HighMixed:

(config-hardware)# connector-group 111/0/3(config-connector-group-111/0/3)# speed HighMixed

5. Configure zones for FCoE initiators on VDX switches.

Example zone on Brocade VDX 6740 switch:

# show zoning enabled-configurationzoning enabled-configuration cfg-name NOS_SFR

zoning enabled-configuration enabled-zone lsan_hb067166_pure member-entry 10:00:8c:7c:ff:1f:7b:00 member-entry 10:00:8c:7c:ff:1f:7b:01 member-entry 52:4a:93:7d:f3:5f:61:00 member-entry 52:4a:93:7d:f3:5f:61:01 member-entry 52:4a:93:7d:f3:5f:61:10 member-entry 52:4a:93:7d:f3:5f:61:11



6. Configure Fibre Channel Routing (an Integrated Routing license is required). Detailed information on FCR setup can be found inthe Brocade Fabric OS Administration Guide.

Example FCR configuration is shown below:

> fcrconfigure –bbfid 100> fosconfig --enable fcr> portcfgexport [port#] -a1 –m[0/5] -f 10 {0=Brocade FC fabric; 5=Brocade NOS fabric}

The prefix 'lsan' is used when configuring zones for use in Fibre Channel Routing.

Example of zone prefixed with 'lsan':

> zoneshow lsan_hb067166_pure zone: lsan_hb067166_pure 10:00:8c:7c:ff:23:b7:00; 10:00:8c:7c:ff:23:b7:01; 52:4a:93:7d:f3:5f:61:00; 52:4a:93:7d:f3:5f:61:01 52:4a:93:7d:f3:5f:61:10; 52:4a:93:7d:f3:5f:61:11

Example output of exported devices:

> fcrproxydevshow Proxy WWN Proxy Device Physical State Created PID Exists PIDin Fabric in Fabric------------------------------------------------------------------------ 40 50:02:4f:40:55:82:01:00 02f906 100 532400 Imported 40 50:02:4f:40:55:82:01:01 02fa06 100 df2400 Imported 40 50:02:4f:40:55:82:02:00 02fb06 100 532500 Imported 40 50:02:4f:40:55:82:02:01 02fc06 100 df2500 Imported 50 50:02:4f:40:55:82:01:00 02fc03 100 532400 Imported 50 50:02:4f:40:55:82:01:01 02fd03 100 df2400 Imported 50 50:02:4f:40:55:82:02:00 02fe03 100 532500 Imported 50 50:02:4f:40:55:82:02:01 02ff03 100 df2500 Imported



Task 2. Pure Storage Array ConfigurationConfiguration steps for the Pure Storage array are covered here.

1. Create host initiator group for each host:

FIGURE 1 New Host Creation on the Pure Storage Array

Task 2. Pure Storage Array Configuration


2. Add WWPNs to host:

FIGURE 2 Adding Host Ports to the Host Configuration on the Pure Storage Array



3. Create new volumes. For this test configuration 8 x 10GB volumes are created:

FIGURE 3 Creating New Volumes on the Pure Storage Array

FIGURE 4 Finished LUN Configuration and Presentation on the Pure Storage Array



Task 3. Host ConfigurationConfiguration settings of servers in the test bed are covered here.

1. Provision a minimum of two uplinks from the host to the FC fabric for redundancy and use native multipath tools to manage theavailable paths and load-balance across them.

Task 3. Host Configuration


2. Multipath configuration on Linux hosts.

This configuration allows all paths to be used in a round-robin fashion. This provides superior performance to the default Linuxsettings which would only use a single active path per LUN.

Recommended /etc/multipath.conf entry on Linux systems:

devices { device { vendor "PURE" path_selector "queue-length 0" path_grouping_policy multibus path_checker tur fast_io_fail_tmo 10 dev_loss_tmo 60 no_path_retry 0 }}

Example multipath configuration on Linux:

# multipath -ll

mpathp (3624a9370a15a66e949f7d1440001009a) dm-19 PURE ,FlashArray size=10.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 1:0:1:13 sdm 8:192 active ready running |- 1:0:2:13 sdu 65:64 active ready running |- 1:0:3:13 sdac 65:192 active ready running |- 1:0:4:13 sdak 66:64 active ready running |- 10:0:2:13 sddm 71:64 active ready running |- 10:0:4:13 sdec 128:64 active ready running |- 10:0:3:13 sddu 71:192 active ready running `- 10:0:5:13 sdek 128:192 active ready runningmpatho (3624a9370a15a66e949f7d1440001009c) dm-20 PURE ,FlashArray size=10.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 1:0:1:15 sdo 8:224 active ready running |- 1:0:2:15 sdw 65:96 active ready running |- 1:0:3:15 sdae 65:224 active ready running |- 1:0:4:15 sdam 66:96 active ready running |- 10:0:2:15 sddo 71:96 active ready running |- 10:0:3:15 sddw 71:224 active ready running |- 10:0:4:15 sdee 128:96 active ready running `- 10:0:5:15 sdem 128:224 active ready runningmpathn (3624a9370a15a66e949f7d14400010099) dm-16 PURE ,FlashArray size=10.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 1:0:1:12 sdl 8:176 active ready running |- 1:0:2:12 sdt 65:48 active ready running |- 1:0:3:12 sdab 65:176 active ready running |- 1:0:4:12 sdaj 66:48 active ready running |- 10:0:2:12 sddl 71:48 active ready running |- 10:0:3:12 sddt 71:176 active ready running |- 10:0:4:12 sdeb 128:48 active ready running `- 10:0:5:12 sdej 128:176 active ready runningmpathm (3624a9370a15a66e949f7d1440001009d) dm-6 PURE ,FlashArray size=10.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 1:0:1:16 sdp 8:240 active ready running |- 1:0:2:16 sdx 65:112 active ready running |- 1:0:3:16 sdaf 65:240 active ready running |- 1:0:4:16 sdan 66:112 active ready running |- 10:0:2:16 sddp 71:112 active ready running |- 10:0:3:16 sddx 71:240 active ready running |- 10:0:4:16 sdef 128:112 active ready running `- 10:0:5:16 sden 128:240 active ready runningmpathl (3624a9370a15a66e949f7d1440001009b) dm-18 PURE ,FlashArray size=10.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active



|- 1:0:1:14 sdn 8:208 active ready running |- 1:0:2:14 sdv 65:80 active ready running |- 1:0:3:14 sdad 65:208 active ready running |- 1:0:4:14 sdal 66:80 active ready running |- 10:0:3:14 sddv 71:208 active ready running |- 10:0:2:14 sddn 71:80 active ready running |- 10:0:4:14 sded 128:80 active ready running `- 10:0:5:14 sdel 128:208 active ready runningmpathk (3624a9370a15a66e949f7d1440001009e) dm-7 PURE ,FlashArray size=10.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 1:0:1:17 sdq 65:0 active ready running |- 1:0:2:17 sdy 65:128 active ready running |- 1:0:4:17 sdao 66:128 active ready running |- 1:0:3:17 sdag 66:0 active ready running |- 10:0:3:17 sddy 128:0 active ready running |- 10:0:2:17 sddq 71:128 active ready running |- 10:0:4:17 sdeg 128:128 active ready running `- 10:0:5:17 sdeo 129:0 active ready runningmpathj (3624a9370a15a66e949f7d14400010097) dm-3 PURE ,FlashArray size=10.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 1:0:1:10 sdj 8:144 active ready running |- 1:0:2:10 sdr 65:16 active ready running |- 1:0:3:10 sdz 65:144 active ready running |- 1:0:4:10 sdah 66:16 active ready running |- 10:0:2:10 sddj 71:16 active ready running |- 10:0:3:10 sddr 71:144 active ready running |- 10:0:4:10 sddz 128:16 active ready running `- 10:0:5:10 sdeh 128:144 active ready runningmpathi (3624a9370a15a66e949f7d14400010098) dm-8 PURE ,FlashArray size=10.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 1:0:1:11 sdk 8:160 active ready running |- 1:0:2:11 sds 65:32 active ready running |- 1:0:3:11 sdaa 65:160 active ready running |- 1:0:4:11 sdai 66:32 active ready running |- 10:0:2:11 sddk 71:32 active ready running |- 10:0:3:11 sdds 71:160 active ready running |- 10:0:5:11 sdei 128:160 active ready running `- 10:0:4:11 sdea 128:32 active ready running



3. Multipath configuration on Windows hosts:

Use Windows MPIO dialog box to discover and manage multipath entries for Pure Storage.

FIGURE 5 Windows MPIO Multipathing with the Pure Storage Array



4. Multipath configuration on VMware hosts.

Change path selection to "Round-Robin". This provides superior performance to the 'Most Recently Used' setting which wouldonly use a single active path per LUN.

FIGURE 6 Round Robin Multipath Configuration on VMware



5. Apply any additional host tuning (for Linux systems). Settings shown below help to support high IOPS operation of the PureStorage flash array with Linux hosts.

Create a file named "/etc/udev/rules.d/99-pure-storage.rules" and add the following lines:

# Recommended settings for Pure Storage FlashArray.# Use noop scheduler for high-performance solid-state storageACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="PURE", ATTR{queue/scheduler}="noop"# Reduce CPU overhead due to entropy collectionACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="PURE", ATTR{queue/add_random}="0"# Spread CPU load by redirecting completions to originating CPUACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="PURE", ATTR{queue/rq_affinity}="2"# Set the HBA timeout to 60 secondsACTION=="add", SUBSYSTEMS=="scsi", ATTRS{model}=="FlashArray ", RUN+="/bin/sh -c 'echo 60 > /sys$DEVPATH/device/timeout'"

6. Set up workload generators.

On Windows and Linux system, Medusa Labs Test Tools is installed. On VMware systems, VMware's IOAnalyzer is installed.



Test Report• What's New in This Report ........................................................................................................................................................................... 21• Test History.......................................................................................................................................................................................................... 21• Test Plan Overview........................................................................................................................................................................................... 21• Scope......................................................................................................................................................................................................................21• Test Configuration............................................................................................................................................................................................. 22• DUT Descriptions..............................................................................................................................................................................................22

What's New in This Report• The array firmware under test is Purity version 4.8.11.

• The Brocade Fabric OS (FOS) version under test is 8.2.0.

• Emulex and QLogic adapters have updated firmware and drivers.

See the DUT Description tables below for detailed information.

Test HistoryStorage Model Storage Firmware Brocade FOS Version Date

Pure FlashArray//m20 Purity 4.7.9 FOS 8.1.0 April 2016

Pure FlashArray//m20 Purity 4.8.11 FOS 8.2.0 January 2018

Test Plan OverviewThe storage array is connected to a Brocade FC fabric and multiple server hosts to drive IO in a multipath configuration. Error injection isintroduced, and failover, and recovery behaviors are observed. IO performance is observed across different workload configurations.

ScopeTesting is performed with a mix of GA and development versions of Brocade's Fabric OS (FOS) in a heterogeneous environment. Testbeds include Brocade directors and switches configured in routed and non-routed fabric configurations.

Testing is centered on interoperability and optimal configuration. Performance is observed within the context of best practice fabricconfiguration; however absolute maximum benchmark reporting of storage performance is beyond the scope of this test.

Details of the test steps are covered under "Test Cases" section. Standard test bed setup includes IBM/HP/Dell chassis server hosts withBrocade/QLogic/Emulex/ATTO HBAs with two uplinks from every host to a Brocade FC fabric. I/O generators included Medusa LabsTest Tools and VMware IOAnalyzer.


Test Configuration

DUT DescriptionsThe following tables provide details about the devices under test (DUTs).

TABLE 1 Storage Array

DUT ID Model Vendor Description

PureStorage //m20

FlashArray//m20 Pure Storage The Pure Storage //m20 flash storage array is an all-flash array that supports up to120-TB useable capacity. The system under test is configured with 2.79-TB useablecapacity, 4x 8Gb Fibre Channel, and 4x 10GbE target connections in an active-activeconfiguration.

Test Configuration


TABLE 2 Switches

DUT ID Model Vendor Description

Edge-1 Brocade 6510 Brocade 48-port Gen 5 16Gb FC switch

Edge-2 Brocade G620 Brocade 64-port Gen 6 32Gb FC switch



Chassis-1 Brocade X6-8 Brocade 8-slot Gen 6 32Gb director

Chassis-2 Brocade X6-4 Brocade 4-slot Gen 6 32Gb director

DUT Descriptions



Test Cases• 1. Fabric Initialization—Base Functionality..............................................................................................................................................26• 2. Fabric – Advanced Functionality ...........................................................................................................................................................29• 3. Stress and Error Recovery with Device Multipath...........................................................................................................................37• 4. Storage Device – Fibre Channel Routing (FCR) Internetworking Tests.................................................................................44• 5. Optional/Additional Tests..........................................................................................................................................................................48

These test cases are designed to verify basic and advanced functionality features between the Brocade FOS Fabric and host and storagedevices, to stress all devices, and to confirm successful error recovery.

1. Fabric Initialization - Base Functionality

1.1 Storage Device - Physical and Logical Login with Speed Negotiation

1.2 Zoning and LUN Mapping

1.3 Storage Device Fabric IO Integrity

1.4 Storage Device Multipath Configuration - Path Integrity

2. Fabric - Advanced Functionality

2.1 Bottleneck Detection Using MAPS FPI - with Congested Host

2.2 Bottleneck Detection Using MAPS FPI - with Congested Fabric

2.3 Flow Monitoring with IO Insight and MAPS

2.4 QoS Integrity with QoS Zone Based Traffic Prioritization

2.5 QoS Integrity with CS_CTL Tag Based Frame Prioritization

2.6 Storage Device - FC Protocol Jammer Test Suite

3 Stress and Error Recovery with Device Multipath

3.1 Storage Device Fabric IO Integrity - Congested Fabric

3.2 Storage Device Nameserver Integrity - Device Recovery with Port Toggle

3.3 Storage Device Nameserver Integrity - Device Recovery with Device Relocation

3.4 Storage Device Nameserver Stress - Device Recovery with Device Port Toggle

3.5 Storage Device Recovery - ISL Port Toggle (Sequential)

3.6 Storage Device Recovery - ISL Port Toggle (Entire Switch)

3.7 Storage Device Recovery - Director Blade Maintenance

3.8 Storage Device Recovery - Switch Offline

3.9 Storage Device Recovery - Switch Firmware Download

4 Storage Device - Fibre Channel Routing (FCR) Internetworking Tests

4.1 Storage Device Internetworking Validation with FC Host

4.2 Storage Device Internetworking Validation with FCoE Using VDX FlexPort

4.3 Storage Device Edge Recovery after FCR Disruptions

4.4 Storage Device Backbone Recovery after FCR Disruptions

5 Optional/Additional Tests

5.1 Storage Device Nondisruptive Firmware Update

5.2 Performance I/O Testing with Varying Workload Dimensions

5.3 Performance Testing with VMware IOAnalyzer Application Workload Suite


1. Fabric Initialization—Base Functionality

1.1—Storage Device – Physical and Logical Login with Speed Negotiation

Test ObjectiveVerify device login to switch and nameserver with all supported speed settings.

Test ConfigurationSet switch ports to supported speed settings.

portcfgspeed <port> [8/4/0]

Result Validation1. Validate link states on the array and verify speed negotiation and device login at different speeds. Use command 'portshow' on

the switch to check the link state and speed, and the command 'nscamshow' to verify device login to the fabric.

2. Check switch port status and verify the "actual" and "configured" link speed. Check name server for device login.

> portshow 24portIndex: 24portName: port24portHealth: HEALTHY

Authentication: NoneportDisableReason: NoneportCFlags: 0x1portFlags: 0x20b03 PRESENT ACTIVE F_PORT G_PORT U_PORT NPIV LOGICAL_ONLINE LOGIN NOELP ACCEPT FLOGILocalSwcFlags: 0x0portType: 26.0POD Port: Port is licensedportState: 1 OnlineProtocol: FCportPhys: 6 In_Sync portScn: 32 F_Portport generation number: 100state transition count: 5

portId: e01800portIfId: 43020027portWwn: 20:18:c4:f5:7c:41:1b:14portWwn of device(s) connected: 52:4a:93:75:8c:54:50:01 52:4a:93:75:8c:54:50:01Distance: normalportSpeed: N8Gbps

Test ResultsPASS. Speed negotiation, device login, and connectivity verified.



1.2—Zoning and LUN Mapping

Test ObjectiveVerify that host-to-LUN access exists with valid zoning.

Test Configuration1. Create an FC zone on the fabric with the initiator and target WWNs.

2. Create host groups and LUNs on the array with access to the initiator WWN.

Result ValidationVerify that LUNs are discovered on the hosts with host-specific tools.

• Linux: Check the output of lsscsi.

• Windows: Check the output of Computer Management > Storage > Disk Management.

• VMware: Check the output at Configuration > Storage > Devices.

Test ResultsPASS. The host has read/write access to the presented LUNs.

1.3—Storage Device Fabric I/O Integrity

Test ObjectiveValidate single path host-to-LUN I/O with write/read/verify testing.

Test Configuration1. Set up read/write I/O to LUN using Medusa.

2. Perform link disruptions with port-toggles, cable pulls.

Result ValidationCheck Medusa I/O logs and verify that I/O resumes after a short downtime. Medusa I/O may pause, but should recover without errors.

Test ResultsPASS. I/O resumes without errors.



1.4—Storage Device Multipath Configuration – Path Integrity

Test ObjectiveVerify that multipath configures successfully, and verify the integrity of each path.

Test Configuration1. Set up the host with at least two initiator ports zoned with two target ports on the array.

2. Set up multipath on host.

3. Start I/O.

4. Isolate paths individually, and run I/O on the separate paths.

Result Validation1. Check the host multipath properties to verify that the toggled path recovers.

• Windows: mpclaim -s -d

• Linux: multipath -ll

• VMware: Check the paths at Configuration > Storage> Devices > Manage Paths

2. Check the host and storage logs for any failures.

3. Check the switch error logs and switch port status after the toggle.

portshow 24portIndex: 24portName: port24portHealth: HEALTHY



4. Check I/O logs, and verify that I/O continues without errors.

Test ResultsPASS. Each individual path is valid, and the paths combine successfully in a multipath configuration.



2. Fabric – Advanced Functionality

2.1—Storage Device Bottleneck Detection Using MAPS FPI – withCongested Host

Test ObjectiveVerify congestion on host ports is detected. Verify storage device and fabric behavior during congestion.

Test Configuration1. Configure MAPS and FPI on all switches. (More info in Task 1. Brocade FC Fabric Configuration on page 7.)

2. Start I/O from single host initiator to multiple targets.

3. Monitor switch logs for IO_PERF_IMPACT/ IO_FRAME_LOSS warnings.

Result ValidationCheck switch error logs and MAPS dashboard for bottleneck warnings.

RASLog

2016/09/14-15:24:04, [MAPS-1003], 2655, FID 128, WARNING, B6510_066_074, port26, F-Port 26, Condition=ALL_PORTS (DEV_LATENCY_IMPACT==IO_PERF_IMPACT), Current Value:[ DEV_LATENCY_IMPACT,IO_PERF_IMPACT, (147.8% of 10 secs) ], RuleName=defALL_PORTS_IO_PERF_IMPACT, Dashboard Category=Fabric Performance Impact.

MAPS Dashboard

> mapsdb --show

Email

Switch Time: Sep 14 15:24:04Affected Entity: port26, F-Port 26Monitor: Device Impact latency (DEV_LATENCY_IMPACT)Rule Name: defALL_PORTS_IO_PERF_IMPACTGroup: ALL_PORTSCondition: ALL_PORTS(DEV_LATENCY_IMPACT==IO_PERF_IMPACT)Current Value: IO_PERF_IMPACTDashboard Category: Fabric Performance Impact Switch Name: B6510_066_074Switch WWN: 10:00:00:05:33:a5:bf:86Switch IP: 10.38.66.74Fabric Name: SSRVFID: 128Additional Info: (147.8% of 10 secs)



Test ResultsPASS. With MAPS reporting configured, performance warnings are reported as expected. The bottlenecked ports are displayed on theMAPS dashboard and also a RASLog warning is created.

2.2—Storage Device Bottleneck Detection Using MAPS FPI – withCongested Fabric

Test ObjectiveValidate bottleneck detection with congested fabric.

Test Configuration1. Configure MAPS on all switches. (More info in Task 1. Brocade FC Fabric Configuration on page 7.)

2. Isolate single ISL in the fabric.

3. Start I/O from multiple host initiators to multiple targets.

4. Monitor switch logs for Traffic Performance warnings.

Result ValidationCheck switch error logs and MAPS dashboard for bottleneck warnings.

RASLog

2016/09/14-15:24:04, [MAPS-1003], 2655, FID 128, WARNING, B6510_066_074, port26, F-Port 26, Condition=ALL_PORTS (DEV_LATENCY_IMPACT==IO_PERF_IMPACT), Current Value:[ DEV_LATENCY_IMPACT,IO_PERF_IMPACT, (147.8% of 10 secs) ], RuleName=defALL_PORTS_IO_PERF_IMPACT, Dashboard Category=Fabric Performance Impact.

MAPS Dashboard

> mapsdb --show

Email

Switch Time: Sep 14 15:24:04Affected Entity: port26, F-Port 26Monitor: Device Impact latency (DEV_LATENCY_IMPACT)Rule Name: defALL_PORTS_IO_PERF_IMPACTGroup: ALL_PORTSCondition: ALL_PORTS(DEV_LATENCY_IMPACT==IO_PERF_IMPACT)Current Value: IO_PERF_IMPACTDashboard Category: Fabric Performance Impact Switch Name: B6510_066_074Switch WWN: 10:00:00:05:33:a5:bf:86Switch IP: 10.38.66.74Fabric Name: SSRVFID: 128Additional Info: (147.8% of 10 secs)



Test ResultsPASS. With MAPS reporting configured, performance warnings are reported as expected.

2.3—Storage Device – IO Insight Latency Monitoring with MAPS Alerts

Test ObjectiveConfigure a latency monitoring threshold on target port, and verify alerts are generated when threshold is exceeded.

Test ConfigurationRun an IO workload generator with a 4k read workload against a target LUN, and record the latency reading as a baseline. Using Medusawe recorded an average 78.661 uS latency.

ssr067168:~/ssr067168_multitarget # pain -f/dev/sdak -r -u -n -N -t1 -b4kAvg Completion Time in Seconds=0.000078661 (78.661 uS)

Configure a flow to monitor array target port traffic.

> flow --create ios_pure_1 -dstdev e02200 -egrport 34 -feature mon

Import the flow into MAPS.

> mapsconfig --import ios_pure_1

Create a custom policy.

> mapspolicy –clone dflt_aggressive_policy –name ios_base_policy

Create a rule to monitor and generate alerts at a threshold level higher than the baseline. The units for the command are microseconds,so in this example is 750µs.

> mapsrule --create ios_pure_latency -group ios_pure_1 -monitor RD_STATUS_TIME_LT_8K -timebase min -op g -value 750 -action raslog,email

Add the rule to the custom policy, and enable the policy.

> mapspolicy --addrule ios_base_policy -rulename ios_pure_latency> mapspolicy --enable ios_base_policy

Confirm the flow is active.

> flow --show ios_pure_1=====================================================================Name : ios_pure_1 Features: mon(Activated) noConfig: OffDefinition: EgrPort(34),DstDev(0xe02200)

Flow Monitor (Activated):Monitor time: | Mon Jan 23 15:57:29 MST 2017 |

Adjust the traffic pattern to cause a rise in latency above the configured monitoring threshold, and confirm RASLog, MAPS Dashboard,and email notifications are generated.

RASLog

2017/01/24-14:39:00:752877, [MAPS-1003], 25354/19474, FID 128, WARNING, G620_066_224,Flow(ios_pure_1),Condition=ios_pure_1(RD_STATUS_TIME_LT_8K/min>750),Current Value:[RD_STATUS_TIME_LT_8K, 1910 Microseconds], RuleName=ios_pure_latency, Dashboard Category=Traffic Performance., raslogAction.c, line: 99, comp:md, ltime:2017/01/24-14:39:00:752620



2017/01/24-14:39:41:221163, [CONF-5010], 25355/0, FID 128, WARNING, G620_066_224, key ether.link.mode is obsolete, newdb.c, line: 84, comp:weblinker, ltime:2017/01/24-14:39:41:220907

Email

Switch Time: Jan 24 14:39:00Affected Entity: Flow(ios_pure_1)Monitor: Read completion time (RD_STATUS_TIME_LT_8K)Rule Name: ios_pure_latencyGroup: ios_pure_1Condition: ios_pure_1(RD_STATUS_TIME_LT_8K/min>750)Current Value: 1910 MicrosecondsDashboard Category: Traffic Performance Severity: WARNINGSwitch Name: G620_066_224Switch WWN: 10:00:c4:f5:7c:41:1b:14Switch IP: 10.38.66.224Fabric Name: SSRVFID: 128

MAPS Dashboard

Test ResultsPASS. Verified I/O stats are reported and alerts are generated when thresholds are exceeded.

2.4—Storage Device – QoS Integrity with QoS Zone Based TrafficPrioritization

Test ObjectiveValidate QoS functionality.

Test Configuration1. Set up initiator-target pairs with Low/Medium/High QoS zones in the fabric.

2. Start I/O across all pairs, and validate traffic priority.

Result Validation1. Check I/O logs, and verify that I/O continues without any errors.

> zoneshowzone: QOSH_ssr067172_purezone: QOSL_ssr067178_purezone: QOSM_ssr067176_purezone: QOSH_ssr067172_pure zone: QOSL_ssr067178_pure zone: QOSM_ssr067176_pure



2. Check switch error logs and switch port status for any errors.

3. Verify traffic prioritization by examining performance and VC utilization.

> portregshow 36 | grep bbc_trcLow0x88c02800: bbc_trc 4 0 2 2 2 2 1 10x88c02820: bbc_trc 0 0 2 2 2 2 2 0

Med0x88d82800: bbc_trc 4 0 2 2 -3 -3 1 10x88d82820: bbc_trc 2 2 2 2 2 2 2 0

High0x88c02800: bbc_trc 4 0 2 2 2 2 1 10x88c02820: bbc_trc 2 2 2 2 -1 2 2 0

Test ResultsPASS. I/O completes without errors in low, medium, and high priority zones.

2.5—Storage Device – QoS Integrity with CS_CTL Based TrafficPrioritization

Test ObjectiveValidate CS_CTL QoS functionality.

Test Configuration1. Enable ExpressLane on Emulex adapter driver by setting value to '1', and set priority to '3' for 'high'.



FIGURE 7 Configuring Driver Parameters in Emulex OneCommand Interface

2. Enable CS_CTL QoS 'Auto-Mode' at the chassis level on all switches in the Brocade fabric.

> configurechassisConfigure...cfgload attributes (yes, y, no, n): [no]Custom attributes (yes, y, no, n): [no] system attributes (yes, y, no, n): [no] fos attributes (yes, y, no, n): [no] y

Reboot needed to effect new CSCTL Mode CSCTL QoS Mode (0 = default; 1 = auto mode): (0..1) [0] 1



To verify:

B6510_066_073:> configshow -all | grep csctlfos.csctlMode:1

3. Enable CS_CTL QoS on initiator and target ports.

B6510_066_073:> portcfgqos --enable 7 csctl_modeEnabling CSCTL mode flows causes QoS zone flows to lose priority on such ports.

Do you want to proceed?(y/n):y

To verify:

B6510_066_073:> portcfgshow 7 | grep -i csctlCSCTL mode: ON

4. Enable ExpressLane QoS on selected LUNs. For multipath LUNs, enable ExpressLane on each device path:



5. Start high throughput IO to all LUNs.

Result Validation1. Check I/O logs and verify I/O continues without any errors for all LUNs.

2. Verify I/O performance is improved on ExpressLane enabled LUN.

3. Verify CS_CTL prioritization in fabric by monitoring the high VC buffer credits on the ISLs.

4. Check switch error logs and switch port status for any errors.

Test ResultsPASS. Prioritized LUNs display higher performance, the traffic receives prioritization through the fabric, and IO completes without errors.

2.6—Storage Device — FC Protocol Jammer Test Suite

Test ObjectivePerform FC Jammer Tests including areas such as: CRC corruption, packet corruption, missing frame, host error recovery, target errorrecovery.

Test Configuration1. Insert Jammer device in the I/O path on the storage link.

2. Execute the following Jammer scenarios:

• Delete one frame

• Delete R_RDY

• Replace CRC of data frame

• Replace EOF of data frame

• Replace "good status" with "check condition"

• Replace IDLE with LR

• Truncate frame

• Create S_ID/D_ID error of data frame

3. Verify Jammer operations and recovery with Analyzer.



Result ValidationCheck Medusa log and verify IO recovers and completes without errors.

Test ResultsPASS. I/O recovers as expected.

3. Stress and Error Recovery with Device Multipath

3.1—Storage Device Fabric I/O Integrity - Congested Fabric

Test ObjectiveValidate I/O integrity in a congested fabric environment.

Test ConfigurationFrom all initiators start a mixture of READ, READ/WRITE, and WRITE traffic continuously to all their targets for a 24-hour period.

Result Validation1. Check the host and storage logs for any errors.

2. Verify the link congestion and check the switch logs for any errors.

3. Check I/O generator tool logs to verify that I/O runs without errors.

Test ResultsPASS. In a congested fabric, I/O runs successfully without errors or failover.



3.2—Storage Device Nameserver Integrity – Device Recovery with PortToggle

Test ObjectiveValidate path recovery behavior on storage and host ports.

Test Configuration1. Set up multipath on the host, and start I/O.

2. Perform multiple iterations of sequential port toggles across initiator and target switch ports.

Result Validation1. Check the switch port status after the toggle, and check for any errors in the switch error logs.




2. Check the host multipath status on hosts to verify that the toggled path recovers.



• VMware: Check the paths at Configuration > Storage > Devices > Manage Paths

3. Check host and storage error logs, and verify that I/O continues without errors.

Test ResultsPASS. Failover between 8 logical paths (2 host x 4 storage) tested successfully. Paths recover, and I/O completes without errors.



3.3—Storage Device Nameserver Integrity – Device Recovery with DeviceRelocation

Test ObjectiveValidate storage device path recovery after physical port relocation.

Test ConfigurationSequentially performed for each storage device port.

Disconnect and reconnect the port to a different switch port in the same fabric.

Result Validation1. Check for any errors in the switch error logs and the switch port status at the new switch port.




2. Check host multipath status on hosts to verify that the toggled path recovers.



• VMware: Check the paths at Configuration > Storage > Devices > Manage Paths

3. Confirm that there are no errors in the Medusa log.

Test ResultsPASS. The physical move of the storage port shows successful recovery. Paths recover, and I/O completes without errors.



3.4—Storage Device – Device Recovery with Device Port Toggle –Extended Run

Test ObjectiveValidate path recovery behavior on storage and host ports for an extended duration run.


2. Sequentially toggle each initiator and target port in the fabric (multiple iterations).

3. Run the test for an extended period.

Result Validation1. Check the switch port status after the toggle, and check for any errors in the switch logs.

portshow 24portIndex: 24portName: port24portHealth: HEALTHY



2. Check host multipath properties for iSCSI hosts to verify the toggled path recovers.

On Windows: mpclaim -s -d

On Linux: multipath -ll

On VMware: Check the paths at Configuration > Storage> Devices > Manage Paths

3. Check the host and storage error logs, and verify that I/O continues without errors.

Test ResultsPASS. 48-hour run; paths recover and I/O completes without errors.



3.5—Storage Device Recovery - ISL Port Toggle

Test ObjectiveValidate path recovery and I/O integrity when ISL links are disabled.


2. Sequentially toggle each ISL path (one at a time) on all switches.

Result Validation1. Check the FC fabric status after the ISL toggle. Verify that all nodes are online.

> fabricshowSwitch ID Worldwide Name Enet IP Addr FC IP Addr Name------------------------------------------------------------------------- 1: fffc01 50:00:53:35:b1:d3:df:5c 0.0.0.0 0.0.0.0 "fcr_xd_1_40" 4: fffc04 50:00:53:35:b1:d3:df:5e 0.0.0.0 0.0.0.0 "fcr_xd_4_50" 73: fffc49 10:00:00:05:33:13:95:9a 10.38.66.73 0.0.0.0 >"B6510_066_073" 74: fffc4a 10:00:00:05:33:a5:bf:86 10.38.66.74 0.0.0.0 "B6510_066_074" 82: fffc52 10:00:00:05:33:13:96:5a 10.38.66.82 0.0.0.0 "B6510_066_082" 83: fffc53 10:00:00:05:33:5b:1d:1d 10.38.66.83 0.0.0.0 "B6510_066_083"223: fffcdf 10:00:c4:f5:7c:2a:8b:c8 10.38.66.223 0.0.0.0 "G620_066_223"224: fffce0 10:00:c4:f5:7c:41:1b:14 10.38.66.224 0.0.0.0 "G620_066_224The Fabric has 8 switchesFabric Name: SSR

2. Check the switch logs for any errors, and verify that the I/O failed over to an alternate ISL path in the fabric.


Test ResultsPASS. Paths recover, and I/O completes without errors.

3.6—Storage Device Recovery – ISL Port Toggle (Entire Switch)

Test ObjectiveValidate path recovery and I/O integrity when all ISL links on a switch are disabled.



Test Configuration1. Ensure ISL redundancy by provisioning multiple ISLs connected to different switches to provide multiple paths through the

fabric.

2. Set up multipath on the host, and start I/O.

3. On each switch, disable all ISL links at once.

Result Validation1. Check the FC fabric status after the ISL toggle. Verify that all nodes are online.

fabricshowSwitch ID Worldwide Name Enet IP Addr FC IP Addr Name------------------------------------------------------------------------- 1: fffc01 50:00:53:35:b1:d3:df:5c 0.0.0.0 0.0.0.0 "fcr_xd_1_40" 4: fffc04 50:00:53:35:b1:d3:df:5e 0.0.0.0 0.0.0.0 "fcr_xd_4_50" 73: fffc49 10:00:00:05:33:13:95:9a 10.38.66.73 0.0.0.0 >"B6510_066_073" 74: fffc4a 10:00:00:05:33:a5:bf:86 10.38.66.74 0.0.0.0 "B6510_066_074" 82: fffc52 10:00:00:05:33:13:96:5a 10.38.66.82 0.0.0.0 "B6510_066_082" 83: fffc53 10:00:00:05:33:5b:1d:1d 10.38.66.83 0.0.0.0 "B6510_066_083"223: fffcdf 10:00:c4:f5:7c:2a:8b:c8 10.38.66.223 0.0.0.0 "G620_066_223"224: fffce0 10:00:c4:f5:7c:41:1b:14 10.38.66.224 0.0.0.0 "G620_066_224"

The Fabric has 8 switchesFabric Name: SSR

2. Check the switch logs for any errors and verify I/O failed over to alternate ISL path in the fabric.


Test ResultsPASS. Paths recover, and I/O completes without errors.

3.7—Storage Device Recovery – Switch Offline

Test ObjectiveValidate path recovery and I/O integrity during switch offline events.

Test Configuration1. Toggle each switch in sequential order.

2. Include switch enable/disable, power on/off, and reboot testing.



Result Validation1. Check the FC fabric status after the switch toggle. Verify that all nodes are present in the fabric.

fabricshowSwitch ID Worldwide Name Enet IP Addr FC IP Addr Name------------------------------------------------------------------------- 1: fffc01 50:00:53:35:b1:d3:df:5c 0.0.0.0 0.0.0.0 "fcr_xd_1_40" 4: fffc04 50:00:53:35:b1:d3:df:5e 0.0.0.0 0.0.0.0 "fcr_xd_4_50" 73: fffc49 10:00:00:05:33:13:95:9a 10.38.66.73 0.0.0.0 >"B6510_066_073" 74: fffc4a 10:00:00:05:33:a5:bf:86 10.38.66.74 0.0.0.0 "B6510_066_074" 82: fffc52 10:00:00:05:33:13:96:5a 10.38.66.82 0.0.0.0 "B6510_066_082" 83: fffc53 10:00:00:05:33:5b:1d:1d 10.38.66.83 0.0.0.0 "B6510_066_083"223: fffcdf 10:00:c4:f5:7c:2a:8b:c8 10.38.66.223 0.0.0.0 "G620_066_223"224: fffce0 10:00:c4:f5:7c:41:1b:14 10.38.66.224 0.0.0.0 "G620_066_224"


2. Check the switch logs for any errors, and verify that the toggled switch has recovered.

> switchshowswitchName: B6510_066_088switchType: 109.1switchState: OnlineswitchMode: NativeswitchRole: SubordinateswitchDomain: 88switchId: fffc58switchWwn: 10:00:00:27:f8:06:23:28zoning: ON (SSR)switchBeacon: OFFFC Router: ONFC Router BB Fabric ID: 100Address Mode: 0Fabric Name: SSR_2


Test ResultsPASS. Paths fail over and recover, and I/O completes without errors.

3.8—Storage Device Recovery – Switch Firmware Download

Test ObjectiveVerify that I/O continues with minimal disruption throughout the switch firmware upgrade process.

Test Configuration1. Set up host multipath with links on different switches in the FC fabric, and start I/O.

2. Sequentially perform firmware upgrades on all switches in the fabric.



Result Validation1. Verify that the firmware upgrade completes successfully on each switch node and that they merge back in the FC fabric.

> versionKernel: 2.6.14.2Fabric OS: v8.1.0Made on: Mon Dec 12 14:23:27 2016Flash: Fri Jan 13 01:04:29 2017BootProm: 1.0.11> fabricshowSwitch ID Worldwide Name Enet IP Addr FC IP Addr Name------------------------------------------------------------------------- 1: fffc01 50:00:53:35:b1:d3:df:5c 0.0.0.0 0.0.0.0 "fcr_xd_1_40" 4: fffc04 50:00:53:35:b1:d3:df:5e 0.0.0.0 0.0.0.0 "fcr_xd_4_50" 73: fffc49 10:00:00:05:33:13:95:9a 10.38.66.73 0.0.0.0 >"B6510_066_073" 74: fffc4a 10:00:00:05:33:a5:bf:86 10.38.66.74 0.0.0.0 "B6510_066_074" 82: fffc52 10:00:00:05:33:13:96:5a 10.38.66.82 0.0.0.0 "B6510_066_082" 83: fffc53 10:00:00:05:33:5b:1d:1d 10.38.66.83 0.0.0.0 "B6510_066_083"223: fffcdf 10:00:c4:f5:7c:2a:8b:c8 10.38.66.223 0.0.0.0 "G620_066_223"224: fffce0 10:00:c4:f5:7c:41:1b:14 10.38.66.224 0.0.0.0 "G620_066_224"


2. Check I/O generator tool logs to verify that I/O runs without errors throughout the firmware upgrade.

3. Check the switch logs for any errors, and verify that I/O resumes on the node after the firmware upgrade is complete.

Test ResultsPASS. I/O failover and recovery are successful during the firmware download. Paths recover, and I/O completes without errors.

4. Storage Device – Fibre Channel Routing (FCR)Internetworking Tests(Testing covered in 8.1.0 and previous releases)

4.1—Storage Device Internetworking Validation with FC Host

Test ObjectiveValidate that storage targets are imported successfully in a routed Fibre Channel environment.

Test Configuration1. Set up FCR in an Edge-Backbone-Edge configuration.

2. Set up LSAN zoning, verify host access to target LUNs, and start I/O.

4. Storage Device – Fibre Channel Routing (FCR) Internetworking Tests


Result Validation1. Verify the name server and FCR fabric state.

> fcrfabricshowFC Router WWN: 10:00:00:05:33:13:96:5a, Dom ID: 82,Info: 10.38.66.82, "B6510_066_082" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ------------------------------------------------------------------------ 40 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088" 41 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92" 26 50 10.38.66.112 10:00:50:eb:1a:20:d3:81 "VDX6740_066_112" 27 50 10.38.66.111 10:00:50:eb:1a:22:27:da "VDX6740_066_111"

FC Router WWN: 10:00:00:05:33:5b:1d:1d, Dom ID: 83,Info: 10.38.66.83, "B6510_066_083" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ------------------------------------------------------------------------ 40 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92" 41 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088" 26 50 10.38.66.111 10:00:50:eb:1a:22:27:da "VDX6740_066_111" 27 50 10.38.66.112 10:00:50:eb:1a:20:d3:81 "VDX6740_066_112"

> fcrproxydevshow Proxy WWN Proxy Device Physical State Created PID Exists PIDin Fabric in Fabric------------------------------------------------------------------- 40 10:00:00:05:1e:60:b4:6b 02ff02 100 132400 Imported 40 10:00:00:05:1e:60:b4:6c 02fe02 100 032400 Imported 40 50:02:4f:40:55:82:01:00 02f906 100 532400 Imported 40 50:02:4f:40:55:82:01:01 02fa06 100 df2400 Imported 40 50:02:4f:40:55:82:02:00 02fb06 100 532500 Imported 40 50:02:4f:40:55:82:02:01 02fc06 100 df2500 Imported 50 50:02:4f:40:55:82:01:00 02fc03 100 532400 Imported 50 50:02:4f:40:55:82:01:01 02fd03 100 df2400 Imported 50 50:02:4f:40:55:82:02:00 02fe03 100 532500 Imported 50 50:02:4f:40:55:82:02:01 02ff03 100 df2500 ImportedTotal devices displayed: 20

2. Verify that I/O runs successfully without any errors.

Test ResultsPASS. All devices are available via the routed fabric. Running I/O confirms successful routing.

4.2—Storage Device Internetworking Validation with FCoE Test Using VDXFlexPort

Test ObjectiveVerify storage device availability from FCoE initiators routed through the VCS fabric.

Test Configuration1. Set up FCoE host initiator ports on VCS fabric using Brocade VDX 6740 switches.

2. Enable Fibre Channel support on VDX switch ports (more detail in the test bed configuration section).

3. Connect the VCS fabric to the FC fabric via FCR ports.



4. Configure LSAN zones in VCS and Fibre Channel fabrics.

Result Validation1. Verify the name server and FCR fabric state.

> fcrfabricshowFC Router WWN: 10:00:00:05:33:13:96:5a, Dom ID: 82,Info: 10.38.66.82, "B6510_066_082" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ------------------------------------------------------------------------ 40 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088" 41 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92" 26 50 10.38.66.112 10:00:50:eb:1a:20:d3:81 "VDX6740_066_112" 27 50 10.38.66.111 10:00:50:eb:1a:22:27:da "VDX6740_066_111"

FC Router WWN: 10:00:00:05:33:5b:1d:1d, Dom ID: 83,Info: 10.38.66.83, "B6510_066_083" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ------------------------------------------------------------------------ 40 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92" 41 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088" 26 50 10.38.66.111 10:00:50:eb:1a:22:27:da "VDX6740_066_111" 27 50 10.38.66.112 10:00:50:eb:1a:20:d3:81 "VDX6740_066_112""VDX6740_066_119"


2. Verify that I/O runs successfully without any errors.

Test ResultsPASS. Storage targets are available through the FCoE/FC routed configuration; I/O completes without errors.

4.3—Storage Device Edge Recovery after FCR Disruptions

Test ObjectiveValidate that storage target paths recover successfully from disruptions in a routed Fibre Channel environment. Edge-Backbone-Edgeconfiguration.

Test Configuration1. Set up FCR in an Edge-Backbone-Edge configuration.

2. Set up LSAN zoning.



3. With I/O running, perform sequential reboots, switch disables, and ISL port toggles on the switches in the backbone fabric.

Result Validation1. Verify the FCR fabric state throughout the disruptions.

fcrfabricshowFC Router WWN: 10:00:00:05:33:13:96:5a, Dom ID: 82,Info: 10.38.66.82, "B6510_066_082" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ----------------------------------------------------------------- 40 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088" 41 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92" 26 50 10.38.66.112 10:00:50:eb:1a:20:d3:81 "VDX6740_066_112" 27 50 10.38.66.111 10:00:50:eb:1a:22:27:da "VDX6740_066_111"

FC Router WWN: 10:00:00:05:33:5b:1d:1d, Dom ID: 83,Info: 10.38.66.83, "B6510_066_083" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ----------------------------------------------------------------- 40 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92" 41 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088" 26 50 10.38.66.111 10:00:50:eb:1a:22:27:da "VDX6740_066_111" 27 50 10.38.66.112 10:00:50:eb:1a:20:d3:81 "VDX6740_066_112"


2. Check the switch logs for errors.

3. Check the host and storage logs, and verify that I/O runs without errors.

Test ResultsPASS. Paths fail over and recover with FCR disruptions, and I/O completes without errors.

4.4—Storage Device Backbone Recovery after FCR Disruptions

Test ObjectiveValidate that storage target paths recover successfully from disruptions in a routed Fibre Channel environment. Edge-Backboneconfiguration.

Test Configuration1. Set up FCR in an Edge-Backbone configuration.



2. Set up LSAN zoning.

3. With I/O running, perform sequential reboots, switch disables, and ISL port toggles on the switches in the backbone fabric.

Result Validation1. Verify the FCR fabric state throughout the disruptions.

> fcrfabricshowFC Router WWN: 10:00:00:05:33:13:96:5a, Dom ID: 82,Info: 10.38.66.82, "B6510_066_082" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ----------------------------------------------------------------- 40 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088" 41 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92" 26 50 10.38.66.112 10:00:50:eb:1a:20:d3:81 "VDX6740_066_112" 27 50 10.38.66.111 10:00:50:eb:1a:22:27:da "VDX6740_066_111"

FC Router WWN: 10:00:00:05:33:5b:1d:1d, Dom ID: 83,Info: 10.38.66.83, "B6510_066_083" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ----------------------------------------------------------------- 40 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92" 41 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088" 26 50 10.38.66.111 10:00:50:eb:1a:22:27:da "VDX6740_066_111" 27 50 10.38.66.112 10:00:50:eb:1a:20:d3:81 "VDX6740_066_112"


2. Check the switch logs for errors.

3. Check the host and storage logs, and verify that I/O runs without any errors.

Test ResultsPASS. Paths fail over successfully, and I/O completes without errors.

5. Optional/Additional Tests

5.1—Storage Device Firmware Update

Test ObjectiveExecute a non-disruptive firmware update on the array while running I/O, and confirm that there are no I/O errors.



Test Configuration1. Run continuous I/O to the array.

2. Execute code update procedure as described in vendor documentation.

Result Validation1. Confirm the updated version on all array components.

2. Check the I/O generator tools logs to verify that I/O completes without error.

3. Check the host and storage logs for any errors throughout the I/O operations.

4. Check the switch error logs and port stats for any errors or I/O drops.

Test ResultsPASS. Update successful with proper failover and no I/O errors.

5.2—Performance I/O Testing with Varying Workload Dimensions

Test ObjectiveRun I/O workload generation with varying dimensions (block size, read/write/mix), and verify that performance characteristics are asexpected.

Test Configuration1. Run an I/O loop at block transfer sizes of 512, 1k, 2k, 4k, 8k, 16k, 32k, 64k, 128k, 256k, 512k, and 1m.

2. Include a nested loop of 100% read, 100% write, and 50% read/write.

3. Repeat the test for the following configurations:

• 1 host port to 1 target port (single path)

• 2 host ports to multiple target ports (multipath)

• Multiple hosts to multiple target ports (multihost multipath)

Result Validation1. Check the I/O generator tools logs to verify that I/O completes without any errors.





Test ResultsPASS. All workload runs are monitored at the host, storage, and fabric, and they complete without any I/O errors or faults. Performancebehavior is as expected.

5.3—VMware IOAnalyzer Workload Test

Test ObjectiveRun the workload test suite including varying application workloads generated from multiple VMs.

Test Configuration1. Configure a 2-host VMware cluster with multipath on 2 initiator ports per host, and 4 target ports.

2. Configure workload generation from 8 worker VMs using VMware IOAnalyzer.

3. Run a variety of application IO workload patterns from the IOAnalyzer suite. For this round of testing, the following workloadswere used:

• Workstation

• Webserver

• Video on Demand

• Max Write Throughput

• Max Write IOPS

• Max IOPS

• Max Throughput

• SQL Server 16k

• 4k Read

• Exchange 2007

• OLTP 4k

Result Validation1. Check the I/O generator tools logs to verify that I/O completes without any errors.





Test ResultsPASS. All workload runs are monitored at the host, storage, and fabric and complete without any I/O errors or faults. Performancebehavior is as expected.




Test Conclusions• Achieved a 100% pass rate on all test cases in the SFR qualification test plan. The network and the storage were able to handle

the various stress and error recovery scenarios without any issues.

• Different I/O workload scenarios were simulated using Medusa and VMware IOAnalyzer tools, and sustained performancelevels were achieved across all workload types.

• The results confirm that the Pure FlashArray//20 array interoperates seamlessly with Brocade FC fabrics, and high availabilityand sustained performance are demonstrated.

• The Brocade Gen 5 (16Gb) and Gen 6 (32Gb) FC switches were able to handle the sustained throughput and latencyperformance requirements efficiently with fewer ISL trunks. Multiple ISLs to different switches in the fabric should be set up toprovide path redundancy through the fabric.

• We recommend enabling the Monitoring and Alerting Policy Suite (MAPS) health monitor on all switches in the FC fabric toreport fabric-wide events and traffic performance metrics. The MAPS feature of Fabric Performance Impact monitoring isenabled by default and should be used to detect fabric bottlenecks in the form of timeouts and latency.

• Implementing the IO Insight feature to monitor the critical flows at the SCSI level can provide valuable data and insight intotraffic performance across the fabric. Combining IO Insight with MAPS custom rules and alerts can provide proactivemonitoring to help preserve the investment in flash storage performance.

• QoS zoning can be used to classify host-target traffic into high-, medium-, or low-priority zones; to provide traffic prioritizationthrough the FC fabric for the desired host-target pair by allocating more resources to the traffic in the higher priority zone.

• Enabling Emulex ExpressLane on a LUN provides prioritized queuing on the HBA for traffic to that LUN and also sets theCS_CTL tag on the frame, which allows the traffic to be prioritized through the FC fabric based on the value of the CS_CTL tagand the corresponding priority level.

• For optimal availability and performance, consideration should be given to multipath configuration on the host side. WhileWindows 2008 and 2012 will provide round-robin behavior by default, Linux systems will benefit from adding a custom entryto /etc/multipath.conf, and VMware hosts systems should be changed from the default "Most Recently Used (VMware)" settingto "Round-Robin (VMware)". Actively using all available paths provides a significant improvement in performance throughput.
