Violin Memory Interoperability Best Practices Guide

ViolinMEMORY

Violin MemoryInteroperability Best Practices Guide

Document Number: 535-0111-00 Rev 07January 2016

LEGAL NOTICE

Copyright © 2010-2016 Violin Memory, Inc. All rights reserved.

Violin, Violin Memory and the Violin logo are registered trademarks of Violin. A complete list of Violin’s trademarks and registered trademarks is available at www.violin-memory.com/company/trademarks/

All other brands, product names, company names, trademarks, and service marks are the properties of their respective owners.

Violin Memory, Inc.4555 Great America ParkwaySanta Clara, CA 95054USA

ABSTRACT

This document is a collection of best practices to follow when connecting a host to a Violin array. It covers the interconnect protocols supported by Violin devices: Fibre Channel (FC), iSCSI, and InfiniBand (IB). For each interconnect protocol, this document includes suggested settings and procedures for configuring the host operating system.

ii Violin Memory Interoperability Best Practices Guide

http://www.violin-memory.com/company/trademarks/

Contents

About This Guide. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

CHAPTER 1. Violin 6000 Series All Flash Array Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Fibre Channel Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Zoning Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3Brocade Fabric Best Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

Fibre Channel Direct Connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4Setting Host-Side HBA Connection Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4Setting Array-Side Connection Type. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4

Fibre Channel Boot from SAN (BFS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5x86 Boot from SAN Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5Windows 2012 & 2008 R2 (SP1) BFS Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7RHEL 5.x & 6.x BFS Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8AIX 6.1 & 7.1 BFS Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9

iSCSI Target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16Enabling iSCSI on the Array . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16Creating an iSCSI Target on the Array . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18

CHAPTER 2. Concerto 7700 Flash Storage Platform Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . 21Fibre Channel Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Concerto 7700 FSP Zoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21Concerto 7700 FSP Southbound Path Status. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21Concerto 7700 FSP Fibre Channel Target Mode Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21Brocade Fabric Best Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22

Fibre Channel Boot from SAN (BFS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22x86 Boot from SAN Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22Windows 2012 & 2008 R2 (SP1) BFS Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25RHEL 5.x & 6.x BFS Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26

SLES11 BFS Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26iSCSI Target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

iSCSI Target Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27

CHAPTER 3. 7000 Series Flash Storage Platform Best Practices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29Fibre Channel Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

7000 FSP Zoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .297000 Series FSP Fibre Channel Target Mode Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .29Brocade Fabric Best Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .29

Fibre Channel Boot from SAN (BFS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30x86 Boot from SAN Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30Windows 2012 & 2008 R2 (SP1) BFS Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33RHEL 5.x & 6.x BFS Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33

SLES11 BFS Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34iSCSI Target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

iSCSI Target Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34

CHAPTER 4. Operating System Considerations for Fibre Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . 35Windows 2012 and Windows 2008 R2 (SP1). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Checking Paths. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35

Windows 2012 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35HBA Driver Parameter Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35Setting Up MPIO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35

Violin Memory Interoperability Best Practices Guide iii

Windows 2008 R2 (SP1). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39HBA Driver Parameter Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39Setting Up MPIO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Red Hat Enterprise Linux 5.x/6.x/7.x. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45Device Mapper Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45Symantec Veritas SFHA Solution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

Red Hat Enterprise Linux 5.x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48Device Mapper Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48Tuning Recommendations for Red Hat Enterprise Linux 5.x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

Red Hat Enterprise Linux 6.x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50Device Mapper Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50Tuning Recommendations for Red Hat Enterprise Linux 6.x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Red Hat Enterprise Linux 7.x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53SUSE Enterprise Linux 11 SP3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

Device Mapper Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53VMware ESXi 5.x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Fibre Channel Zoning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57Initiator Groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57RHEL 6.x Virtual Machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58Tuning Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

Citrix 6.x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59Device Mapper Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59Checking for Paths. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

AIX 6.1 & 7.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61For Violin 6000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61NACA LUN Creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62For Violin 7x00 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

PathManager Devices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64Solaris . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

For Violin 6000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65For Violin 7x00 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69Tuning Recommendations for Solaris SPARC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

HP-UX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71Checking Paths of Multipathed Disks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71Concerto 7x00 LUN Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72Exporting LUNs to HP-UX 11 Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72Discovering LUNs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73Checking Paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

SAN Appliances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75IBM SAN Volume Controller (SVC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75FalconStor Network Storage Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

Citrix XenServer 6.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79For Violin 7x00 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

CHAPTER 5. Operating System Considerations for iSCSI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81Windows 2012. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

iSCSI Initiator Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81Red Hat Enterprise Linux 6.x & 5.x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

For Violin 6000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87For Concerto 7100/7200 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90Checking for Paths. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92Symantec Veritas SFHA Solution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

CHAPTER 6. InfiniBand Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95Configuring the Host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

Downloading and Installing the Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95Configuring Driver Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95LUN Discovery Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

iv Violin Memory Interoperability Best Practices Guide

CHAPTER 7. iSER Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97Enabling iSER on the Array . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97Creating an iSER Target on the Array . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98Operating System Considerations for iSER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

Redhat Enterprise Linux 6.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .99

Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103Sample ‘/etc/infiniband/openib.conf’ Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

InfiniBand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .103iSER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .104

Mapping of iSCSI Ports to Ethernet Interfaces on the Memory Gateway. . . . . . . . . . . . . . . . . . . . . 107

Violin Memory Interoperability Best Practices Guide v

vi Violin Memory Interoperability Best Practices Guide

About This Guide

Intended Audience

This guide is intended for storage industry professionals seeking optimal configurations for Violin Memory storage deployments.

How to Use This Guide

This guide contains best practices for the supported interconnect protocols: Fibre Channel (FC), iSCSI, and InfiniBand (IB) on the Violin 6000 Series All Flash Array (Violin 6000), 7000 Series Flash Storage Platform, and Violin 7300 Flash Storage Platform (FSP). For each interconnect protocol, you will find common best practices for supported operating systems: Windows 2012, Windows 2008 R2, Red Hat Enterprise Linux (RHEL) 5, RHEL 6, RHEL 7, AIX 6.1, AIX 7.1, Solaris, HP-UX, IBM SVC and VMware.

Note that this document is meant to be a general guide only. Actual environments will differ from the ones listed in this document.

Not all versions of the operating systems listed in this document are supported. Contact Violin Memory Customer Support for the most current information about supported operating systems and configurations.

Violin Memory Interoperability Best Practices Guide 1

2 Violin Memory Interoperability Best Practices Guide

Fibre Channel Fabric

CHAPTER 1 Violin 6000 Series All Flash Array Best Practices

This chapter contains best practices that apply to the Violin 6000 Series All Flash Array (Violin 6000).


Zoning Best Practices

For zoning, Violin recommends one initiator port to many target ports. One-to-one zoning is also supported. This recommendation applies to Windows, RHEL, AIX, Solaris, and HP-UX. For ESXi zoning, see Fibre Channel Zoning on page 57.

For example:

• 1 Client, 1 HBA, 2 ports.

• Zone 1

pwwn of initiator - HBA port 0pwwn of target port 1 - i.e. (MG-A, HBA-A1)pwwn of target port 2 … … …pwwn of target port n

• Zone 2

pwwn of initiator - HBA port 1pwwn of target port 1 - i.e. (MG-A, HBA-A1)pwwn of target port 2… … …pwwn of target port n

Brocade Fabric Best Practice

For Brocade 8G switch ports connecting between initiator ports and target ports, Violin recommends that you set the fillword to mode 3 (aa-then-ia) using the portcfgfillword command.

The following fillword modes are supported on a Brocade 8G switch running FOS firmware 6.3.1a and later:

admin> portcfgfillwordUsage: portCfgFillWord PortNumber Mode [Passive]Mode: 0/-idle-idle - IDLE in Link Init, IDLE as fill word (default)1/-arbff-arbff - ARBFF in Link Init, ARBFF as fill word2/-idle-arbff - IDLE in Link Init, ARBFF as fill word (SW)3/-aa-then-ia - If ARBFF/ARBFF failed, then do IDLE/ARBFF


Chapter 1 - Violin 6000 Series All Flash Array Best Practices

Note: The 16G platform (Condor3) automatically detects which fillword it should use, so there is no need to use the portcfgfillword command.

Fibre Channel Direct Connect

Setting Host-Side HBA Connection Type

Use the HBA vendor’s utility within the running OS or the HBA BIOS utility during system boot to change the HBA connection setting to loop.

• If you are using an Emulex HBA, set the topology to loop first, or FC-AL.

• If you are using a QLogic HBA, set the connection type to loop preferred, otherwise point to point, or loop only.

Setting Array-Side Connection Type

Make sure that the array-side connection setting is loop-preferred.

To set loop-preferred on the MGs, log on to each MG and enter these commands:

# enable# configure terminal# fc target hba-a1 type loop-preferred

To set all ports on the array, enter this command:

# fc target all type loop-preferred

To check the connection type setting, enter the following command on each MG:

# show targets detailTarget: hba-a1 Node : 1 Hostname : violin-mg-b Protocol : fc Address : wwn.21:00:00:1b:32:94:a3:8f Enabled : yes State : Online Port Type Setting : loop-preferred Port Type : LPort (private loop) Speed Setting : Auto Speed : 8 Gbit

Note: Remember to set the connection type for each HBA port you would like to configure, as well as on each MG.


Fibre Channel Boot from SAN (BFS)


x86 Boot from SAN Example

The following is an example of a general x86 BFS case.

Note: Before you start this process, make sure that the Boot LUN has been exported to the host. If the host is running Windows, make sure to also only have one path to the Boot LUN. For RHEL, install the mpath package during OS installation to install RHEL on a Boot LUN with more than one path.

For more details about how to install RHEL on a BFS LUN with multipathing, refer to the RHEL documentation.

This example uses an x86 server, one QLogic QLE 2562 HBA, and one Violin array.

1. Watch the POST messages during server startup to determine how to access the HBA’s BIOS utility. The example below shows a POST message indicating you can access the HBA’s BIOS utility by pressing Ctrl-Q or Alt-Q.


https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/DM_Multipath/move_root_to_multipath.html



2. Enter the BIOS utility and enable the HBA BIOS (if not already enabled).

3. Find ‘Selectable Boot Settings’ in the BIOS, and enable Selectable Boot. Choose ‘Primary Boot Port Name, Lun’.



4. Choose the Violin LUN that you would like to use as the boot device.

5. After you have saved the changes and rebooted, you should see a POST message from the HBA card indicating that it has found the boot LUN. (If you are using both FC ports on the HBA, don’t forget to enable and configure the other port.)

Windows 2012 & 2008 R2 (SP1) BFS Notes

• The Boot LUN is exported to the host as LUN 0.

• The Boot LUN has only one path from the array to the host. This can be achieved by disabling a target port or initially setting up limited zoning. After configuring MPIO on the host, you may connect all paths from the Boot LUN to the array. This may include re-exporting the Boot LUN.

• Run the command ‘mpclaim -s -p’ to ensure that the BOOT LUN is controlled by MPIO. For example:

C:\Users\Administrator> mpclaim -s -d 0

MPIO Disk0: 08 Paths, Round Robin with Subset, Implicit Only

Controlling DSM: Microsoft DSM

SN: 601B970F29DAC9F29DAC9BF2DCD76



Supported Load Balance Policies: FOO RRWS LQD WP LB

Path ID State SCSI Address Weight

--------------------------------------------------------------

0000000077080003 Active/Unoptimized 008|000|003|000 0

TPG_State : Active/Unoptimized, TPG_Id: 2, : 4100

0000000077080002 Active/Optimized 008|000|002|000 0

* TPG_State : Active/Optimized, TPG_Id: 1, : 4













RHEL 5.x & 6.x BFS Notes

• Install the mpath package during installation of RHEL. Otherwise, you will need to use only one path during installation. Refer to the RHEL documentation for details. After configuring device mapper on the host, you can connect all paths from the Boot LUN to the array.

• Run the command ‘multipath -ll’ to ensure that the Boot LUN is controlled by device mapper multipathing. For example:

[violin@vmem-10-1-15-103 ~]# multipath -ll

mpatha (36001b970f29d0ac9f29d0ac9913baff0) dm-0 VIOLIN,SAN ARRAY

size=100G features='0' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

|- 1:0:0:0 sda 8:0 active ready running

|- 1:0:2:0 sdc 8:32 active ready running

|- 1:0:3:0 sdd 8:48 active ready running

|- 1:0:1:0 sdb 8:16 active ready running

|- 2:0:0:0 sde 8:64 active ready running

|- 2:0:1:0 sdf 8:80 active ready running

|- 2:0:2:0 sdg 8:96 active ready running

`- 2:0:3:0 sdh 8:112 active ready running




AIX 6.1 & 7.1 BFS Notes

The following procedure shows an example of installing AIX 7.1 on a Violin LUN through the SAN.

1. Boot the server and start SMS.

2. At the prompt, enter ‘5’ for “Select Boot Options”.



3. Enter ‘1’ for “Select Install/Boot Device”.

4. Enter ‘7’ for “List All Devices”.



5. Select the device that contains the installation image. In this example, it is device 6, the SATA CD-ROM drive.

6. Enter ‘2’ for “Normal Mode Boot”.



7. After following the on-screen instructions for terminal windows and language choice, the screen below is displayed. Enter ‘5’ for “Select Storage Adapters”.

8. Locate your Fibre Channel adapter in the list and enter its number to select the device. In this example, you would enter ‘2’ for “fscsi0”.



9. Go back to the “Installation and Maintenance” menu. This time, enter ‘2’ for “Change/Show Installation Settings and Install”.

10.Enter ‘1’ for “System Settings”.



11.Enter ‘1’ for “New and Complete Overwrite”.

12.The Boot LUN should appear on this screen. If you do not know which disk it is, choose one disk at a time and follow these next steps.

In this example, we know the Boot LUN is hdisk1. We know this because the size of the Boot LUN is 100 GB. Ensure only the BFS disk is selected before continuing. Enter ‘2’ for “hdisk1”. Then Enter ‘77’ for “Display More Disk Information”.



13.Enter ‘77’ again.

14.Looking at the location of the disk, we can see that the port WWN from the target port of the Violin array is in the location string (circled in red). We have now verified that we are indeed going to install on the Boot LUN presented by the Violin array. Enter ‘0’ for “Continue with choices indicated above”.



15.You can now proceed with a normal AIX installation.

iSCSI Target

Enabling iSCSI on the Array

For iSCSI port number identification, see Mapping of iSCSI Ports to Ethernet Interfaces on the Memory Gateway in the Appendix.

1. Log into the master MG.

2. Enable iSCSI.

[memgate-cluster: master] (config) # iscsi enable global

[memgate-cluster: master] (config) # wr memory

3. Use the ‘show targets’ command to check to make sure iSCSI is enabled. You should see two targets, one for each HBA.

[memgate-cluster: master] (config) # show targetsNode Hostname Proto Target State Address----------------------------------------------------------------------1 violin-mgb iscsi iscsi Active iqn.2004-02.com.vmem:violin-mgb2 violin-mga iscsi iscsi Active iqn.2004-02.com.vmem:violin-mga


iSCSI Target

4. Configure the interfaces. Remember to configure both MG’s interfaces.

[memgate-cluster: master] (config) # show int eth6

Interface eth6 status:

Comment:

Admin up: yes

Link up: yes

IP address:

Netmask:

Speed: 10000Mb/s (auto)

Duplex: full (auto)

Interface type: ethernet

Interface ifindex: 8

Interface source: physical

MTU: 1500

HW address: 90:E2:BA:26:80:78

RX bytes: 22350 TX bytes: 798

RX packets: 197 TX packets: 19

RX mcast packets: 0 TX discards: 0

RX discards: 0 TX errors: 0

RX errors: 0 TX overruns: 0

RX overruns: 0 TX carrier: 0

RX frame: 0 TX collisions: 0

TX queue len: 1000

Note: Ensure that the link is up, and the speed is 10000 Mb/s (auto).

5. Set the interface’s IP address and netmask. Do this for each Interface you would like to configure on this MG.

[memgate-cluster: master] (config) # interface eth6 ip address 10.100.1.12 255.255.255.0

[memgate-cluster: master] (config) # wr mem

Note: The Mapping of iSCSI Ports to Ethernet Interfaces on the Memory Gateway section in the Appendix contains an illustration of the port-to-interface mapping.

6. Set the MTU to Jumbo.

[memgate-cluster: master] (config) # no network global mtu enable

[memgate-cluster: master] (config) # interface eth4 mtu 9000

[memgate-cluster: master] (config) # write memory

7. Validate that the changes you made have taken effect.

[memgate-cluster: master] (config) # show int eth6

Interface eth6 status:

Comment:

Admin up: yes

Link up: yes



IP address: 10.100.2.12

Netmask: 255.255.255.0

Speed: 10000Mb/s (auto)

Duplex: full (auto)

Interface type: ethernet



MTU: 9000

HW address: 90:E2:BA:26:80:78








TX queue len: 1000

Note: Remember to do the above steps on each interface and on both MGs that you want to configure.

Creating an iSCSI Target on the Array

1. Create an iSCSI target on the master MG.

ssh admin@violin-mg

violin-mgb [memgate-cluster: master] > en

violin-mgb [memgate-cluster: master] # co t

violin-mgb [memgate-cluster: master] (config) # iscsi target create test-iscsi

violin-mgb [memgate-cluster: master] (config) # write memory

2. Bind the interface’s IP addresses to the iSCSI target.

violin-mgb [memgate-cluster: master] (config) # iscsi target bind test-iscsi to 10.100.2.12

violin-mgb [memgate-cluster: master] (config) # write memory

3. Log on to the standby MG and bind the interface’s IP address to the iSCSI target.

ssh admin@violin-mga

violin-mga [memgate-cluster: standby] > en

violin-mga [memgate-cluster: standby] # co t

violin-mga [memgate-cluster: standby] (config) # iscsi target bind test-iscsi to 10.100.2.11

violin-mga [memgate-cluster: master] (config) # write memory


iSCSI Target

4. Check that the interfaces are bound:

violin-mgb [memgate-cluster: master] (config) # show targetsNode Hostname Proto Target State Address------------------------------------------------------------------------1 violin-mgb iscsi iscsi Active iqn.2004-02.com.vmem:violin-mgb1 violin-mgb iscsi test-iscsi Active iqn.2004-02.com.vmem:violin-mgb:test-iscsi2 violin-mga iscsi test-iscsi Active iqn.2004-02.com.vmem:violin-mga:test-iscsi2 violin-mga iscsi iscsi Active iqn.2004-02.com.vmem:violin-mga





CHAPTER 2 Concerto 7700 Flash Storage Platform Best Practices

This chapter contains best practices that apply to the Concerto 7700 Flash Storage Platform (FSP).


Concerto 7700 FSP Zoning

The 7700 FSP requires 1:n zoning (single initiator, multiple targets) in order to minimize I/O interruptions by non-related FC activities, such as logging in and out, resets, and so on. For example, if there are two client initiator ports, and two 7000 FSP target ports on the same FC fabric, two zones should be created, one for each initiator port. Each of the zones should have both target ports added.

A single Fibre Channel port failure does not trigger a failover. If a host is zoned with only one target northbound port on the 7700 node and this port is lost, the host will lose access to its LUNs. For high availability (HA) purposes, it is recommended that initiators are zoned with all ports on the 7700 node from which it has LUNs presented. Similarly all southbound ports on 7700 node must be configured and zoned.

Concerto 7700 FSP Southbound Path Status

Note: Southbound refers to paths between the Controller for the Concerto 7700 FSP and the Violin array (storage shelf).

When all southbound paths are up, the paths to the Fibre Channel devices should always show as “Load Balancing”. When a southbound path is brought down, either by reloading a Violin gateway node or removing a path, upon restoring the path, the path status would go to “Standby-Active”, where an adapter rescan is required to change the path status back to “Load Balancing”.

Concerto 7700 FSP Fibre Channel Target Mode Considerations

All Fibre Channel target mode ports need to be connected (have a link up) when configuring failover.


Chapter 2 - Concerto 7700 Flash Storage Platform Best Practices



































--------------------------------------------------------------






















[root@vmem-10-1-15-103 ~]# multipath -ll

mpatha (36000d77300000d1a7fb0fb8300f903c0) dm-0 VIOLIN,CONCERTO ARRAY

size=49G features='1 queue_if_no_path' hwhandler='0' wp=rw


|- 3:0:2:0 sdy 65:128 active ready running

|- 4:0:2:0 sdz 65:144 active ready running

|- 3:0:3:0 sdak 66:64 active ready running

`- 4:0:3:0 sdal 66:80 active ready running

SLES11 BFS Notes

Complete the following steps to enable multipath I/O at install time on an active/active multipath storage LUN.

1. During the install on the YaST Installation Settings page, click Partitioning to open the YaST partitioner.

2. Select Custom Partitioning (for experts).

3. Select the Hard Disks main icon, click the Configure button, and then select Configure Multipath.

4. Start multipath.

YaST starts to rescan the disks and shows available multipath devices (such as /dev/disk/by-id/dm-uuid-mpath-3600a0b80000f4593000012ae4ab0ae65). This is the device that should be used for all further processing.



iSCSI Target

5. Click Next to continue with the installation.

Note: For more BFS info on SLES11, see the SUSE support page https://www.suse.com/documentation/sles11/stor_admin/data/mpioroot.html

iSCSI Target

iSCSI Target Considerations

For iSCSI, use IP addresses on different subnets when using more than one port on the Concerto 7100/7200 iSCSI target port. An example would be using IP addresses 1.0.2.x and 1.0.3.x for the target ports.





CHAPTER 3 7000 Series Flash Storage Platform Best Practices

This chapter contains best practices that apply to the 7000 Series Flash Storage Platform (FSP).


7000 FSP Zoning

The FSP requires isolated zoning, where one initiator is zoned to one target in order to minimize I/O interruptions by non-related FC activities, such as logging in and out, resets, and so on.

With isolated zoning, each zone can contain no more than two ports or two WWPNs. This applies to both initiator zones (storage) and target zones (hosts).

In the case of northbound (to client) zoning, for example, if there are two client initiators, and two FSP targets on the same FC fabric, four specific zones – one for each path – should be used. The same should be done for southbound (to storage) zoning. If there are two FSP initiators and two storage targets on the same fabric, there should be four zones.

Make sure that the storage devices are not zoned directly connected to the client.

7000 Series FSP Fibre Channel Target Mode Considerations

All Fibre Channel target mode ports need to be connected (have a link up) when configuring failover.






Chapter 3 - 7000 Series Flash Storage Platform Best Practices































--------------------------------------------------------------























[root@vmem-10-1-15-103 ~]# multipath -ll

mpatha (36000d77300000d1a7fb0fb8300f903c0) dm-0 VIOLIN,CONCERTO ARRAY

size=49G features='1 queue_if_no_path' hwhandler='0' wp=rw


|- 3:0:2:0 sdy 65:128 active ready running

|- 4:0:2:0 sdz 65:144 active ready running

|- 3:0:3:0 sdak 66:64 active ready running

`- 4:0:3:0 sdal 66:80 active ready running

SLES11 BFS Notes

Complete the following steps to enable multipath I/O at install time on an active/active multipath storage LUN.

1. During the install on the YaST Installation Settings page, click Partitioning to open the YaST partitioner.

2. Select Custom Partitioning (for experts).

3. Select the Hard Disks main icon, click the Configure button, and then select Configure Multipath.

4. Start multipath.

YaST starts to rescan the disks and shows available multipath devices (such as /dev/disk/by-id/dm-uuid-mpath-3600a0b80000f4593000012ae4ab0ae65). This is the device that should be used for all further processing.

5. Click Next to continue with the installation.

Note: For more BFS info on SLES11, see the SUSE support page https://www.suse.com/documentation/sles11/stor_admin/data/mpioroot.html

iSCSI Target

iSCSI Target Considerations

For iSCSI, use IP addresses on different subnets when using more than one port on the FSP iSCSI target port. An example would be using IP addresses 1.0.2.x and 1.0.3.x for the target ports.


Windows 2012 and Windows 2008 R2 (SP1)

CHAPTER 4 Operating System Considerations for Fibre Channel

Windows 2012 and Windows 2008 R2 (SP1)

Checking Paths

To check the paths of a particular LUN, use the mpclaim command:

• > mpclaim -s -d Lists all multipath disks

• > mpclaim -s -d [disknum] Lists paths for this particular disk

Note: Windows disk numbers do not necessarily represent the exported LUN IDs.

LUN Limitations

Windows disk range is 0 - 254.

Note: A rescan after unexporting LUNs from Windows is recommended; however, a reboot can clean up devices that are no longer there.

Clustering

Make sure that the LUNs are exported to both host cluster nodes. Both the Violin 6000 Series All Flash Array and Concerto 7x00 support SCSI-3 reservations.

Windows 2012

HBA Driver Parameter Settings

Use the default HBA driver parameters unless noted otherwise below. Contact Violin Support for supported HBA drivers.

Setting Up MPIO

Installing Hotfixes

For MPIO, install this hotfix from Microsoft: KB2784679


http://support.microsoft.com/kb/2784679/en-us

Chapter 4 - Operating System Considerations for Fibre Channel

To set up MPIO on Windows 2012:

1. Navigate to the Server Manager and select “Add Roles and Features” from the Manage pull down menu. When the Add Roles and Features Wizard window appears, click Next to page through the screens until you get to the “Select Features” screen.

Under Features, scroll down, and check the box next to Multipath I/O.

2. Click Next to complete the MPIO installation.


Windows 2012

3. After MPIO is installed, go to the Windows Control Panel, open Administrative Tools, and open the MPIO Properties window.

4. For the Violin 6000, do the following:

a. In the MPIO Properties window, click the Discover Multi-Paths tab. Under Others, select VIOLIN SAN ARRAY. (You must first export a Violin LUN to the Windows host.)



b. Select VIOLIN SAN ARRAY or VIOLIN SAN ARRAY ALUA. and click Add. Other storage devices might be present. Reboot the server for the changes to take effect.

5. For the Concerto 7x00, do the following:

a. In the MPIO Properties window, click the Discover Multi-Paths tab. Under SPC-3 compliant Device Hardware ID, select VIOLIN CONCERTO ARRAY. (You must first export a Concerto LUN to the Windows host.)

b. Select VIOLIN CONCERTO ARRAY and click Add. Other storage devices might be present. Reboot the server for the changes to take effect.

c. The DSM PDO remove period needs to be changed from the default value 20 to 300. When the server is back up, go to Computer Management. Under Device Manager, expand Disk Drives and right-click the VIOLIN CONCERTO ARRAY Multi-Path Disk Device, then select Properties.

d. Click the MPIO tab, and then click Details. Change the PDO Remove Period parameter to 300, then click OK and then OK again. You only need to change this parameter for one Concerto disk, and it will be changed for all the Concerto disks.


Windows 2008 R2 (SP1)


HBA Driver Parameter Settings

Use the default HBA driver parameters unless noted otherwise below. Contact Violin Support for supported HBA drivers.

Note: If using the Qlogic QLE256x HBA driver 9.1.11.20 with Windows 2008 R2 Server, change the HBA driver parameter ‘Link Down Timeout’ from its default of 45 seconds to 30 seconds, and then reboot the server. You can do this either using the QLogic BIOS utility during the system boot, or with the QLogic QConverge utility.



Installing Hotfixes

Install these hotfixes from Microsoft:

• KB2591462

• KB2468345

• KB2522766

• KB2460971

For Windows clustering, install this hotfix from Microsoft:

• KB2531907

Note: This is only needed for clustering since it fixes an issue when running the MS cluster validation test.

After installing the hotfixes and a reboot of the host machine, ensure that the MPIO version is the same as in the example below. If it is an older version, then make sure the hotfixes mentioned above installed correctly.








Setting Up MPIO

To set up MPIO on Windows 2008 R2 (SP1):

1. Navigate to the Server Manager and click Features in the left pane. Then click Add Features.

2. Select Multipath I/O in the list of features, and click Next.




a. Once the feature is installed, find MPIO in the Control Panel. (Ensure at least one LUN is exported to the host before proceeding.)

b. Click the ‘Discover Multi-Paths’ tab. Click ‘VIOLIN SAN ARRAY ALUA’ and/or ‘VIOLIN SAN ARRAY’, then click Add. This will add support for both Active/Active and/or ALUA disks. A reboot of the host is required.




a. Once the feature is installed, find MPIO in the Control Panel. (Ensure at least one LUN is exported to the host before proceeding.)

b. Click the ‘Discover Multi-Paths’ tab, click ‘VIOLIN CONCERTO ARRAY', then click Add. Other storage devices might be present. A reboot of the host is required.



c. The DSM PDO remove period needs to be changed from the default value 20 to 300. When the server is back up, go to Computer Management. Under Device Manager, expand Disk Drives and right-click the VIOLIN CONCERTO ARRAY Multi-Path Disk Device, then select Properties.


Red Hat Enterprise Linux 5.x/6.x/7.x

d. Click the MPIO tab, and then click Details. Change the PDO Remove Period parameter to 300, then click OK and then OK again. You only need to change this parameter for one Concerto disk, and it will be changed for all the Concerto disks.


Device Mapper Setup

To check if device mapper is installed:

# chkconfig --list | grep multipathdmultipathd 0:off 1:off 2:on 3:on 4:on 5:on 6:off

If you do not see any output, find the appropriate device mapper package and install it before proceeding. If you see that run levels 3, 4, 5 are off but the daemon is present; use the following command to turn on the daemon for run levels 3, 4, 5.

# chkconfig --level 345 multipathd on

If you enter ‘multipath’ with no arguments and get the error that the multipath driver is not loaded, enter the following commands:

# modprobe dm_multipath# multipath -v 2



Checking for Paths

For Violin 6000

When using device mapper multipathing, use the ‘multipath’ command to check paths.

Active/Active example:

[root@vmem-10-1-15-103 ~]# multipath -llmpatha (36001b970f29d0ac9f29d0ac9913baff0) dm-0 VIOLIN,SAN ARRAYsize=100G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 1:0:0:0 sda 8:0 active ready running |- 1:0:2:0 sdc 8:32 active ready running |- 1:0:3:0 sdd 8:48 active ready running |- 1:0:1:0 sdb 8:16 active ready running |- 2:0:0:0 sde 8:64 active ready running |- 2:0:1:0 sdf 8:80 active ready running |- 2:0:2:0 sdg 8:96 active ready running `- 2:0:3:0 sdh 8:112 active ready running

ALUA example:

[root@vmem-srv13 ~]# multipath -llmpathacp (SVIOLIN_SAN_ARRAY_ALUA_F29D0AC9EC89F492) dm-5 VIOLIN,SAN ARRAY ALUAsize=10G features='0' hwhandler='1 alua' wp=rw|-+- policy='round-robin 0' prio=130 status=active| |- 2:0:2:3 sdl 8:176 active ready running| |- 2:0:3:3 sdp 8:240 active ready running| |- 1:0:2:3 sdab 65:176 active ready running| `- 1:0:3:3 sdaf 65:240 active ready running`-+- policy='round-robin 0' prio=10 status=enabled |- 2:0:0:3 sdd 8:48 active ready running |- 2:0:1:3 sdh 8:112 active ready running |- 1:0:0:3 sdt 65:48 active ready running `- 1:0:1:3 sdx 65:112 active ready running

For Concerto 7x00

[root@vmem-srv16 ~]# multipath -llmpatha (36000d77300000d1a7fb0fb8300f903c0) dm-0 VIOLIN,CONCERTO ARRAYsize=49G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=130 status=active |- 3:0:2:0 sdy 65:128 active ready running |- 4:0:2:0 sdz 65:144 active ready running |- 3:0:3:0 sdak 66:64 active ready running `- 4:0:3:0 sdal 66:80 active ready running



Symantec Veritas SFHA Solution

For Violin 6000

Violin recommends the defaults for Symantec Veritas SFHA multipathing.

Examples:

To show all of the default parameters:

[root@vmem-cn10 ~]# vxdmpadm gettune all Tunable Current Value Default Value------------------------------ ------------- -------------dmp_cache_open on ondmp_daemon_count 10 10dmp_delayq_interval 15 15dmp_fast_recovery on ondmp_health_time 60 60dmp_log_level 1 1dmp_low_impact_probe on ondmp_lun_retry_timeout 0 0dmp_path_age 300 300dmp_pathswitch_blks_shift 9 9dmp_probe_idle_lun on ondmp_probe_threshold 5 5dmp_restore_cycles 10 10dmp_restore_interval 300 300dmp_restore_policy check_disabled check_disableddmp_restore_state enabled enableddmp_retry_count 5 5dmp_scsi_timeout 20 20dmp_sfg_threshold 1 1dmp_stat_interval 1 1dmp_monitor_ownership on ondmp_monitor_fabric on ondmp_monitor_osevent on ondmp_native_support off off

To show the default multipathing I/O policy:

[root@vmem-cn10 ~]# vxdmpadm getattr enclosure aluadisk0 iopolicyENCLR_NAME DEFAULT CURRENT============================================aluadisk0 MinimumQ MinimumQ

To list the paths of a given disk:

[root@vmem-g7-srv1 ~]# vxdmpadm getsubpaths dmpnodename=violin0_8NAME STATE[A] PATH-TYPE[M] CTLR-NAME ENCLR-TYPE ENCLR-NAME ATTRS=================================================================sdal ENABLED(A) - c3 VIOLIN violin0 -sdaz ENABLED(A) - c4 VIOLIN violin0 -sdj ENABLED(A) - c3 VIOLIN violin0 -sdx ENABLED(A) - c4 VIOLIN violin0 -



Red Hat Enterprise Linux 5.x

Device Mapper Configuration

For Violin 6000

Edit the multipath.conf file, located in /etc/multipath.conf, to contain the following device tree to support ALUA and Active/Active Violin disks.

Note: If running a BFS configuration, change the ‘getuid_callout’ line to this: getuid_callout "/sbin/scsi_id -p 0x83 -g -u -s /block/%n"

defaults{user_friendly_names yes

}

devices{device{

vendor "VIOLIN"product "SAN ARRAY ALUA"path_grouping_policy group_by_priogetuid_callout "/sbin/scsi_id -p 0x80 -g -u -s /block/%n"prio_callout "/sbin/mpath_prio_alua /dev/%n"path_checker turpath_selector "round-robin 0"hardware_handler "1 alua"failback immediaterr_weight uniformno_path_retry failrr_min_io 4

}

device{vendor "VIOLIN"product "SAN ARRAY"path_grouping_policy group_by_serialgetuid_callout "/sbin/scsi_id -p 0x80 -g -u -s /block/%n"path_checker turpath_selector "round-robin 0"hardware_handler "0"failback immediaterr_weight uniformno_path_retry failrr_min_io 4

}}



For Concerto 7x00


}devices{

device{vendor "VIOLIN"product "CONCERTO ARRAY"path_grouping_policy group_by_priogetuid_callout "/sbin/scsi_id -p 0x80 -g -u -s /block/%n"prio_callout "/sbin/mpath_prio_alua /dev/%n"path_checker turpath_selector "round-robin 0"hardware_handler "1 alua"failback immediaterr_weight uniformno_path_retry 300rr_min_io 4features "1 queue_if_no_path"

}}

Tuning Recommendations for Red Hat Enterprise Linux 5.x

According to the RHEL documentation (Filesystem on VMware Red Hat Enterprise Linux 4, 5, & 6 guests went read-only - - https://access.redhat.com/solutions/35329), /proc/sys/kernel/hung_task_timeout_secs needs to be increased accordingly.

Typically with an io default I/O timeout value of 60 seconds. We recommend the hung_task_timeout_secs is set to 120 seconds.

Note: Setting hung_task_timeout_secs to at least I/O timeout+60 seconds up to twice the timeout value is suggested.

For example: echo <value in seconds> > /proc/sys/kernel/hung_task_timeout_secs

Linux Configuring For High Performance

It is reported that there is significant improvement of Linux IO scheduler on kernel v2.6 and later, so Violin strongly recommends that you upgrade your kernel to at least v2.6.

It is important to modify the Linux IO scheduler and queue depth as they were designed for spinning media. This can be done both by a udev rule which will be persistent across reboots, and interactively for experimentation to discover the optimal values for your specific environment.

udev rules are run in order, by file name. A lower numbered file (such as 12-violin.rules) will run before a higher numbered file (such as 60-violin.rules). Other software, such as Oracle ASM, may depend on the Violin rules running before their rules, such that the Violin storage is mounted prior to their rules being run. Validate the other software, verify the order in which the rules should run, and modify the Violin rules file names as appropriate.


https://access.redhat.com/solutions/3532



Setting UDEV Rules For Persistent Values

1. Create a file in /etc/udev/rules.d/60-violin.rules with just this line (all on one line):

KERNEL=="sd*[!0-9]|sg*", BUS=="scsi", SYSFS{vendor}=="VIOLIN", SYSFS{model}=="SANARRAY*", RUN+="/bin/sh -c 'echo noop> /sys/$devpath/queue/scheduler&& echo 512>/sys/$devpath/queue/nr_requests'"

2. Then run the following commands to get the rule to take effect:

# chmod 0644 /etc/udev/rules.d/60-violin.rules# udevcontrol --reload_rules# udevtrigger



For Violin 6000


Note: If running a BFS configuration, change the ‘getuid_callout’ line to this: getuid_callout "/lib/udev/scsi_id --whitelisted --replace-whitespace --page=0x83 --device=/dev/%n"


}

devices{device{

vendor "VIOLIN"product "SAN ARRAY ALUA"path_grouping_policy group_by_priogetuid_callout "/lib/udev/scsi_id --whitelisted --replace-

whitespace --page=0x80 --device=/dev/%n"prio "alua"path_checker turpath_selector "round-robin 0"hardware_handler "1 alua"failback immediaterr_weight uniformno_path_retry failrr_min_io_rq 4

}



device{vendor "VIOLIN"product "SAN ARRAY"path_grouping_policy group_by_serialgetuid_callout "/lib/udev/scsi_id --whitelisted --replace-whitespace --page=0x80 --device=/dev/%n"path_checker turpath_selector "round-robin 0"hardware_handler "0"failback immediaterr_weight uniformno_path_retry failrr_min_io_rq 4

}}

For Violin 7x00

Note: The following /etc/multipath.conf file is same for when booting from the Violin LUN.

defaults { user_friendly_names yes

}

devices { device {

vendor "VIOLIN" product "CONCERTO ARRAY" path_selector "round-robin 0" path_grouping_policy multibus getuid_callout "/lib/udev/scsi_id -p 0x80 -g -u /dev/%n" prio alua path_checker tur

rr_min_io 100

rr_weight priorities failback immediate features "1 queue_if_no_path" no_path_retry 300 }

}

Tuning Recommendations for Red Hat Enterprise Linux 6.x

According to the RHEL documentation (Filesystem on VMware Red Hat Enterprise Linux 4, 5, & 6 guests went read-only - https://access.redhat.com/solutions/35329), /proc/sys/kernel/hung_task_timeout_secs needs to be increased accordingly.

Typically with an io default I/O timeout value of 60 seconds. We recommend the hung_task_timeout_secs is set to 120 seconds. Note: Setting hung_task_timeout_secs to at least I/O timeout+60 seconds up to twice the timeout value is suggested.

For example:





echo <value in seconds> > /proc/sys/kernel/hung_task_timeout_secs

Linux Configuring For High Performance

It is reported that there is significant improvement of Linux IO scheduler on kernel v2.6 and later, so Violin strongly recommends that you upgrade your kernel to at least v2.6.

It is important to modify the Linux IO scheduler and queue depth as they were designed for spinning media. This can be done both by a udev rule which will be persistent across reboots, and interactively for experimentation to discover the optimal values for your specific environment.

udev rules are run in order, by file name. A lower numbered file (such as 12-violin.rules) will run before a higher numbered file (such as 60-violin.rules). Other software, such as Oracle ASM, may depend on the Violin rules running before their rules, such that the Violin storage is mounted prior to their rules being run. Validate the other software, verify the order in which the rules should run, and modify the Violin rules file names as appropriate.

Setting UDEV Rules For Persistent Values

1. Create a file in /etc/udev/rules.d/12-violin.rules. It should contain the following:

KERNEL=="sd*[!0-9]|sg*", BUS=="scsi", SYSFS{vendor}=="VIOLIN", RUN+="/bin/sh -c 'echo noop > /sys/$devpath/queue/scheduler'"KERNEL=="sd*[!0-9]|sg*", BUS=="scsi", SYSFS{vendor}=="VIOLIN", RUN+="/bin/sh -c 'echo1024 > /sys/$devpath/queue/nr_requests'"KERNEL=="sd*[!0-9]|sg*", BUS=="scsi", SYSFS{vendor}=="VIOLIN", RUN+="/bin/sh -c 'echo 1 > /sys/$devpath/queue/rq_affinity'"KERNEL=="sd*[!0-9]|sg*", BUS=="scsi", SYSFS{vendor}=="VIOLIN", RUN+="/bin/sh -c 'echo 0 > /sys/$devpath/queue/rotational'"KERNEL=="sd*[!0-9]|sg*", BUS=="scsi", SYSFS{vendor}=="VIOLIN", RUN+="/bin/sh -c 'echo 1 > /sys/$devpath/queue/nomerges'"KERNEL=="sd*[!0-9]|sg*", BUS=="scsi", SYSFS{vendor}=="VIOLIN", RUN+="/bin/sh -c 'echo4096 > /sys/$devpath/queue/max_sectors_kb'"ENV{DM_UUID}=="mpath-SVIOLIN_SAN_ARRAY_?*", RUN+="/bin/sh -c 'echo noop >/sys/$devpath/queue/scheduler'"ENV{DM_UUID}=="mpath-SVIOLIN_SAN_ARRAY_?*", RUN+="/bin/sh -c 'echo 1024 >/sys/$devpath/queue/nr_requests'"ENV{DM_UUID}=="mpath-SVIOLIN_SAN_ARRAY_?*", RUN+="/bin/sh -c 'echo 1 > /sys/$devpath/queue/rq_affinity'"ENV{DM_UUID}=="mpath-SVIOLIN_SAN_ARRAY_?*", RUN+="/bin/sh -c 'echo 0 > /sys/$devpath/queue/rotational'"ENV{DM_UUID}=="mpath-SVIOLIN_SAN_ARRAY_?*", RUN+="/bin/sh -c 'echo 1 >/sys/$devpath/queue/nomerges'"ENV{DM_UUID}=="mpath-SVIOLIN_SAN_ARRAY_?*", RUN+="/bin/sh -c 'echo 4096 >/sys/$devpath/queue/max_sectors_kb'"

2. Run the following commands to get the rule to take effect:

# chmod 0644 /etc/udev/rules.d/12-violin.rules# udevadm control --reload-rules# udevadm trigger




For Violin 7x00

defaults { user_friendly_names yes }

devices { device { vendor "VIOLIN" product "CONCERTO ARRAY" path_selector "round-robin 0" path_grouping_policy multibus prio alua path_checker tur rr_min_io 100 rr_weight priorities failback immediate features "1 queue_if_no_path" no_path_retry 300 } }

SUSE Enterprise Linux 11 SP3


For Violin 6000



}

devices{

device{vendor "VIOLIN"product "SAN ARRAY"path_grouping_policy group_by_serialpath_checker turpath_selector "round-robin 0"hardware_handler "0"failback immediaterr_weight uniformno_path_retry 100}

device{



vendor "VIOLIN"product "SAN ARRAY ALUA"path_grouping_policy group_by_prioprio "alua"path_checker turpath_selector "round-robin 0"hardware_handler "0"failback immediateno_path_retry 100detect_prio yes

}}

For Violin 7x00

defaults {user_friendly_names yes}

devices {

device {vendor "VIOLIN"product "CONCERTO ARRAY"path_selector "round-robin 0"path_grouping_policy multibusgetuid_callout "/lib/udev/scsi_id -p 0x80 -g -u /dev/%n"prio "alua"path_checker "tur"rr_min_io 100rr_weight prioritiesfailback immediatefeatures "1 queue_if_no_path"no_path_retry 300 }

}

# multipath -llmpathe (SVIOLIN_CONCERTO_ARRAY_HVZBVKP6KJ6P) dm-4 VIOLIN ,CONCERTO ARRAYsize=15G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=50 status=active |- 5:0:2:4 sdl 8:176 active ready running |- 5:0:3:4 sdp 8:240 active ready running |- 6:0:1:4 sdw 65:96 active ready running `- 6:0:3:4 sdad 65:208 active ready runningmpathd (SVIOLIN_CONCERTO_ARRAY_03Z7XWPJSD1P) dm-3 VIOLIN ,CONCERTO ARRAYsize=15G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=50 status=active |- 5:0:2:3 sdk 8:160 active ready running |- 5:0:3:3 sdo 8:224 active ready running |- 6:0:1:3 sdv 65:80 active ready running `- 6:0:3:3 sdac 65:192 active ready running


VMware ESXi 5.x

mpathn (SVIOLIN_CONCERTO_ARRAY_YK57AP584QYN) dm-2 VIOLIN ,CONCERTO ARRAYsize=25G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=50 status=active |- 5:0:0:2 sdg 8:96 active ready running |- 5:0:1:2 sdj 8:144 active ready running |- 6:0:0:2 sdu 65:64 active ready running `- 6:0:2:2 sdab 65:176 active ready runningmpathm (SVIOLIN_CONCERTO_ARRAY_4HQDIKLVFF5E) dm-1 VIOLIN ,CONCERTO ARRAYsize=25G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=50 status=active |- 5:0:0:1 sdf 8:80 active ready running |- 5:0:1:1 sdi 8:128 active ready running |- 6:0:0:1 sdt 65:48 active ready running `- 6:0:2:1 sdaa 65:160 active ready runningmpathl (SVIOLIN_CONCERTO_ARRAY_A98G910PX7J7) dm-0 VIOLIN ,CONCERTO ARRAYsize=25G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=50 status=active |- 5:0:0:0 sde 8:64 active ready running |- 5:0:1:0 sdh 8:112 active ready running |- 6:0:0:0 sds 65:32 active ready running `- 6:0:2:0 sdz 65:144 active ready runningmpathk (3600605b00454c540020734af50610cbc) dm-9 IBM ,ServeRAID M1015size=80G features='0' hwhandler='0' wp=rw`-+- policy='service-time 0' prio=1 status=active `- 0:2:3:0 sdd 8:48 active ready runningmpathj (3600605b00454c540020734af5060a062) dm-8 IBM ,ServeRAID M1015size=40G features='0' hwhandler='0' wp=rw`-+- policy='service-time 0' prio=1 status=active `- 0:2:2:0 sdc 8:32 active ready runningmpathi (3600605b00454c540020734af50602e62) dm-7 IBM ,ServeRAID M1015size=80G features='0' hwhandler='0' wp=rw`-+- policy='service-time 0' prio=1 status=active `- 0:2:1:0 sdb 8:16 active ready runningmpathg (SVIOLIN_CONCERTO_ARRAY_4MIKA9DCN4UI) dm-6 VIOLIN ,CONCERTO ARRAYsize=20G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=50 status=active |- 5:0:2:6 sdn 8:208 active ready running |- 5:0:3:6 sdr 65:16 active ready running |- 6:0:1:6 sdy 65:128 active ready running `- 6:0:3:6 sdaf 65:240 active ready runningmpathf (SVIOLIN_CONCERTO_ARRAY_TBJ27OY8K5DN) dm-5 VIOLIN ,CONCERTO ARRAYsize=20G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=50 status=active |- 5:0:2:5 sdm 8:192 active ready running |- 5:0:3:5 sdq 65:0 active ready running |- 6:0:1:5 sdx 65:112 active ready running `- 6:0:3:5 sdae 65:224 active ready running

VMware ESXi 5.x

Check the VMware site for supported guest operating systems.

Note: The following applies to both the Violin 6000 and Concerto 7x00.


http://www.vmware.com/resources/compatibility/search.php?deviceCategory=software&rct=j&q=&esrc=s&frm=1&source=web&cd=2&ved=0CDQQFjAB&url=http://www.vmware.com/go/ws9ossupport&ei=nrZpUqr8KoPniwL0hYHoDQ&usg=AFQjCNG3D6Y4EcNl_JrFMgF7VlSVCoac0Q


Multipathing Policy

Round robin is used for the VMware multipathing policy. The illustration below is for an ALUA LUN.

Note: Set the multipathing policy for each LUN.

Setting MPIO Path Switching to IOPS

By default, the Round Robin path selection plugin chooses a new path to send I/Os by number of IOPS. The setting defaults to changing paths every 1000 IOPS. Reducing this value can have positive performance impact.

Use the following ESXi CLI command to reduce the number of IOPS that trigger path switching for round robin on all Violin devices:

# for i in èsxcli storage nmp device list | grep -B5 VMW_PSP_RR | grep -B1 VIOLIN / | grep ̂ naa`; do echo esxcli storage nmp psp roundrobin deviceconfig set / --iops=4 --type=iops --device=$i; done

For more information on MPIO on ESXi, refer to the VMware document vSphere Storage at www.vmware.com/support/pubs/


http://www.vmware.com/support/pubs

VMware ESXi 5.x

Fibre Channel Zoning

On VMware ESX, Violin recommends modified zoning if you are planning on using raw device mappings (RDM). The zoning example below is for raw device mappings for the virtual machine with Fibre Channel NPIV enabled for a system that contains two HBA ports.

• Zone 1:

pwwn of ESX server HBA port 1pwwn of virtual machine (NPIV) port 1pwwn of target port 1pwwn of target port 2… … …pwwn of target port n

• Zone 2:

pwwn of ESX server HBA port 2pwwn of virtual machine (NPIV) port 2pwwn of target port 1pwwn of target port 2… … …pwwn of target port n

Note: If you are not using raw device mappings, follow the common Fibre Channel best practice of one initiator port to many target ports.

Initiator Groups

Note: The following applies only to the Violin 6000.

When creating igroups for VMware hosts and Virtual Machines (VM), as a best practice first create an igroup with the port worldwide name of the ESX server HBAs. If you are using raw device mappings, devices for the VMs and have NPIV enabled, it is recommended to create a separate igroup for each of the virtual machines, which can be nested with the ESX server HBA igroup under a different igroup.

Example:

# show igroupInitiator group: esx_server_hbas #Initiators: 2 wwn.21:00:00:1b:32:91:c6:b5 wwn.21:01:00:1b:32:b1:c6:b5Initiator group: esx51_vm1 #Initiators: 2 wwn.28:1c:00:0c:29:00:00:02 wwn.28:1c:00:0c:29:00:00:03Initiator group: esx_server_wrapper #Initiators: 2 esx51_vm1 esx_server_hbas

In this example, there is igroup esx_server_wrapper, which has the ESX HBA pwwn igroup and the virtual machine pwwn igroup nested within.



RHEL 6.x Virtual Machines

Violin recommends using the VMware Paravirtual option for the SCSI Controller when creating the Virtual Machines.

Tuning Recommendations

According to the RHEL documentation (Filesystem on VMware Red Hat Enterprise Linux 4, 5, & 6 guests went read-only - https://access.redhat.com/solutions/35329), please install latest available version of VMware Tools, which installs their udev rule to increase the timeout to 180 sec in /etc/udev/rules.d/99-vmware-scsi-udev.rules.

Setting MPIO Path Switching to IOPS

By default, the Round Robin path selection plugin chooses a new path to send I/Os by number of IOPS. The setting defaults to changing paths every 1000 IOPS. Reducing this value can have positive performance impact.

Set RR for all Violin Disks

https://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vcli.examples.doc_50%2Fcli_advanced_storage.8.2.html

1. From ESXi CLI (Note: remove # from lines for cut-n-paste):

#for i in $(esxcli storage core device list | grep -i 'Violin Fibre ChannelDisk' | awk '{print $7}' | sed 's/[â-zA-Z0-9.]/ /g'); do#echo setting $i to Round Robin#esxcli storage nmp device set --device=$i --psp=VMW_PSP_RR#done

Set SATP configuration for Multipathing

1. From ESXi CLI (Note: remove # from lines for cut-n-paste):

#esxcli storage nmp satp set --default-psp=VMW_PSP_RR --satp=VMW_SATP_ALUA

Note: This requires a reboot, and any LUNs added after this will already be RR and no need to set the individual device.

To display Paths set (including IO Operation Limits):

1. From ESXi CLI (Note: Remove # from lines for cut-n-paste):

# for i in $(esxcli storage core device list | grep -i 'Violin Fibre ChannelDisk' | awk '{print $7}' | sed 's/[â-zA-Z0-9.]/ /g'); do# esxcli storage nmp psp roundrobin deviceconfig get --device=$i# done


https://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vcli.examples.doc_50%2Fcli_advanced_storage.8.2.html

Citrix 6.x

To display all devices multipath policy:

1. From ESXi CLI:

esxcli storage nmp device list

2. Set the IOPS per RR path to lower than 1000 (default)

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2069356Set IOPS = 4 for RR NMP

3. From ESXi CLI (Note: Remove # from lines for cut-n-paste):

#for i in $(esxcli storage core device list | grep -i 'Violin Fibre ChannelDisk' | awk '{print $7}' | sed 's/[â-zA-Z0-9.]/ /g'); do#echo doing $i#esxcli storage nmp psp roundrobin deviceconfig set --iops=4 --type=iops--device=$i#done

For more information on MPIO on ESXi, refer to the VMware document vSphere Storage Guide at the VMware Support Web site: https://www.vmware.com/support/pubs/

Citrix 6.x

Device Mapper Setup

To check if Device Mapper is installed:

# chkconfig --list | grep multipathdmultipathd 0:off 1:off 2:on 3:on 4:on 5:on 6:off

If you do not see any output, find the appropriate device mapper package and install it before proceeding. If you see that run levels 3, 4, 5 are off but the daemon is present; use the following command to turn on the daemon for run levels 3, 4, 5.

# chkconfig --level 345 multipathd on

If you enter ‘multipath’ with no arguments and get the error that the multipath driver is not loaded, enter the following commands:

# modprobe dm_multipath# multipath -v 2


https://www.vmware.com/support/pubs/


Checking for Paths

For Concerto 7x00

[root@qa-dl360g8-srv1 ~]# multipath -ll36001b97b02d693bdc219000055dba68b dm-4 VIOLIN,CONCERTO ARRAYsize=100G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw`-+- policy='round-robin 0' prio=50 status=enabled |- 1:0:0:0 sdb 8:16 active ready running |- 1:0:2:0 sdc 8:32 active ready running |- 1:0:4:0 sdd 8:48 active ready running |- 1:0:5:0 sde 8:64 active ready running |- 4:0:0:0 sdf 8:80 active ready running |- 4:0:2:0 sdg 8:96 active ready running |- 4:0:4:0 sdh 8:112 active ready running `- 4:0:7:0 sdi 8:128 active ready running

For Violin 7x00


defaults { user_friendly_names no replace_wwid_whitespace yes dev_loss_tmo 30}

devices { device { vendor "VIOLIN" product "CONCERTO ARRAY" features "1 queue_if_no_path" path_selector "round-robin 0" path_grouping_policy "multibus" hardware_handler "1 alua" prio "alua" failback immediate path_checker "tur" rr_min_io 100 rr_weight "priorities" no_path_retry 300 }}


AIX 6.1 & 7.1

AIX 6.1 & 7.1

For Violin 6000

Installing Violin’s PCM

To enable support for Violin LUNs, download the latest vmem PCM package from Violin Memory Customer Support.

Example package to install (AIX 6.1):

• 6.1.1.4.devices.fcp.disk.vmem.rte

To check if you have the vmem PCM package properly installed:

# lslpp -l | grep vmemdevices.fcp.disk.vmem.rte 6.1.1.4 COMMITTED Violin memory array disk

After installing the vmem PCM package, verify that the correct values to these parameters have been set:

• algorithm - round robin

• rw_timeout - 30

• fast_fail -yes

• dynamic tracking - yes

# lsattr -El hdisk10PCM PCM/friend/vmem_pcm Path Control Module TruePR_key_value none Persistante reservation value Truealgorithm round_robin Algorithm Trueclr_q no Device CLEARS its Queue on error Truedist_err_pcnt 0 Distributed Error Sample Time Truedist_tw_width 50 Distributed Error Sample Time Truehcheck_cmd test_unit_rdy Health Check Command Truehcheck_interval 30 Health Check Interval Truehcheck_mode nonactive Health Check Mode Truelocation Location Label Truelun_id 0x9000000000000 Logical Unit Number ID Truemax_transfer 0x40000 N/A Truenode_name 0x20000024ff3a3088 FC Node Name Falsepvid none Physical volume identifier Falseq_err yes Use QERR bit Trueq_type simple Queuing TYPE Falsequeue_depth 255 Queue DEPTH Truereassign_to 120 REASSIGN unit time out value Truereserve_policy no_reserve Reserve Policy Truerw_timeout 30 READ/WRITE time out value Truescsi_id 0x8d5400 SCSI ID Truestart_timeout 180 START unit time out value Truetimeout_policy retry_path Timeout Policy Trueunique_id 2F106E2B9224E06940070ESAN ARRAY ALUA06VIOLINfcp Unique device identifier Falseww_name 0x21000024ff3a3088 FC World Wide Name False

# lsattr -El fscsi0attach switch How this adapter is CONNECTED Falsedyntrk yes Dynamic Tracking of FC Devices Truefc_err_recov fast_fail FC Fabric Event Error RECOVERY Policy Truescsi_id 0xa4c00 Adapter SCSI ID Falsesw_fc_class 3 FC Class for Fabric True



NACA LUN Creation

Ensure when creating LUNs for AIX that the ‘naca’ option is selected. This improves performance and reliability.

Here is an example of creating a LUN for AIX from the command line:

[violin2503: master] (config) # lun create container 41243F00695 name aix_test_01 size 100 thin alua preferredport 1 naca

To ensure the LUNs are created with the naca bit on use:

violin2503-mg-a [violin2503: master] (config) # show luns naca

Container: 41243F00695LUN NACA------------------------------------aix_aluaA_01 1aix_aluaA_02 1..aix_aluaB_09 1aix_aluaB_10 1svc_aluaA_01 0svc_aluaB_02 0ucs1-n6_aluaA_01 0ucs1-n6_aluaA_02 0

For Violin 7x00

Install Violin ODM fileset

Object Data Manager (ODM) attributes 'reserve_lock' and 'reserve_policy' define access to Violin disk devices and control the SCSI reservation behavior. The violindisk_ha.rte ODM file contains the following default values for SCSI reservation attributes to open a Violin disk in exclusive mode in order to avoid any accidental data corruption on disk devices that are shared across nodes/systems:

• 'reserve_lock' = 'yes

• 'reserve_policy' = 'single_path'

To install:

1. Navigate to the temp directory (cd /tmp) and copy the ODM fileset to /tmp.

2. Install the unzipped ODM fileset:

# installp -ad violindisk_ha.rte all

3. Rescan the configuration:

# cfgmgr -v


AIX 6.1 & 7.1

4. Verify the configuration:

# lsdev -Cc disk

hdisk2 Available 03-00-01 Violin Concerto Disk Drive




Install PathManager

Installation of PathManager on AIX is managed by the SMIT software management commands.

This tool (the installp command) can be run either from the command line or from a Graphical User Interface (GUI).

The following instructions illustrate using the command line. However, the same tasks can be accomplished using the GUI. For more information on the GUI alternative, consult your IBM-AIX documentation.

Follow these steps to install the PathManager driver and its supporting files.

1. Set up your system hardware.

2. Log in as “root”.

Object Data Manager (ODM) fileset must be installed to recognize devices provided by the storage server if it is not already installed. If you have existing storage server devices, they must all be removed before the ODM fileset is updated. See Install Violin ODM fileset on page 62 for install information.

3. If a previous version of PathManager is installed on your system, remove it and reboot before continuing with this installation.

To determine if an older version of PathManager is present on your system, examine the output of the following command:

# lslpp –L|grep PathManager

4. Assign devices from the storage server to the PathManager client.

5. Verify that the client is able to see and access all storage devices through all available paths.

For example, If you have two paths to your storage system, each logical drive should be displayed twice in the output of the following command:

# lsdev –Cc disk

If you cannot see all available storage devices through every path, verify that you have the latest Fibre Channel HBA drivers loaded on your system.

6. Navigate to the temp directory (cd /tmp) and copy the PathManager package to /tmp.

7. Install PathManager:

# installp -a -d `pwd`/PathManager-<version number>.rte all

For example:

# installp -a -d `pwd`/PathManager-1.0-8017-AIX.rte all



8. Start PathManager:

# dpcli start

9. Run initial setup to create a new configuration file and detect existing devices:

# dpcli setup

Note: Any time you re-assign SAN Resources to the same LUN, you must reboot the AIX client after unassigning a SAN Resource.

Manage PathManager

Enable Dynamic Tracking and Delayed FC Error Recovery

Dynamic Tracking allows PathManager to detect certain device changes and then re-route traffic destined for that device to the new address while the device is still online.

1. Before enabling FC error recovery and Dynamic Tracking, remove all of the sub devices beneath the FC adapter.

For example, if fscsi0 contains the devices hdisk3 and hdisk4, you must remove the devices before enabling FC error recovery and Dynamic Tracking.

2. Turn on Dynamic Tracking and Delayed FC error recovery for each initiator:

# chdev -l fscsix -a 'dyntrk=yes fc_err_recov=delayed_fail' -P

Where x represents the adapter instance number.

3. Reboot:

# shutdown -Fr

4. Verify that the appropriate attributes have been changed on each initiator:

# lsattr -E -l fscsix

Note: The chdev -P flag, changes the device characteristics permanently in the Customized Devices object class. If the -P parameter is not used, dynamic tracking will be reset back to the default value (which is disabled) after an AIX TL/SP update or after the rmdev -dl fcs# -R; cfgmgr command is run.

PathManager Devices

Disk device filenames in AIX have names in the following format:

/dev/hdiskX

/dev/rhdiskX

Where X = the controller instance number. A sample listing follows:

hdisk1 Available 30-60-00-0,0 Fibre Channel SCSI FCP Disk Drive





Solaris

In a typical, single-path disk storage system, each physical disk is represented in the client’s /dev directory by two special device files representing the character and block mode device instances. For simplicity, we reference the set of device files as a single device filename and disregard the first part of the path names that distinguish between raw and block modes.

As a layered driver, PathManager creates its own device files for devices; it is through these device filenames that applications access the underlying devices.

When multiple paths to a single device are present, multiple device files are present, one for each path via the native disk driver. In these cases, PathManager creates a single additional device filename for applications to access the device and manages the original device files transparently to those applications.

PathManager is created as a new layered device driver and it is presented as a unique device instead of a native AIX device.

# lsdev -Cc disk





hdiskdpd0 Available PathManager Multipath Device

hdiskdpd1 Available PathManager Multipath Device

hdiskdpdctrl Available PathManager Multipath Device

Note: It is the virtual device file that will be used by applications to access that device. The original, redundant data paths should never be used to access the device, or the data it contains could be corrupted.

Solaris

For Violin 6000

ALUA LUNs are required for Solaris.

For vMOS 6.x, Asymetric ALUA LUNs are supported.

vMOS 5.x has a Symetric LUN implementation, however the LUNs should be identified as ALUA for Solaris to properly identify them. Ensure that the ALUA bit is enabled when creating the LUNs.

Enabling Mulitpathing

Note: For multipathing to work properly, it is important to ensure that there are two blank spaces between "VIOLIN" and "SAN ARRAY" / "SAN ARRAY ALUA" in the device-type-scsi-options-list and the scsi-vhci-failover-override parameters when used in the /kernel/drv/scsi_vhci.conf file.



For ALUA LUNs for Solaris 10/11 and vMOS 6.x, add the following paragraph to the end of the /kernel/drv/scsi_vhci.conf file:

scsi-vhci-failover-override ="VIOLIN SAN ARRAY ALUA","f_tpgs";

Note: There are two spaces between VIOLIN and SAN ARRAY ALUA.

For Solaris 10/11, make sure the following options are also set in this file:

load-balance="round-robin"; auto-failback="enable";

For Active/Active LUNs for Solaris 10 and vMOS 5.x, add the following paragraph to the end of the /kernel/drv/scsi_vhci.conf file:

device-type-scsi-options-list ="VIOLIN SAN ARRAY", "symmetric-option";symmetric-option = 0x1000000;

For Active/Active LUNs for Solaris 11 and vMOS 5.x, add the following paragraph to the end of the /kernel/drv/scsi_vhci.conf file:

scsi-vhci-failover-override ="VIOLIN SAN ARRAY","f_sym";

To enable multipathing for SPARC machines, edit the /kernel/drv/fp.conf file and add the following:

mpxio-disable="no";

To make the changes persist:

Note whenever you change any parameters for mpxio, issue the following command:

# stmsboot -u

To enable device multipathing:

# stmsboot -e

To disable device multipathing:

# stmsboot -d

Discovering LUNs

To discover exported LUNs for Solaris 10 and 11, first issue the following command to verify the target connections:

# cfgadm -al


Solaris

To discover new LUNs:

# cfgadm -c configure <controller #>

For example:

# cfgadm -c configure c8

Or run the cfgadm command for each target:

# cfgadm -c confgure c4::21000024ff45f9bd# cfgadm -c configure c4::21000024ff460e19# cfgadm -c configure c4::21000024ff460e08# cfgadm -c configure c4::21000024ff460e90

Checking Paths

To check paths on a specific LUN, do the following:

1. Run the format command and note the device ID string.

# format

Searching for disks...done

AVAILABLE DISK SELECTIONS:

1. c0t6001B970AFE01A9DAFE01A9D1C8CF35Ad0 <VIOLIN-SANARRAYALUA-630 cyl 1303 alt 2 hd 255 sec 63>

/scsi_vhci/ssd@g6001b970afe01a9dafe01a9d1c8cf35a

Specify disk (enter its number):

2. Grep for the device ID string in luxadm probe command output:

# luxadm probe | grep c0t6001B970AFE01A9DAFE01A9D1C8CF35Ad0

Logical Path:/dev/rdsk/c0t6001B970AFE01A9DAFE01A9D1C8CF35Ad0s2

3. Use this logical path as an argument to the ‘luxadm display’ command to see all paths

# luxadm display /dev/rdsk/c0t6001B970AFE01A9DAFE01A9D1C8CF35Ad0s2

DEVICE PROPERTIES for disk: /dev/rdsk/c0t6001B970AFE01A9DAFE01A9D1C8CF35Ad0s2

Vendor: VIOLIN

Product ID: SAN ARRAY ALUA

Revision: 630

Serial Num: AFE01A9D1C8CF35A

Unformatted capacity: 10240.000 MBytes

Write Cache: Enabled

Device Type: Disk device

Path(s):

/dev/rdsk/c0t6001B970AFE01A9DAFE01A9D1C8CF35Ad0s2

/devices/scsi_vhci/ssd@g6001b970afe01a9dafe01a9d1c8cf35a:c,raw

Controller /devices/pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0/fp@0,0



Device Address 21000024ff39f7f0,8

Host controller port WWN 21000024ff3ee778

Class secondary

State ONLINE


Device Address 2100001b32941834,8


Class secondary

State ONLINE


Device Address 21000024ff48a426,8


Class primary

State ONLINE


Device Address 21000024ff3a31ce,8


Class primary

State ONLINE

Controller /devices/pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0

Device Address 21000024ff39f7f0,8


Class secondary

State ONLINE


Device Address 2100001b32941834,8


Class secondary

State ONLINE


Device Address 21000024ff48a426,8


Class primary

State ONLINE


Device Address 21000024ff3a31ce,8


Class primary

State ONLINE


Solaris

Adding New LUNs with Solaris 10

When adding new LUNs to Solaris 10 while I/O is running on other LUNs, the native multipathing software may not correctly display the multipath LUNs when running the format, mpathadm or luxadm commands. When this occurs, do one of the following:

• Stop I/O when exporting new LUNs from the Violin array.

or

• Reboot the server after exporting the new LUNs.

For Violin 7x00

Solaris 10

Violin 7x00 LUNs are only ALUA LUNs.

Enabling Mulitpathing

For Solaris 10, add the following paragraph to the end of the /kernel/drv/scsi_vhci.conf file:

device-type-scsi-options-list =

"VIOLIN CONCERTO ARRAY", "symmetric-option";

symmetric-option = 0x1000000;

Note: For multipathing to work properly, it is important to ensure that there are two blank spaces between "VIOLIN" and "SAN ARRAY" in the scsi-vhci-failover-override parameter when used in the /kernel/drv/scsi_vhci.conf file.

To enable multipathing for SPARC machines, edit the /kernel/drv/fp.conf file and add the following:

mpxio-disable="no";

To make the changes persist:

Note whenever you change any parameters for mpxio, issue the following command:

# stmsboot -u

To enable device multipathing:

# stmsboot -e

To disable device multipathing:

# stmsboot -d



Exporting LUNs to Solaris 10

To discover LUN 0 exported to the Solaris 10 client, run the "cfgadm -c configure" command:

# cfgadm -al

c3

c3::2100001b97189090 disk connected unconfigured unknown


c4 fc-fabric connected unconfigured unknown



# cfgadm -c configure c3::2100001b97189090




Tuning Recommendations for Solaris SPARC

Tuning variable # 1:

This parameter is responsible for the expected minimum time in milliseconds to service a transfer completion interrupt and initiate a new transfer on the same disk. The default value is 4 milliseconds. We recommend changing it to 30 milliseconds.

1. Edit /etc/system file and add the text " set ssd:ssd_rot_delay=0x1e ".

2. Reboot the host with command " reboot -- -r "

3. Issue command " echo "ssd_rot_delay::print" | mdb -k " to verify.

Tuning variable # 2:

This parameter is responsible for the timing on how the fcp driver delays the offline trigger. If the device comes back within this interval, this timer safeguards premature offline events. The default is 20 seconds. We recommend changing it to 60 seconds.

1. Edit /kernel/drv/fcp.conf file and add the text " set fcp:fcp_offline_delay=0x3c; ".

2. Reboot the host with command "reboot -- -r "

3. Issue command " echo "fcp_offline_delay::print" | mdb -k " to verify.

Note: If you are using more than 50 Violin LUNs on a particular Solaris SPARC hosts, please contact Violin Memory Support for additional recommendations.


HP-UX

HP-UX

Checking Paths of Multipathed Disks

Note: The following applies to both the Violin 6000 and Concerto 7x00.

To check paths of multipathed disks, run the following command:

# scsimgr lun_mapLUN PATH INFORMATION FOR LUN : /dev/rdisk/disk286

Total number of LUN paths = 8World Wide Identifier(WWID) = 0x6001b970f29d0ac9f29d0ac9ecbdc3bb

LUN path : lunpath2493Class = lunpathInstance = 2493Hardware path = 0/4/0/0/0/0.0x21000024ff204931.0x9000000000000SCSI transport protocol = fibre_channelState = UNOPENLast Open or Close state = ACTIVE

LUN path : lunpath2474Class = lunpathInstance = 2474Hardware path = 0/4/0/0/0/0.0x21000024ff39f98b.0x9000000000000SCSI transport protocol = fibre_channelState = UNOPENLast Open or Close state = STANDBY

LUN path : lunpath2489Class = lunpathInstance = 2489Hardware path = 0/4/0/0/0/0.0x21000024ff3a2d4b.0x9000000000000SCSI transport protocol = fibre_channelState = UNOPENLast Open or Close state = STANDBY

LUN path : lunpath2486Class = lunpathInstance = 2486Hardware path = 0/4/0/0/0/0.0x2101001b32b4a38f.0x9000000000000SCSI transport protocol = fibre_channelState = UNOPENLast Open or Close state = ACTIVE



LUN path : lunpath2506Class = lunpathInstance = 2506Hardware path = 0/4/0/0/0/1.0x21000024ff204931.0x9000000000000SCSI transport protocol = fibre_channelState = UNOPENLast Open or Close state = ACTIVE

LUN path : lunpath2503Class = lunpathInstance = 2503Hardware path = 0/4/0/0/0/1.0x21000024ff39f98b.0x9000000000000SCSI transport protocol = fibre_channelState = UNOPENLast Open or Close state = STANDBY

LUN path : lunpath2536Class = lunpathInstance = 2536Hardware path = 0/4/0/0/0/1.0x2101001b32b4a38f.0x9000000000000SCSI transport protocol = fibre_channelState = UNOPENLast Open or Close state = ACTIVE

LUN path : lunpath2527Class = lunpathInstance = 2527Hardware path = 0/4/0/0/0/1.0x21000024ff3a2d4b.0x9000000000000SCSI transport protocol = fibre_channelState = UNOPENLast Open or Close state = STANDBY

Concerto 7x00 LUN Requirements

LUN 0 must be exported from the Concerto 7x00 in order for the HP-UX machine to see the LUNs.

Exporting LUNs to HP-UX 11 Client

Note: LUN 0 must be exported from the 7x00 in order for the HP-UX client to properly see the LUNs.


HP-UX

Discovering LUNs

Run the following commands on HP-UX to discover the newly mapped LUNs.

#ioscan –fnC disk#ioscan –m lun

disk 821 64000/0xfa00/0x339 esdisk CLAIMED DEVICE online VIOLIN CONCERTO ARRAY0/2/0/0/0/1.0x2100001b97189080.0x40060000000000000/2/0/0/0/1.0x2100001b97189030.0x40060000000000000/2/0/0/0/0.0x2100001b97189081.0x40060000000000000/2/0/0/0/0.0x2100001b97189031.0x4006000000000000/dev/disk/disk821 /dev/rdisk/disk821disk 822 64000/0xfa00/0x33a esdisk CLAIMED DEVICE online VIOLIN CONCERTO ARRAY0/2/0/0/0/1.0x2100001b97189080.0x40050000000000000/2/0/0/0/1.0x2100001b97189030.0x40050000000000000/2/0/0/0/0.0x2100001b97189081.0x40050000000000000/2/0/0/0/0.0x2100001b97189031.0x4005000000000000/dev/disk/disk822 /dev/rdisk/disk822

Checking Paths

1. Run the following command to check the paths for discovered LUNs.

# scsimgr lun_map

LUN PATH INFORMATION FOR LUN: /dev/rdisk/disk821

Total number of LUN paths = 4

World Wide Identifier (WWID) = 0x6001b9757a08574925eb00005567acd6

LUN path : lunpath626

Class = lunpath

Instance = 626

Hardware path = 0/2/0/0/0/1.0x2100001b97189030.0x4006000000000000

SCSI transport protocol = fibre_channel

State = UNOPEN

Last Open or Close state = ACTIVE


Class = lunpath

Instance = 638



State = UNOPEN





Class = lunpath

Instance = 678



State = UNOPEN



Class = lunpath

Instance = 653



State = UNOPEN


2. Run the following command to verify the native multipathing policy:

#scsimgr get_attr -D /dev/rdisk/disk821 -a load_bal_policy

SCSI ATTRIBUTES FOR LUN: /dev/rdisk/disk821

name = load_bal_policy

current = round_robin

default = round_robin

saved =

3. Run the following command to verify that ALUA is enabled on the discovered LUNs:

# scsimgr get_attr -D /dev/rdisk/disk821 -a alua_enabled

SCSI ATTRIBUTES FOR LUN: /dev/rdisk/disk821

name = alua_enabled

current = true

default = true

4. Run the following command to verify the health of the Concerto LUNs:

# ioscan -P health -C disk

Class I H/W Path health

==============================================

disk 821 64000/0xfa00/0x339 on


SAN Appliances

SAN Appliances

IBM SAN Volume Controller (SVC)

Note: The following applies to the Violin 6000.

Refer to IBM’s interoperability matrix for supported operating systems with the SVC.

SVC Zoning

Connect Violin arrays via Fibre Channel (FC) switches because the SVC does not support direct connection. The fabric must have three distinct zones:

1. SVC clustered system zone: Create one zone per fabric, with all of the SVC ports cabled to this fabric to allow SVC internode communication. Storage and hosts should not be included.

2. Host zones: Create an SVC host zone for each server that is accessing storage from the SVC system. The host zone should include all initiator-oriented SVC port WWNs and the host HBA port WWNs.

3. Storage zone: Create one SVC storage zone for each Violin array subsystem that is virtualized by the SVC. This is important because all SVC nodes need to be able to access the LUNs being created; otherwise, the SVC will report it is in a degraded state. The storage zone should include all target-oriented SVC port WWNs and the Violin array target port WWNs.

Host bus adapters (HBAs) in dissimilar hosts, or dissimilar HBAs in the same host, need to be in separate zones. For example, AIX and Windows hosts need to be in separate zones. In this case, “dissimilar” means that the hosts are running separate operating systems or are using separate hardware platforms. Therefore, various levels of the same operating system are regarded as similar.

Note that this requirement is a SAN interoperability issue, rather than an SVC requirement. Host zones must contain only one initiator (HBA) each and as many SVC Violin arrays with IBM System Storage SAN Volume Controller node ports as you need, depending on the high availability and performance that you want from your configuration.

Presently each Memory Gateway (MG) zone port shows as a controller on the SVC external storage screen. If eight ports of the MG are zoned, then eight controllers will be presented on the SVC external storage screen. MDisks will appear as if they are only available on a single controller, but they are actually accessible on all the defined controllers.

As shown in the screen below, each MG port on a Violin array shows up as a controller under Pools > External Storage. LUNs exported from the Violin array to the SVC host show up only under one controller; however, they are balanced and seen by all controllers.


http://www-03.ibm.com/systems/storage/software/virtualization/svc/interop.html


LUN Sector Size Considerations for SVC

LUNs must be created with a 512-byte sector size on the Violin array. The SVC does not support 4096-byte sector disks.

MDisk Size Considerations for SVC

To achieve better performance, Violin recommends having a few large MDisks, rather than many small ones. Violin supports MDisk sizes of 2TB or more. For SVC performance considerations, refer to IBM SVC Best Practice and Performance Guidelines.

Multipathing Support for SVC

Host multipathing for volumes is provided on the SVC through the IBM Subsystem Device Driver (SDD). This is provided by IBM for supported platforms, unless otherwise stated in the hardware interoperability matrix. Support for multipathing of MDisks is provided within the SVC itself.

Disabling Caching on LUNs

Unless you are using copy services on the SVC, Violin recommends disabling caching on the volumes used within the SVC. To do this:

1. Navigate to Volumes > Volumes, and select the volumes for which you want to disable caching.

2. Right-click a volume and select Properties.

The following screen is displayed:


http://www.redbooks.ibm.com/redpieces/abstracts/sg247521.html

SAN Appliances

3. To disable Cache Mode, make sure the box next to Show Details is checked, and then click the Edit button. Clear the box that enables Cache Mode, then click Save to save the changes.

4. Repeat this procedure for each vDisk.



FalconStor Network Storage Server

Note: The following applies to the Violin 6000 only.

Zoning

FalconStor NSS requires isolated zoning, where one initiator is zoned to one target in order to minimize I/O interruptions by non-related FC activities, such as port login/out, rest, etc. With isolated zoning, each zone can contain no more than two ports or two WWPNs. This applies to both initiator zones (storage) and target zones (hosts).

For example, for the case of upstream (to host) zoning, if there are two host initiators, and two NSS targets on the same FC fabric, four specific zones, one for each path should be used. The same should be done for downstream (to storage) zoning. If there are two NSS initiators and two storage targets on the same fabric, there should be four zones.

Make sure that the storage devices are not zoned directly connected to the host.

Device Configuration File

For vMOS 6.x for ALUA LUNs, on both Dell servers, make sure that the string VIOLIN SAN ARRAY ALUA is added to the /usr/local/ipstor/etc/certified_alua_storage.conf file.

Note: It is important to ensure that there are two blank spaces between VIOLIN and SAN ARRAY ALUA in the /usr/local/ipstor/etc/certified_alua_storage.conf file.

For vMOS 5.x for Active/Active LUNs, on both Dell servers, make sure that the string VIOLIN SAN ARRAY is added to the /usr/local/ipstor/etc/certified_active_active_storage.conf file.

Note: It is important to ensure that there are two blank spaces between VIOLIN and SAN ARRAY ALUA in the /usr/local/ipstor/etc/certified_active_active_storage.conf file.

For both cases above, reboot both servers after adding the files.

Load Balancing

To get the paths from the "Standby-Active" state back to "Load Balancing" state after a downstream path (between the array and the appliance) is recovered, FalconStor requires all FC adapters to be scanned. For example, this is needed after the reboot of one of the array gateway modules, or after a FC cable pull between the appliance and the array.

Multipathing Software

Refer to the FalconStor Multipath Configuration Best Practice Guides for using multipathing software on the hosts.


Citrix XenServer 6.5

Citrix XenServer 6.5

For Violin 7x00


defaults { user_friendly_names no replace_wwid_whitespace yes dev_loss_tmo 30}

devices { device { vendor "VIOLIN" product "CONCERTO ARRAY" features "1 queue_if_no_path" path_selector "round-robin 0" path_grouping_policy "multibus" hardware_handler "1 alua" prio "alua" failback immediate path_checker "tur" rr_min_io 100 rr_weight "priorities" no_path_retry 300 }}




Windows 2012

CHAPTER 5 Operating System Considerations for iSCSI

Windows 2012

iSCSI Initiator Setup

The iSCSI Initiator Properties dialog can be found under Server Manager > Tools > iSCSI Initiator.

1. Navigate to the ‘Discovery’ tab, and click ‘Discover Portal’. Enter the IP address and port of the target you want to connect to and click ‘OK’. Do this for each target on the Array that you would like to connect to.


Chapter 5 - Operating System Considerations for iSCSI


a. Click the Targets tab. For the Violin 6000, the ‘Discovered Targets’ window should now show the initiator names of the targets on the array.


Windows 2012

b. To connect to the Array’s initiators, select them and click Connect. Check the box ‘Enable Multipath’, then click OK.




Windows 2012

c. The screen above shows that the target initiators are now connected. You can now export LUNs to the host and have them be discovered.

d. Ensure that the Discovered Targets have a status of “Connected”.




a. Click the Targets tab. For the Concerto 7x00, the 'Discovered Targets' window should now show the initiator names of the targets on the Concerto 7x00.


Red Hat Enterprise Linux 6.x & 5.x

b. To connect to the Concerto 7x00's initiators, select them and click Connect. Check the Enable Multipath box, then click OK.

c. The target initiators should be now connected. Ensure that the Discovered Targets have a status of Connected. You can now export LUNs to the host and have them be discovered.


For Violin 6000

Configuring iscsid.conf Parameters

Configuring these parameters on the host can ensure that the I/O pause time is reduced to an appropriate value.

The configuration file can be found here: /etc/iscsi/iscsid.conf. The parameters that need to be changed are:

• node.session.timeo.replacement_timeout = 10

• node.session.initial_login_retry_max = 4

After making these changes, restart the iscsid daemon, log out of all existing iSCSI sessions, and remove IP bindings on your hosts’ interfaces. Then set the IP bindings on the host, set the MTU to 9000 (Jumbo), and log into the iSCSI sessions you wish to connect to.



To make sure that the changes have taken place:

1. List all sessions and find the node you would like to check the parameter changes on:

[root@vmem-is1-cn3 iscsi]# iscsiadm -m sessiontcp: [10] 172.16.25.102:3260,1 iqn.2004-02.com.vmem:vmem-is1-a2-mgbtcp: [11] 172.16.24.102:3260,1 iqn.2004-02.com.vmem:vmem-is1-a2-mgb:eth4tcp: [12] 172.16.25.102:3260,1 iqn.2004-02.com.vmem:vmem-is1-a2-mgb:eth5tcp: [13] 172.16.25.101:3260,1 iqn.2004-02.com.vmem:vmem-is1-a2-mga:eth5tcp: [14] 172.16.24.101:3260,1 iqn.2004-02.com.vmem:vmem-is1-a2-mga:eth4tcp: [9] 172.16.24.102:3260,1 iqn.2004-02.com.vmem:vmem-is1-a2-mgb

2. Execute the iscsiadm command with the node option and the node you would like to check

[root@vmem-is1-cn3 iscsi]# iscsiadm -m node -T iqn.2004-02.com.vmem:vmem-is1-a2-mgb:eth4 -p 172.16.24.102:3260

# BEGIN RECORD 2.0-872.16.el5

node.name = iqn.2004-02.com.vmem:vmem-is1-a2-mgb:eth4

node.tpgt = 1

node.startup = automatic

node.leading_login = No

iface.hwaddress = <empty>

iface.ipaddress = <empty>

iface.iscsi_ifacename = default

iface.net_ifacename = <empty>

iface.transport_name = tcp

iface.initiatorname = <empty>

iface.bootproto = <empty>

iface.subnet_mask = <empty>

iface.gateway = <empty>

iface.ipv6_autocfg = <empty>

iface.linklocal_autocfg = <empty>

iface.router_autocfg = <empty>

iface.ipv6_linklocal = <empty>

iface.ipv6_router = <empty>

iface.state = <empty>

iface.vlan_id = 0

iface.vlan_priority = 0

iface.vlan_state = <empty>

iface.iface_num = 0

iface.mtu = 0

iface.port = 0

node.discovery_address = 172.16.24.102

node.discovery_port = 3260

node.discovery_type = send_targets

node.session.initial_cmdsn = 0

node.session.initial_login_retry_max = 4

node.session.xmit_thread_priority = -20

node.session.cmds_max = 128

node.session.queue_depth = 32



node.session.nr_sessions = 1

node.session.auth.authmethod = None

node.session.auth.username = <empty>

node.session.auth.password = <empty>

node.session.auth.username_in = <empty>

node.session.auth.password_in = <empty>

node.session.timeo.replacement_timeout = 10

node.session.err_timeo.abort_timeout = 15

node.session.err_timeo.lu_reset_timeout = 30

node.session.err_timeo.tgt_reset_timeout = 30

node.session.err_timeo.host_reset_timeout = 60

node.session.iscsi.FastAbort = Yes

node.session.iscsi.InitialR2T = No

node.session.iscsi.ImmediateData = Yes

node.session.iscsi.FirstBurstLength = 262144

node.session.iscsi.MaxBurstLength = 16776192

node.session.iscsi.DefaultTime2Retain = 0

node.session.iscsi.DefaultTime2Wait = 2

node.session.iscsi.MaxConnections = 1

node.session.iscsi.MaxOutstandingR2T = 1

node.session.iscsi.ERL = 0

node.conn[0].address = 172.16.24.102

node.conn[0].port = 3260

node.conn[0].startup = manual

node.conn[0].tcp.window_size = 524288

node.conn[0].tcp.type_of_service = 0

node.conn[0].timeo.logout_timeout = 15

node.conn[0].timeo.login_timeout = 15

node.conn[0].timeo.auth_timeout = 45

node.conn[0].timeo.noop_out_interval = 5

node.conn[0].timeo.noop_out_timeout = 5

node.conn[0].iscsi.MaxXmitDataSegmentLength = 0

node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144

node.conn[0].iscsi.HeaderDigest = None

node.conn[0].iscsi.IFMarker = No

node.conn[0].iscsi.OFMarker = No

# END RECORD



For Concerto 7100/7200

Configuring iscsd.conf Parameters

There is no change to the parameters in Configuring /etc/iscsi/iscsid.conf and the default values should be used. Check to make sure that these parameters are set to the following:

• node.session.timeo.replacement_timeout = 120

• node.session.initial_login_retry_max = 8

1. List all sessions to check the parameter settings.

[root@vmem-srv15 iscsi]# iscsiadm -m session

tcp: [1] 10.100.10.11:3260,0 iqn.2004-02.com.vmem-concerto.lab-srv3567.vmem-srv15-0




2. Execute the iscsiadm command with the node option and the node you would like to check.

[root@vmem-srv15 iscsi]# iscsiadm -m node -T iqn.2004-02.com.vmem-concerto.lab-srv3565.vmem-srv15-0 -p 10.100.10.10:3260

# BEGIN RECORD 6.2.0-873.2.el6

node.name = iqn.2004-02.com.vmem-concerto.lab-srv3565.vmem-srv15-0

node.tpgt = 0





iface.iscsi_ifacename = default


iface.transport_name = tcp











iface.vlan_id = 0



iface.iface_num = 0

iface.mtu = 0

iface.port = 0















































# END RECORD



Checking for Paths

For Violin 6000

When using device mapper multipathing, use the multipath command to check paths.

Active/Active Example

[root@vmem-10-1-15-103 ~]# multipath -llmpatha (36001b970f29d0ac9f29d0ac9913baff0) dm-0 VIOLIN,SAN ARRAYsize=100G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 1:0:0:0 sda 8:0 active ready running |- 1:0:2:0 sdc 8:32 active ready running |- 1:0:3:0 sdd 8:48 active ready running |- 1:0:1:0 sdb 8:16 active ready running |- 2:0:0:0 sde 8:64 active ready running |- 2:0:1:0 sdf 8:80 active ready running |- 2:0:2:0 sdg 8:96 active ready running `- 2:0:3:0 sdh 8:112 active ready running

ALUA Example

[root@vmem-srv13 ~]# multipath -llmpathacp (SVIOLIN_SAN_ARRAY_ALUA_F29D0AC9EC89F492) dm-5 VIOLIN,SAN ARRAY ALUAsize=10G features='0' hwhandler='1 alua' wp=rw|-+- policy='round-robin 0' prio=130 status=active| |- 2:0:2:3 sdl 8:176 active ready running| |- 2:0:3:3 sdp 8:240 active ready running| |- 1:0:2:3 sdab 65:176 active ready running| `- 1:0:3:3 sdaf 65:240 active ready running`-+- policy='round-robin 0' prio=10 status=enabled |- 2:0:0:3 sdd 8:48 active ready running |- 2:0:1:3 sdh 8:112 active ready running |- 1:0:0:3 sdt 65:48 active ready running `- 1:0:1:3 sdx 65:112 active ready running

For Concerto 7100/7200

[root@vmem-srv15 ~]# multipath -llmpathz (36000d77a00008b230bfcfb1e5fc3f51c) dm-8 VIOLIN,CONCERTO ARRAYsize=49G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=130 status=active |- 14:0:0:0 sdm 8:192 active ready running `- 13:0:0:0 sdl 8:176 active ready running



Symantec Veritas SFHA Solution

For Violin 6000

Violin recommends the defaults for Symantec Veritas SFHA multipathing.

To list the paths of a given disk:

[root@vmem-cn10 ~]# vxdmpadm getsubpaths dmpnodename=aluadisk0_100NAME STATE[A] PATH-TYPE[M] CTLR-NAME ENCLR-TYPE ENCLR-NAME ATTRS=======================================================================sdaz ENABLED(A) PRIMARY c12332 aluadisk aluadisk0 -sddl DISABLED PRIMARY c12323 aluadisk aluadisk0 -sdfx ENABLED(A) PRIMARY c12324 aluadisk aluadisk0 -sdij ENABLED(A) PRIMARY c12333 aluadisk aluadisk0 -sdkv DISABLED PRIMARY c12326 aluadisk aluadisk0 -sdnh DISABLED PRIMARY c12327 aluadisk aluadisk0 -sdpt ENABLED(A) PRIMARY c12328 aluadisk aluadisk0 -sdsf ENABLED(A) PRIMARY c12334 aluadisk aluadisk0 -




Configuring the Host

CHAPTER 6 InfiniBand Best Practices

Configuring the Host

Downloading and Installing the Driver

Contact Violin Memory Customer Support for the currently supported driver, and download and install the appropriate driver from Mellanox. Drivers can be found here.

Scroll to the bottom of the page and click on the Archive tab to download the driver currently supported by the Violin Interoperability Matrix.

Configuring Driver Parameters

After installing the driver, Violin recommends changing parameters in the openib configuration file, /etc/infiniband/openib.conf, to the values listed below. See the Sample ‘/etc/infiniband/openib.conf’ Files section for an example file with the appropriate changes. (Note that these configuration file changes apply to the driver version 2.0-2.0.5.)

• IPOIB_LOAD=no

• SRP_LOAD=yes

• SRPHA_ENABLE=yes

• SRP_DAEMON_ENABLE=yes

Reboot the host after these changes have been made.

LUN Discovery Troubleshooting

Ensure that the open subnet manager daemon is set to ‘on’ for the run levels you will be using.

• chkconfig --list | grep opensmd

To check IB link status:

• /usr/bin/hca_self_test.ofed

If you see port errors, try clearing them with the command below and then rerun the test above.

• ibdiagnet –pc


http://www.mellanox.com/page/software_overview_ib

Chapter 6 - InfiniBand Best Practices


Enabling iSER on the Array

CHAPTER 7 iSER Best Practices

Enabling iSER on the Array

1. Log into the master MG.

2. Enable iSER.

[memgate-cluster: master] (config) # iser enable global


3. Identify and confirm that each InfiniBand interface is online.

[memgate-cluster: master] (config) # show int ib0

Interface ib0 status:

Comment:

Admin up: yes

Link up: yes

IP address:

Netmask:

Speed: 40 Gb/sec (4X QDR)

Duplex: full

Interface type: ib



MTU: 2044

HW address: 80:00:00:48:fe:80:00:00:00:00:00:00:00:02:c9:03:00:4e:71:95








TX queue len: 256

4. Configure the interface IP address and netmask for each interface. Remember to configure the interfaces on both MGs.

[memgate-cluster: master] (config) # interface ib0 ip address 192.168.0.101 /24


5. Add additional IP addresses are to the same interface if required:

[memgate-cluster: master] (config) # interface ib0 alias 0 ip address 192.168.2.101 /24



Chapter 7 - iSER Best Practices

6. Verify that the changes have taken effect.

[memgate-cluster: master] (config) # sho int ib0

Interface ib0 status:

Comment:

Admin up: yes

Link up: yes

IP address: 192.168.0.101

Netmask: 255.255.255.0

Secondary address: 192.168.2.101/24 (alias: 'ib0:0')

Speed: 40 Gb/sec (4X QDR)

Duplex: full

Interface type: ib



MTU: 2044

HW address: 80:00:00:48:fe:80:00:00:00:00:00:00:00:02:c9:03:00:4e:71:95








TX queue len: 256

Note: Remember to do the above steps on each interface and on both MGs that you want to configure.

Creating an iSER Target on the Array

iSER assigns iSCSI targets to InfiniBand interfaces.

1. Create an iSCSI target on the master MG.

ssh admin@vmem-mg

violin-mga [violin-mg: master] > en

violin-mga [violin-mg: master] # co t

violin-mga [violin-mg: master] (config) # iscsi target create iser-0

violin-mga [violin-mg: master] (config) # write memory

2. Bind the interface’s IP addresses to the iSCSI target on the Master-MG.

violin-mga [violin-mg: master] (config) # iscsi target bind iser-0 to 192.168.0.101

violin-mga [violin-mg: master] (config) # write memory

3. Bind the interface’s IP addresses to the iSCSI target on the Standby-MG.


Operating System Considerations for iSER

ssh admin@violin-mgb

violin-mgb [violin-mg: standby] > en

violin-mgb [violin-mg: standby] # conf t

violin-mgb [violin-mg: standby] (config) # iscsi target bind iser-0 to 192.168.0.102

violin-mgb [violin-mg: standby] (config) # write memory

4. Confirm that the interfaces are bound.

violin-mga [violin-mg: master] (config) # show targetsNode Hostname Proto Target State Address --------------------------------------------------------------------1 violin-mga ib hca-a1 Active guid.0002:c903:004e:7195 1 violin-mga ib hca-a2 Down guid.0002:c903:004e:7196 1 violin-mga iser iser-0 Active iqn.2004-02.com.vmem:violin-mga:iser-02 violin-mgb ib hca-a1 Active guid.0002:c903:002b:0fab 2 violin-mgb ib hca-a2 Down guid.0002:c903:002b:0fac 2 violin-mgb iser iser-0 Active iqn.2004-02.com.vmem:violin-mgb:iser-0


Redhat Enterprise Linux 6.4

Driver and Utility Installation

1. Download and install the latest iSER-supporting OFED drivers (currently version 2.2-1.0.1) from Mellanox:

http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers

2. Install the latest iscsi-initiator-utils.

Configuring InfiniBand Settings to Use iSER Over SRP

Configuring these parameters on the host enables iSER and disables SRP. Failure to make these changes will prevent iSER from functioning properly.

The configuration file is /etc/infiniband/openib.conf. The following must be changed:

If present, remove the following lines:

# Load UMAD module

UMAD_LOAD=yes

# Load UVERBS module

UVERBS_LOAD=yes

Modify the following lines:

MTHCA_LOAD=no


http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers


QIB_LOAD=no

IPATH_LOAD=no

MLX4_LOAD=yes

MLX5_LOAD=no

MLX4_EN_LOAD=yes

MLX4_VNIC_LOAD=no

CXGB3_LOAD=no

NES_LOAD=no

IPOIB_LOAD=yes

SET_IPOIB_CM=auto

E_IPOIB_LOAD=no

SRP_LOAD=no

RDS_LOAD=no

SRPHA_ENABLE=no

SRP_DAEMON_ENABLE=no

You must reboot the host for these parameters to take effect. See page 104 for a sample /etc/infiniband/openib.conf file for iSER.

Configuring iscsid.conf Parameters

Configuring these parameters on the host can ensure that the I/O pause time is reduced to an appropriate value.

The configuration file is /etc/iscsi/iscsid.conf. The following parameters must be changed:








After making these changes, log out of all existing iSCSI sessions, delete all known sessions with the iscsiadm ‘-o delete’ flag, restart the iscsid daemon, then rediscover and log into targets.

Logging in Sessions

When using iSER, the iscsiadm -I flag must be set to iser:

iscsiadm -m discovery -t sendtargets -p 192.168.0.101 -I iser -l



Verifying the iSCSI Changes

To verify that the iSCSI changes you made have taken effect:

1. List all active sessions and find the node you want to verify:

[root@vmem-srv2461 ~]# iscsiadm -m sessioniser: [63] 192.168.0.101:3260,1 iqn.2004-02.com.vmem:violin-mga:iser-0iser: [64] 192.168.0.102:3260,1 iqn.2004-02.com.vmem:violin-mgb:iser-0

2. Execute the iscsiadm command with the node option and the node you want to verify.

[root@vmem-srv2461 ~]# iscsiadm -m node -T iqn.2004-02.com.vmem:violin-mga:iser-0 -p 192.168.0.101

# BEGIN RECORD 6.2.0-873.10.el6

node.name = iqn.2004-02.com.vmem:violin-mga:iser-0

node.tpgt = 1





iface.iscsi_ifacename = iser


iface.transport_name = iser











iface.vlan_id = 0



iface.iface_num = 0

iface.mtu = 0

iface.port = 0















































# END RECORD


Appendix

Sample ‘/etc/infiniband/openib.conf’ Files

The /etc/infininband/openib.conf file is used in InfiniBand and iSER configurations. Following are sample versions of the file for InfiniBand and iSER.

InfiniBand

(Changes made in bold)

# Start HCA driver upon bootONBOOT=yes

# Node descriptionNODE_DESC=$(hostname -s)

# Wait for NODE_DESC_TIME_BEFORE_UPDATE sec before node_desc updateNODE_DESC_TIME_BEFORE_UPDATE=10

# Set rx_channels/tx_channels to 1 to disable IPoIB RSS/TSSSET_IPOIB_CHANNELS=no

# Run /usr/sbin/mlnx_affinityRUN_AFFINITY_TUNER=no

# Load UCM moduleUCM_LOAD=yes

# Load RDMA_CM moduleRDMA_CM_LOAD=yes

# Load RDMA_UCM moduleRDMA_UCM_LOAD=yes

# Increase ib_mad thread priorityRENICE_IB_MAD=no

# Run sysctl performance tuning scriptRUN_SYSCTL=yes

# Load MLX4 modulesMLX4_LOAD=yes


# Load MLX4_EN module


MLX4_EN_LOAD=yes

# Load IPoIBIPOIB_LOAD=no

# Enable IPoIB Connected ModeSET_IPOIB_CM=auto

# Load E_IPoIBE_IPOIB_LOAD=no

# Load SRP moduleSRP_LOAD=yes

# Load ISER moduleISER_LOAD=no

# Load RDS moduleRDS_LOAD=no

# Enable SRP High Availability daemonSRPHA_ENABLE=yesSRP_DAEMON_ENABLE=yes

iSER

# Start HCA driver upon bootONBOOT=yes

# Node descriptionNODE_DESC=$(hostname -s)

# Wait for NODE_DESC_TIME_BEFORE_UPDATE sec before node_desc updateNODE_DESC_TIME_BEFORE_UPDATE=20

# Set rx_channels/tx_channels to 1 to disable IPoIB RSS/TSSSET_IPOIB_CHANNELS=no

# Run /usr/sbin/mlnx_affinityRUN_AFFINITY_TUNER=no

# Load UCM moduleUCM_LOAD=yes

# Load RDMA_CM moduleRDMA_CM_LOAD=yes

# Load RDMA_UCM moduleRDMA_UCM_LOAD=yes


# Increase ib_mad thread priorityRENICE_IB_MAD=no

# Run sysctl performance tuning scriptRUN_SYSCTL=yes

# Load MTHCAMTHCA_LOAD=no

# Load QIBQIB_LOAD=no

# Load IPATHIPATH_LOAD=no


# Load MLX5 modulesMLX5_LOAD=no

# Load MLX4_EN moduleMLX4_EN_LOAD=yes

# Load MLX4_VNIC moduleMLX4_VNIC_LOAD=no

# Load CXGB3 modulesCXGB3_LOAD=no

# Load NES modulesNES_LOAD=no

# Load IPoIBIPOIB_LOAD=yes



# Load SRP moduleSRP_LOAD=no


# Enable SRP High Availability daemon


SRPHA_ENABLE=noSRP_DAEMON_ENABLE=no

# Load MTHCAMTHCA_LOAD=no

# Load QIBQIB_LOAD=no

# Load IPATHIPATH_LOAD=no



# Load MLX4_EN moduleMLX4_EN_LOAD=yes

# Load MLX4_VNIC moduleMLX4_VNIC_LOAD=no



# Load NES modulesNES_LOAD=no

# Load IPoIBIPOIB_LOAD=yes



# Load SRP moduleSRP_LOAD=no


# Enable SRP High Availability daemonSRPHA_ENABLE=no


SRP_DAEMON_ENABLE=no

Mapping of iSCSI Ports to Ethernet Interfaces on the Memory Gateway

The following figure shows the perspective from the back of the array.



Documents

Violin Memory Interoperability Best Practices Guide