HPE Synergy installation and recommended practices guide VMware vSAN ReadyNode on HPE Synergy

Contents About this document ............................................................................................................................................................................................................................................................................................................................ 2

Introduction ............................................................................................................................................................................................................................................................................................................................................ 2

Audience .................................................................................................................................................................................................................................................................................................................................................... 2

Assumptions .......................................................................................................................................................................................................................................................................................................................................... 2

Design scenario ......................................................................................................................................................................................................................................................................................................................................... 2

Blueprint for vSAN on HPE Synergy ...................................................................................................................................................................................................................................................................................... 4

Configuration process .......................................................................................................................................................................................................................................................................................................................... 4

Step 1: vSAN network creation in HPE OneView ................................................................................................................................................................................................................................................. 4

Step 2: HPE OneView server profile templates and vSAN node creation ..................................................................................................................................................................................... 5

Step 3: Installation and setup of ESXi ............................................................................................................................................................................................................................................................................. 6

Step 4: vSAN Health remediation ................................................................................................................................................................................................................................................................................... 13

Summary ...................................................................................................................................................................................................................................................................................................................................................... 17

Resources .................................................................................................................................................................................................................................................................................................................................................... 18

Technical white paper

Technical white paper Page 2

About this document Introduction IT organizations are looking for robust infrastructure, which is flexible, cost-efficient, scalable, and provides a stable platform for mixed workloads without compromising on performance and the IT organization’s ability to meet the required service-level agreements to its users. HPE Synergy is architected from the ground up for Composable Infrastructure. The performance, flexibility, and manageability that HPE Synergy provides help businesses provision efficiently, scale on-premises cloud infrastructure resources quickly, and reduce costs associated with stranded resources.

Composable Infrastructure is the next-generation platform customers are adopting and through partnership, Hewlett Packard Enterprise and VMware® have collaborated to certify HPE Synergy for VMware vSAN. This disaggregated architecture of compute, storage, and fabric allows the infrastructure administrator to avoid fixed ratios found in traditional infrastructure. It also provides a single unified API to assemble the infrastructure to meet business needs. VMware has certified vSAN ReadyNodes for vSAN 6.6/VMware® ESXi™ 6.5 U1 for all-flash configurations and VMware ESXi 6.0 U3 for hybrid deployments on HPE Synergy.

Audience This document is intended to aid infrastructure and virtualization administrators in configuring their HPE Synergy infrastructure for VMware vSAN 6.6.

Assumptions It is assumed that the infrastructure for the use case complies with VMware Hardware Compatibility List (HCL).

Design scenario The solution discussed in this white paper assumes a deployment of at least:

• One HPE Synergy frame including, at a minimum:

– HPE Virtual Connect SE 40Gb F8 Module for Synergy

– Three HPE Synergy 480 Gen9 nodes with HCL compatible components (see “HCL AF-6 example” section)

– HPE Synergy D3940 Storage Module with SAS expanders

– Redundant HPE Synergy Composer modules running HPE OneView 3.1

– Highly available HPE Synergy Frame Link Modules, power supplies, and fans

– VMware® vCenter™ 6.6 appliance (VMware® vCenter™ Server Appliance™ [VCSA]) or stand-alone vCenter installation using Microsoft® Windows Server® and SQL database

• Additionally, administrators may wish to obtain:

– ESXi Image for HPE ProLiant

– HPE OneView for VMware vCenter plug-in appliance

– HPE Synergy Configuration and Compatibility Guide

– kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=2151225

Technical white paper Page 3

HCL AF-6 example

Figure 1. Example of HPE Synergy ReadyNode

Q. Can the administrator change options in the build list? A. Yes, but only certain options. Visit blogs.vmware.com/virtualblocks/2017/03/14/can-cannot-change-vsan-readynode for changes such as boot devices, group configurations, and others. Disk groups must be HCL compatible for minimum—only drives in higher performance SKUs will still be allowed.

Q. Can the administrator use HPE Synergy Image Streamer? A. Yes, according to VMware, modifying the boot device is an allowable configuration change, and this will be covered as part of the document. The administrator must be using the 3-frame logical enclosure to support this use case. For more information on HPE Synergy Image Streamer, visit hpe.com/us/en/product-catalog/synergy/synergy-management/pip.hpe-synergy-image-streamer.1008615214.html.

Q. Does this document represent the only way to deploy vSAN on HPE Synergy? A. No. This document is not intended to be the last word on vSAN deployments on HPE Synergy. Just as the build list for a ReadyNode can be modified, so can the deployment options illustrated in this guide be modified. For example, this guide leverages HPE FlexNic technology as well the VMware vSphere® Distributed Switch™. The vSAN administrator may leverage different networking schemes provided it does not violate any vSAN constraints.

Technical white paper Page 4

Blueprint for vSAN on HPE Synergy The remainder of this document assumes HPE Synergy infrastructure has undergone initial discovery of HPE Synergy frames and installed components. A firmware baseline compliant with VMware HCL should be applied to the logical enclosure to meet compatibility requirements.

The basic process for configuring vSAN is as follows:

Note The steps listed here are generic. If you are unsure of how to perform each step, use the HPE OneView Deployment and Management Guide (DMG) for detailed steps on each portion of the configuration. The page numbers for each step will be listed next to the step:

1. Initial configuration of the HPE Synergy environment (DMG: page 25).

a. Define IP ranges for management and HPE Synergy Composer powered by HPE OneView

b. Service Pack for ProLiant uploaded to HPE Composer powered by HPE OneView

2. Network creation and configuration (bandwidth requested, max. = 20 Gb) (DMG: page 31).

a. Management (0.1 Gb–1 Gb)

b. vSAN (10 Gb)

c. VMware vSphere® vMotion® (4 Gb–8 Gb)

d. Production guest networks (1 Gb–5 Gb)

3. A logical enclosure created within HPE OneView consisting of all HPE Synergy frames interconnected by the same fabric resources (DMG: page 45).

a. Logical interconnect groups

I. 40 Gb module pair

II. SAS module pair

b. Enclosure groups created with appropriate networks and uplinks included in logical interconnect group

c. Firmware baseline SPP added

d. Update logical enclosure firmware

e. HPE Synergy Image Streamer (optional)

4. VMware vCenter 6.5 sets up externally to the logical enclosure with connectivity to the management and production networks.

Configuration process Step 1: vSAN network creation in HPE OneView Depending on the needs of the local environment for VMware vSAN, the private vSAN network can be used internally within the logical enclosure that provides up to 240 Gb east-west traffic. It can also be externally exposed to top-of-rack (ToR) switches to extend the cluster beyond a single logical enclosure. Utilizing an internal-only vSAN network eliminates the need for data center-wide Jumbo Frame enablement on all ToR and core switches as Jumbo Frames are natively enabled with the HPE Synergy fabric.

If the environment requires that vSAN cluster nodes participate with nodes beyond an HPE Synergy logical enclosure, then it is necessary to define the vSAN network on an HPE Synergy logical interconnect group uplink set to enable vSAN traffic to participate in the data center network. Scaling beyond a single logical enclosure can also be accomplished using VMware Stretched Clusters without plumbing the vSAN network externally.

Technical white paper Page 5

Figure 2. Configuring the vSAN network in HPE OneView

The recommended vSAN bandwidth is always 10 Gb for all-flash configurations and from 1 Gb to 10 Gb for hybrid.

Note VLAN tags are not specific to vSAN and can be adjusted by the administrator.

Step 2: HPE OneView server profile templates and vSAN node creation HPE OneView provides infrastructure architects with a software-defined methodology to define a template that specifies all necessary parameters for the compute module. It includes firmware baseline, network connections to deployment, management and production networks, array controller settings, configuration of local and HPE Synergy D3940 enclosures and drives, and BIOS settings.

HPE OneView architects may choose to define templates to use HPE Synergy Image Streamer or PXE-based deployment services or may choose to use manual methods to install VMware ESXi on the front-facing drives in RAID configuration.

The following steps incorporate best practices in accordance with typical ESXi deployments and can be modified to fit the environment’s needs (other than firmware baseline).

In HPE Synergy Image Streamer or PXE-based OS deployments, it is required to map the first port of each LOM (3:1a and 3:2a) to the deployment services. If manual methods are used, ports 3:1a and 3:2a are available for management or other networks.

1. Profile naming and basics

a. Naming and description for vSAN node consistency (that is, vsan node no I3S)

b. Make sure template is applied to correct hardware type for HPE Synergy 480 nodes

2. Firmware baseline attached to correct SPP with HCL approved firmware

3. Network connections (See procedure Blueprint for vSAN on HPE Synergy, step # 2 for bandwidth recommendation)

a. 3:1a and 3:2a Open/administrator use

b. 3:1b and 3:2b Management and/or vMotion

c. 3:1c and 3:2c VM Guests and/or vMotion (depending on 3:1b)

Technical white paper Page 6

d. 3:1d and 3:2d vSAN Storage Network

Note This network configuration and NIC assignment is only one recommendation of many valid configurations and should not be considered the preferred configuration or layout. Other valid configurations could include a redundantly configured Network Set or Tunnel network with all VLANs trunked to each compute node. If leveraging a 2-NIC configuration with a Network Set or Tunnel network, then the VMware Distributed Switch and Network I/O Control (NIOC) is strongly recommended. Networks for replication, backup, or disaster recovery should also be considered as part of the solution.

4. Storage

a. Internal drives RAID 1 (if not using HPE Synergy Image Streamer)

I. Set to Boot

II. Initialize at startup

b. HPE Synergy D3940 drives HBA mode (use administrator predetermined group for cache-to-data ratio; ReadyNode example using 1 to 3)

I. SSD cache drives (one per group)

II. SSDs/HDDs (nodes per group x cache drive amount)

c. SAN storage Not necessary, leave unconfigured

5. Boot settings

a. Boot modes UEFI Optimized

b. PXE Boot Auto

c. Manage Boot Order Hard Disk (unless using PXE or HPE Image Streamer)

6. Optional: BIOS settings

a. Select Manage BIOS

b. Set Power Profile to Max. Performance (optional)

c. Set Power Regulator to Static High Performance Mode (optional)

Now that the server profile template (SPT) has been created, the next step is to apply profiles to each of your available HPE Synergy 480 nodes available for vSAN.

1. Select SPT created earlier and click the Actions tab on right and select Create Server Profile

2. Pick the applicable HPE Synergy 480 node, apply the template, and click OK

Step 3: Installation and setup of ESXi There are two ways to install ESXi that are both valid for HCL compatibility—HPE Synergy Image Streamer or standard ISO installation to local RAID 1 drives. In this document, we will focus on the local media installation procedure.

Note There is a configuration change that must be performed on the ESXi node via Shell commands that allows true active/passive SAS drive failover in ESXi.

1. After all profiles are applied, record the HPE OneView assigned MAC addresses to the networks. These will be necessary to keep congruence with the vmnic (that is, vmnic0 = Mgmt1) when assigning physical NICs to vDS uplinks in later steps.

2. Boot the nodes and install ESXi through HPE iLO mount of ISO media.

3. After installation is completed, select the management networks in setup and configure applicable IP, DNS, Gateway, and FQDN.

Technical white paper Page 7

4. Enable SSH and ESXi Shell.

5. Connect to each host via SSH or local Shell and run this set of commands in the following order:

a. esxcli storage nmp satp rule add -D nhpsa -s VMW_SATP_DEFAULT_AP

b. esxcli storage core claiming unclaim -t driver -D nhpsa

c. esxcli storage core claimrule load

d. esxcli storage core claimrule run

Figure 3. Modifying VMware vSphere® claim rules

Note The unable to unclaim path warning is to be expected. This is pointing to the local boot disk.

Technical white paper Page 8

6. The commands allow ESXi to see both paths to the SAS drives and allow true active/passive failover in case of a SAS connection module or I/O adapter being offline. The following screenshot should be the result shown on each SAS disk.

Figure 4. Storage Devices Path detail

7. If not already created, in vCenter create a new data center as well as a new cluster (vSAN Cluster 1); put your required settings for VMware vSphere® High Availability, VMware vSphere® Distributed Resource Scheduler™, and others. Add the ESXi hosts to this cluster.

Figure 5. vSAN Cluster settings

Technical white paper Page 9

8. In vCenter, create a virtual distributed switch (vDS). Since HPE Synergy profile is configured for six HPE FlexNics, configure the vDS for six uplinks. Uncheck Create a default port group. Right-click and choose Settings, and then Edit Settings. On the General Tab, choose Edit Uplink Names and edit the uplink names to correspond to HPE FlexNics set in the profile. This will help eliminate confusion when hosts are added to the vSwitch.

Figure 6. Edit the Uplink Names

9. If Jumbo Frames will be used, select Advanced, then set MTU at 9000, and optionally change discovery protocol to Link Layer Discovery Protocol. Jumbo Frames are not required for vSAN, but performance gains may be realized if they are leveraged.

Figure 7. vDS Advanced settings

10. Create a Distributed Port Group named Management and accept all defaults.

11. Create a Distributed Port Group vSAN Network and accept all defaults.

12. Continue creating additional Port Groups, configuring VLAN IDs as appropriate.

Technical white paper Page 10

13. Note: HPE Synergy 480 compute nodes do not have LACP available but rather standard LAG configuration (side A/side B). In the VMware document, VMware® vSAN™ Network Design, there are multiple examples of LAG configurations that can be used by either rackmount or blade/modular servers. For this best practices document, the standard Basic NIC Teaming example is used, as the nodes are LAG but the HPE Virtual Connect modules are active/active.

a. Some performance gains may be achieved by adjusting the teaming and load balancing algorithms to Load Based Teaming. These are configured on a per-port-group basis.

14. Right-click the vDS and choose Add and Manage Hosts. Choose the Add Host option and select the Template Mode option. Add each host. Choose the Manage physical network adapters and Manage VMkernel network adapters. For Manage physical network adapters, assign each of the vmnics to the corresponding appropriate vDS uplink name. Choose Apply to all to apply the physical network adapter assignments to all other hosts.

Figure 8. Add and Manage Hosts—Manage Physical Network Adapters

Technical white paper Page 11

15. Migrate the default vmk0 adapter to the vDS.

Figure 9. Add and Manage Hosts—Manage VMkernel Network Adapters

16. Choose New Adapter and create a new VMkernel adapter for vSAN Network. (optionally, continue creating additional VMkernel adapters, such as those to be used for other storage topologies, vMotion, and more).

17. After choosing Finish in this wizard, all hosts and host adapters will be migrated to the VMware Distributed Switch. Following completion of this wizard, additional Port Groups can be created, vMotion settings configured, and so on.

Technical white paper Page 12

18. Next, configure each Port Group to leverage only those uplinks that are carrying that Port Group’s traffic (for example, the vSAN network Port Group should be bound only to the vSAN uplinks).

Figure 10. Assigning Uplinks

19. Go to the vSAN Cluster tab Configure and select the vSAN tab on the left menu. Check General submenu and make sure vSAN is turned ON and all disks are using the correct On-Disk Format Version. If not, proceed to update these. Next, click the Disk Management submenu and if the option to automatically import drives is set to OFF, then the administrator will need to add each host’s disk group manually. This can be done by allowing vSAN to claim the correct cache and capacity tier drives. Once this is completed, VMware will import the drives and place them in the disk groups to be available for use within the cluster.

a. Note: Should a disk be presented to the host via HPE Composer, but not available for vSAN to claim, the most likely culprit is stale partition data. To clear the old partition data, navigate to Configure -> Storage -> Storage Devices, select the offending disk, select All Actions, and choose Erase Partitions.

Figure 11. Erase Partitions

Technical white paper Page 13

20. Under the Monitor tab -> vSAN -> Health, click Retest and follow the prompts to verify warnings and fix issues that vSAN needs to correctbefore proceeding.

Step 4: vSAN Health remediation Depending on the Service Pack for ProLiant used during HPE Synergy profile creation as well as the VMware vSphere HPE Custom ISO used to install VMware vSphere, inconsistencies between the installed driver version and firmware version of the storage controller and what VMware vSAN Hardware Compatibility List (HCL) mandate may result in a warning in the vSAN Health Check service. When an inconsistency arises, the vSAN HCL recommendations take priority. Therefore, for a valid configuration, the driver and firmware for the storage controller must be remediated to what the vSAN HCL dictates.

Figure 12. vSAN Health Check results

Multiple methods can be employed to remediate the host, such as creating a custom SPP with the downgraded firmware or utilizing VMware vSphere® ESXi™ Image Builder CLI to build a custom OEM vSphere ISO with the down-level storage controller.

Remediating the storage controller firmware and driver version to comply with the vSAN HCL The fastest method, depending on the number of hosts in the vSAN cluster, may be to simply download the correct firmware and driver bundles from the HPE Support site.

Once the bundles have been downloaded, the following steps may be used to remediate the host:

If the hosts were created with a managed firmware in HPE Composer, it will be necessary to edit the firmware selection in the host’s HPE OneView Server Profile to manually managed.

Figure 13. Firmware management

Upload the bundle to the vSAN datastore using the vSphere Client or Host Client.

Using HPE iLO or a different method, establish a CLI session on the host.

Technical white paper Page 14

Place the host into maintenance mode by issuing the following command:

esxcli system maintenanceMode set –e true

Copy the firmware and driver bundle to /tmp

Downgrade the firmware Unzip the firmware bundle using the unzip command and the name of the zipped firmware bundle.

Figure 14. Unzipping the firmware

Enable execute permissions on the firmware bundle using chmod

Figure 15. Setting execute permissions

Run the vmexe file with the –f option to force deploy the new firmware.

Figure 16. Executing the firmware package

Technical white paper Page 15

Downgrade the driver

Verify the current driver version by issuing the following command:

esxcli software vib get –n <driver name>

Figure 17. Verifying the current driver version

Remove the current driver.

esxcli software vib remove –n <driver name> -f

Figure 18. Removing the current driver

Install the new downgraded driver.

esxcli software vib install –v <driver location and filename>

Figure 19. Installing the new downgraded driver

Technical white paper Page 16

Reboot the host.

esxcli system shutdown reboot –r ‘driver and firmware downgrade’

Validate the new downgraded driver version.

esxcli software vib get –n <driver name>

Figure 20. Validate the new downgraded driver version

Exit maintenance mode from the CLI or from the vSphere Client.

esxcli system maintenanceMode set –e false

Wait a few moments and then Retest in vSAN Health.

Figure 21. vSAN Health Check results after firmware and driver changes

Technical white paper Page 17

Figure 22. vSAN Health Check details after firmware and driver changes

Note If the environment will also leverage the VMware vSphere® Update Manager™ content depot at vibsdepot.hpe.com, care needs to be taken to make sure that newer drivers are not automatically downloaded and applied during future host remediation activities.

Summary HPE Composable Infrastructure powered by HPE Synergy enables administrators to accelerate application and service delivery through a single interface that precisely composes infrastructure at near-instant speeds. Engineered with software-defined intelligence at its core and powered by HPE OneView, HPE Synergy enables you to bring up services in minutes with a single line of code.

Through the HPE partnership with VMware, vSAN on HPE Synergy is your software-defined storage platform of today and the future.

Technical white paper

Sign up for updates

© Copyright 2017 Hewlett Packard Enterprise Development LP. The information contained herein is subject to change without notice. The only warranties for Hewlett Packard Enterprise products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. Hewlett Packard Enterprise shall not be liable for technical or editorial errors or omissions contained herein.

Microsoft and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. VMware, VMware vSphere, VMware vSphere Distributed Resource Scheduler, VMware vSphere Distributed Switch, VMware vSphere ESXi Image Builder CLI, VMware vSphere High Availability, VMware vSphere Update Manager, VMware vSphere vMotion, VMware ESXi, VMware vCenter, and VMware vCenter Server Appliance are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. All other third-party trademark(s) is/are property of their respective owner(s).

a00036264ENW, November 2017

Resources HPE Synergy

HPE OneView

VMware Compatibility Guide

VMware vSAN Compatibility Guide for HPE

VMware® vSAN™ Network Design

Why HPE Synergy is a winning vSAN solution

To help us improve our documents, provide feedback at hpe.com/contact/feedback.

Learn more at hpe.com/info/synergy