Course Notes: Know enterprise storage performance Hands-on Lab

Hewlett-Packard Company Tactical Methods Performance Analysis and Presentation HOL2996 ndash Storage Performance LAB

Prepared by Chris amp Greg Tinker

christinkerhpcom

gregtinkerhpcom

Document Revision 10

Date Prepared 042013

HOL2996 HOL2996 ndash Storage Performance LAB

LABS Solution Performance -- Tactical Methods Analysis and Presentation

HP GSSE TEAM HP RestrictedDiscover 2013 Page 2 of 19

TinkerTemplate (Version 10April-2013) copy Copyright 2013 Hewlett-Packard Development Company LP

Storage HOL2996_Tinker_Know_enterprise_storage_performa

nce_LABdocx Last changed 08 July 2013 at 1240

Document Revision History

Ver No Ver Date Prepared By Reviewed By

Approved By

Affected Section amp Summary of Change

10 04012013 Chris amp Greg Initial Version

Contents

1 Real World ndash Grab performance from ESXTop 3

11 Fault Isolation to mitigate business impact 3

111 MOD7-Sto-311ndash ESXTop 3

2 ISCSI Performance 8

21 Attach scsi device from an HP VSA (P4000) 8

211 MOD5-IO-311ndash ISCSI IO Profiling 8

3 Real World ndash esxplot 16


311 MOD7-Sto-511ndash MAC 16

4 Real World ndash Collectl Linux Graphing with COLPLOT 18









1 Real World ndash Grab performance from ESXTop

Determine if the Storage Subsystem is the root cause for business impact

11 Fault Isolation to mitigate business impact

Isolate the performance issue on VSA Using Iscsi to ESXi 5X server

111 MOD7-Sto-311ndash ESXTop

Test Case ID Acceptance Criteria

MOD7-Sto-311 Pass Fail Partial NA

Lab Time

60 minutes

Description

Configuration ESXi with guest VSA using iscsi over GigEther with performance issue

PrerequisitesPre-conditions

1 ESXi 2 VSA W ISCSI 3 Windows Perfmon

Procedures

1 Get ESX IP address from instructor 2 Login via putty (using ssh) ndash user = (root) password = (classroom)

Collect ESXTOP data ndash Make sure Instructor loads the systemhellip Then collect data You can find location of space using (df) df Filesystem Bytes Used Available Use Mounted on

VMFS-5 494994980864 277853765632 217141215232 56 vmfsvolumesdatastore1

(esxtop -a -b -d 2 -n 240 gt vmfsvolumesdatastore1Greg_Tinker_esxtop_datacsv) Change NAME to your name Note -a means all -b means batch mode -d is delay between collections and ndashn means number of iterations

Once data is collected ndash esxtop will stop then move the file to your local machine using VSphere storage explorer Make sure to know the location of the esxtop data collection To collect the data ndash you can use SCP or VSPHEREhellip







Example 1 ndash SCP Using Linux machine (scp root[VSA]locationdata_file_namecsv ) Using WinSCP ndash download from herehellip

Drag and drop it to a local directory

Next step you can use VSphere Step 1Using ESX Vpshere ndash copy the esxtop data to your local machine







Step 2

3 Open esxtop data via windows Perfmon step 1 Start with opening perfmon







Change the source of the data to the new ESXTop DATAhellip

4 Start picking your counters and review your performance CPU Network etchellip

Expected Results

1 Review esxtop data know how to collect and manipulate using perfmonhellip







Comments

When deciding on a tool ndash list prorsquos and conrsquos is it the best choice if so why

Notes Student area







2 ISCSI Performance

Discover a SCSI Device via ISCSI

21 Attach scsi device from an HP VSA (P4000)

Issue sequential readwrite command and capture performance data isolate where the data is traversing

211 MOD5-IO-311ndash ISCSI IO Profiling


MOD5-IO-311 Pass Fail Partial NA

Lab Time

60 minutes

Description

Attach to a SCSI disk via a TCP Network and issue work to the device Capture performance metrics using tools previously provided Document all attributes of the IO as well as path being traversed


1 Virtual-Box ndash Linux Guest 2 Windows 7 or gt HP PCOE Laptop 3 Gbit network 4 VSA virtual client installed and configured ndash will be done by instructor

Procedures

1 On Ubuntu install iscsi (apt-get install open-iscsi) NOTE must be ROOT 2 Provide ISCSI InitiatorName (cat etciscsiinitiatornameiscsi) to the instructor so

that it can be added to the VSA instance so that your system can gain access to the LUN that will be provided Example (InitiatorName=iqn1993-08orgdebian0122977b948ac)

3 (ps ndashef | grep iscsi) look for iscsid Example

root 854 1 0 0834 000000 sbiniscsid

if not running type iscsid otherwise proceed

4 Issue the iscsi (iscsiadm -m discovery -t sendtargets -p 1021655) to discover the VSA WARNING IP to be provided by instructor may change by LAB time

5 Bind the SCSI device to the OS using the iscsi driver The following command requires the ndashi (name) which is found by step 2 The ndasht (target name) MUST be provided by the VSA admin (instructor) see picture







To bind a LUN

EXAMPLE command

iscsistart -i iqn1993-08orgdebian0122977b948ac -t iqn2003-

10comlefthandnetworksvsa-mgt-group16vsa-volume -g Tgpt -a 1021655 -d

8)

The above command broken down iqn1993-08orgdebian0122977b948ac ~ contained within

etciscsiinitiatornameiscsi ACTION ITEM 3

iqn2003-10comlefthandnetworksvsa-mgt-group16vsa-volume = provided by

ACTION ITEM 4

1021655 = provided by instructor

NOTE -d debug_level

6 Issue (dmesg) look for the scsi disk to bind to the SD layer [ 821782787] scsi3 iSCSI Initiator over TCPIP

[ 822049721] scsi 3000 Direct-Access LEFTHAND iSCSIDisk a000 PQ 0 ANSI 5

[ 822050063] sd 3000 Attached scsi generic sg2 type 0

[ 822059119] sd 3000 [sdb] 41943040 512-byte logical blocks (214 GB200 GiB)

[ 822065822] sd 3000 [sdb] Write Protect is off

[ 822065827] sd 3000 [sdb] Mode Sense 77 00 00 08

[ 822068300] sd 3000 [sdb] Write cache disabled read cache disabled doesnt support DPO or FUA

[ 822085803] sdb unknown partition table

[ 822096982] sd 3000 [sdb] Attached SCSI disk







7 Scan the disk to see if there is a partition tablehellip rootgreg-VirtualBox~ fdisk -l

Disk devsdb 215 GB 21474836480 bytes

64 heads 32 sectorstrack 20480 cylinders total 41943040 sectors

Units = sectors of 1 512 = 512 bytes

Sector size (logicalphysical) 512 bytes 512 bytes

IO size (minimumoptimal) 512 bytes 512 bytes

Disk identifier 0x00000000

Disk devsdb doesnt contain a valid partition table

8 Note the time on the system by issuing the following command (date) this is important as you will need it much later in this lab

9 Make sure colelctl is running (ps ndashef | grep collectl) if not start collectl (etcinitdcollectl start) if collectl is not installed you skipped a previous lab

10 Place load on the disk ndash Using the buffered devicehellip Watch the performance rsquos while doing dd command MUST capture performance metrics and isolate the path that the traffic takes Please note the merges KBsec Request size wait service time and other stats

while true

do

dd if=devsdb of=devnull bs=1024k

sleep 1

done

Allow this to run for 5 minutes then ctrl^C to stop the dd

In another terminal window you can

Measure the performance iostat ndashxd 1 100 (run as root)

Please take note that the (sdb) may be different device on your system ndash make sure you are using the SCSI disk from the iscsi device driver

11 Bind the SDB scsi device to the raw character device file for raw (direct) access modprobe raw This installs the Raw device driver ndash a prerequisite to the

following command if the driver is not already loaded See modprobeconf raw ndashqa Make sure you do not step on existing raw binding in the next

step Bind the raw device using this examplehellip Notice we use raw1 you can use any any

unused raw raw devrawraw1 devsdb

Please note that sdb may be different on your box Please take note that the raw1 can be any you choose ndash just use raw command to

keep track of it List the bindings with raw ndashqa

12 Once again note the time on the system (date) this is important as you will need it much later in this lab







13 Issue--

while true

do

dd if=devrawraw1 of=devnull bs=64k

sleep 1

done

NOTE raw1 is because we bound in step 11 yours may be different

14 The following tools are used to quickly view performance

At the same time issue dstat ndash what is the performance of the READ()

At the same time issue iostat ndashxd 1 100

Pay attention to IO Merges rrqms (iostat) explain what this means Look at Service time ndash note the value

Answer the question ndashis this a Block level or character (raw) access

The following Graphing section is optional ndash please proceed past it if you do not want to graph Otherwise proceed to step 20

15 (optional) Now we must pretend that this is a REAL live customer and that you were not present when the customer ran the benchmarkhellip The customer did you a favor by documenting what they did and exact time when they did ithellip Now all you need is the historical performance data ndash to that end collect the historical data that collectl collected

etcinitdcollectl stop cp varlogcollectlrawgz mediasf_c_driveworking

Virtual box shared folder of C drive

Once done start 11ollect back up etcinitdcollectl start

16 (optional) Parse the collectl data into CSV format and import into excel to build pivot tables and customer viewable graphs Use the following loop to parse your collectlrawgz file Make sure that the collectl file is the only thing in the directory and that you are on a linux machine in the directory where the file exist

cd mediasf_c_driveworking

run format program ndash format_collectlzip

i format_collectlsh rawfilenamerawgz gt

parseDetailcsv

Or the following loop and cleanup the resulting csv files for I in $(ls rawgz)

do

collectl -p $I -oD -sD --hr 0 2gt devnull | sed s +g gt $Iparse_sDetailcsv amp

collectl -p $I -oD -sd --hr 0 2gt devnull | sed s +g gt $Iparse_sdsummarycsv amp

done

-p [filename for playback ]

-oD = format of output TIME in DATETIME output

-sDd = DISK d = Summary or D = Detail

--hr 0 = Header just one time

2gt send stderr to bit bucket







17 Steps to import into excel a Open Excel 2010 64 bit b Choose the ldquodatardquo Tab then chose ldquoFrom Textrdquo

c After choosing from txt locate the csv file from above (import one at a time) d Choose Delimited e Check ldquoCommardquo all others unchecked f Change the first column to MDY g Second Column remains General h Third column must be TXT i All remain can be general j Press ldquoFinishrdquo

k All remain columns can remain as general l Delete all rows down to when the actual data starts

Row should look like this

18 Now time to create pivot table and graph data 19 Chose the insert tab and create new Pivot table making sure to select all DATA

Choose your Columnrsquos row labels and Values correctlyhellip Discussed in class ndash here







is a screen of what yours should look like

Choose options and Pivot chart to graph over time

Then you can see the problem stick out real easy







20 Modify TCP Parms Note for this lab we must go down in size due to hardware speed limits (only using Gig Ether devices)

Note the defaults are already documented in the file the following are smaller

vi etciscsiiscsidconf

nodesessioniscsiFirstBurstLength = 4096

nodesessioniscsiMaxBurstLength = 8192

nodeconn[0]iscsiMaxRecvDataSegmentLength = 4096

discoverysendtargetsiscsiMaxRecvDataSegmentLength = 1024

Save your work and exist the file

Note this will slow it down ndash defaults are

21 reboot the guest 22 Repeat performance testing using steps 10 - 14 23 Answer the following question ndash

Does the disk access driver play a role on blocking size What is the IO size reported What is the service time What is the IOPS (reads vs writes) Are there merges (buffered IO) Graph this data over time using Excel pivot tables

24 Manipulate the system parameters to get more performance list all system parameters (sysctl ndasha) Make a note of all of these (record to a file) Modify the TCP send and receive buffers sysctl netipv4tcp_mem=4096 87380 16777216

sysctl netipv4tcp_rmem=4096 87380 16777216

sysctl netipv4tcp_wmem=4096 87380 16777216

Just note there is the old way of modifying the same TCP parameters using proc echo 4096 87380 16777216 gt procsysnetipv4tcp_rmem echo 4096 65536 16777216 gt procsysnetipv4tcp_wmem

Not Required for RHEL5 as Default is finehellip echo 1 gt procsysnetipv4tcp_timestamps

Default is 1 in RHEL5 No need to change echo 1 gt procsysnetipv4tcp_sack

Default is 1 in RHEL5 no need to change Other parameters that will change the behavior echo 1048576 gt procsysnetcorermem_default

Default ldquo114688rdquo should be fine however you can set it to 1048576 which will allow setsockopt (SO_RCVBUF) up to 1 MB

echo 262142 gt procsysnetcorermem_max







all sockets allowed to go up to 128k (131071 default) you might need to take this up by factor of 2hellip Rare

echo 1048576gt procsysnetcorewmem_default default same as rmem_default 114688 ndash please confirm change this to 1MB but this too is RAREhellip

echo 16777216 gt procsysnetcorewmem_max This will allow setsockopt (SO_SNDBUF) up to 16 MB 128K default (131071)

echo 50000 gt procsysnetcorenetdev_max_backlog ifconfig ltethXgt txqueuelen 10000 You can try Higher txqueuelen but I do not expect to see huge benefit (if any) Note in case you need to Change MTU to jumbo frames

ifconfig ltethXgt mtu 8100 I think you are using larger but maybe downstream hardware is having

Expected Results 1 Load iscsi drivercode function 2 Bind to SCSI device via ISCS transport 3 Determine if block and character access are different when using ISCSI vs SCSI direct

access

Comments

We usually run the following loop to look at a VAST amount of data from collectl at any point in time

for I in `ls rawgz` do

collectl -p $I -oD -sC --hr 0 2gt devnull | sed s +g gt $Iparse_sC_windowscsv amp collectl -p $I -oD -sc --hr 0 2gt devnull | sed s +g gt $Iparse_sccsv amp collectl -p $I -oD -sD --hr 0 2gt devnull | sed s +g gt $Iparse_sD_windowscsv amp collectl -p $I -oD -sd --hr 0 2gt devnull | sed s +g gt $Iparse_sdcsv amp collectl -p $I -oD -sf --hr 0 2gt devnull | sed s +g gt $Iparse_sfcsv amp collectl -p $I -oD -sj --hr 0 2gt devnull | sed s +g gt $Iparse_sjcsv amp collectl -p $I -oD -sm --hr 0 2gt devnull | sed s +g gt $Iparse_smcsv amp collectl -p $I -oD -sN --hr 0 2gt devnull | sed s +g gt $Iparse_sN_windowscsv amp collectl -p $I -oD -sn --hr 0 2gt devnull | sed s +g gt $Iparse_sncsv amp

done

Student Notes







3 Real World ndash esxtop Using esxtop pull out driver disk latency for disk access for a 24 hour period of time


Isolate the performance issue to any disks

311 MOD7-Sto-511ndash MAC



Lab Time

60 minutes

Description

Customer has a problem with performance on their ESXi Graph the performance of a Linux machinecluster using esxtop from a 24 hour period focus on driver disk latency


1 Linux (with virtualization) 2 Esxtop

Procedures

1 Esxtop data will be provided ndash real customer data for a 24 hour period of time 2 Open a file (on a Linux machine) and parse the files provided by the instructor

code (vi parsersh) Copy below this line binbash cat $1 | sed sg|awk BEGIN FS= NF==1 for(i=1ilt=NFi++) if($i~LDev 06 || $i~LDev 07) f[n++]=i else if($i~PDH-CSV) f[n++]=i else if ($i~Physical Disk) else if ($i~Physical Disk|| $i~Physical Cpu ||$i~Network)

The above line will allow for multiple counters to be parsed f[n++]=i for(i=0iltni++) printf ssi$f[i] printfn | sed s$

Copy above this line







3 Run this new shell program (parsersh ESXTOP_datacsv_file gt my_new_Datacsv) You can also do this in a LOOP so that all files are parsed at one timehellip (for i in $(ls esxtop) do parsersh $i done gt PERFcsv)

4 Do the same for the other files adding the ldquoappendrdquo to the previous file already created (parsersh ESXTOP_DATAcsv_File gtgt my_new_Datacsv)

5 Now follow LAB 3 to import data into Windows Perfmon

Expected Results Narrow the focus of large ESX FARM esxtop data collection to disk stats

Comments


Notes Student area







4 Real World ndash Collectl Linux Graphing with COLPLOT Determine if the Storage Subsystem is the root cause for business impact Customer wants to know why the HOST cannot achieve more throughput


Isolate the performance issue to any disks which are suturing the bus




Lab Time

60 minutes

Description

Customer has a problem with performance on their machine Graph the performance of a Linux machinecluster


3 Linux (with virtualization) 4 Collectl 5 Bridged network 6 colplot

a Install colplot httpcollectl-utilssourceforgenet on your local LINUX machine

Procedures

1 Grab varlogcollectlcolrawgz a Found in

2 Place file on local Linux machine 3 Using collectl create the raw files for plotting

a collectl ndashp varlogcollectlDL380G5P45U10-20130422-144423rawgz -P -f usrsharecollectlplotfiles -oc

4 open web interface 5 http[linux_hostname]colplot







6 Select item to graph NOTE the FROM or LAST fields a You must know when the file was collected to use the FROM field ndash NOTE the

filename contains 20130422-144423ndash hence you know the timeframe However for us we will use the last 4 hours AKA 240 minutes

Expected Results section Expected Results

Comments


Notes Student area







Document Revision History

Ver No Ver Date Prepared By Reviewed By

Approved By

Affected Section amp Summary of Change

10 04012013 Chris amp Greg Initial Version

Contents

1 Real World ndash Grab performance from ESXTop 3


111 MOD7-Sto-311ndash ESXTop 3

2 ISCSI Performance 8

21 Attach scsi device from an HP VSA (P4000) 8

211 MOD5-IO-311ndash ISCSI IO Profiling 8

3 Real World ndash esxplot 16



4 Real World ndash Collectl Linux Graphing with COLPLOT 18
















Lab Time

60 minutes

Description




Procedures





















Step 2










Expected Results








Comments


Notes Student area







2 ISCSI Performance







Lab Time

60 minutes

Description




Procedures














To bind a LUN

EXAMPLE command



8)




ACTION ITEM 4


NOTE -d debug_level






[ 822065827] sd 3000 [sdb] Mode Sense 77 00 00 08





















while true

do


sleep 1

done


















13 Issue--

while true

do


sleep 1

done
















parseDetailcsv


do



done

































































access

Comments




done

Student Notes













Lab Time

60 minutes

Description




Procedures















Comments


Notes Student area













Lab Time

60 minutes

Description





Procedures














Comments


Notes Student area














Lab Time

60 minutes

Description




Procedures





















Step 2










Expected Results








Comments


Notes Student area







2 ISCSI Performance







Lab Time

60 minutes

Description




Procedures














To bind a LUN

EXAMPLE command



8)




ACTION ITEM 4


NOTE -d debug_level






[ 822065827] sd 3000 [sdb] Mode Sense 77 00 00 08





















while true

do


sleep 1

done


















13 Issue--

while true

do


sleep 1

done
















parseDetailcsv


do



done

































































access

Comments




done

Student Notes













Lab Time

60 minutes

Description




Procedures















Comments


Notes Student area













Lab Time

60 minutes

Description





Procedures














Comments


Notes Student area
















Step 2










Expected Results








Comments


Notes Student area







2 ISCSI Performance







Lab Time

60 minutes

Description




Procedures














To bind a LUN

EXAMPLE command



8)




ACTION ITEM 4


NOTE -d debug_level






[ 822065827] sd 3000 [sdb] Mode Sense 77 00 00 08





















while true

do


sleep 1

done


















13 Issue--

while true

do


sleep 1

done
















parseDetailcsv


do



done

































































access

Comments




done

Student Notes













Lab Time

60 minutes

Description




Procedures















Comments


Notes Student area













Lab Time

60 minutes

Description





Procedures














Comments


Notes Student area







Step 2










Expected Results








Comments


Notes Student area







2 ISCSI Performance







Lab Time

60 minutes

Description




Procedures














To bind a LUN

EXAMPLE command



8)




ACTION ITEM 4


NOTE -d debug_level






[ 822065827] sd 3000 [sdb] Mode Sense 77 00 00 08





















while true

do


sleep 1

done


















13 Issue--

while true

do


sleep 1

done
















parseDetailcsv


do



done

































































access

Comments




done

Student Notes













Lab Time

60 minutes

Description




Procedures















Comments


Notes Student area













Lab Time

60 minutes

Description





Procedures














Comments


Notes Student area









Expected Results








Comments


Notes Student area







2 ISCSI Performance







Lab Time

60 minutes

Description




Procedures














To bind a LUN

EXAMPLE command



8)




ACTION ITEM 4


NOTE -d debug_level






[ 822065827] sd 3000 [sdb] Mode Sense 77 00 00 08





















while true

do


sleep 1

done


















13 Issue--

while true

do


sleep 1

done
















parseDetailcsv


do



done

































































access

Comments




done

Student Notes













Lab Time

60 minutes

Description




Procedures















Comments


Notes Student area













Lab Time

60 minutes

Description





Procedures














Comments


Notes Student area







Comments


Notes Student area







2 ISCSI Performance







Lab Time

60 minutes

Description




Procedures














To bind a LUN

EXAMPLE command



8)




ACTION ITEM 4


NOTE -d debug_level






[ 822065827] sd 3000 [sdb] Mode Sense 77 00 00 08





















while true

do


sleep 1

done


















13 Issue--

while true

do


sleep 1

done
















parseDetailcsv


do



done

































































access

Comments




done

Student Notes













Lab Time

60 minutes

Description




Procedures















Comments


Notes Student area













Lab Time

60 minutes

Description





Procedures














Comments


Notes Student area







2 ISCSI Performance







Lab Time

60 minutes

Description




Procedures














To bind a LUN

EXAMPLE command



8)




ACTION ITEM 4


NOTE -d debug_level






[ 822065827] sd 3000 [sdb] Mode Sense 77 00 00 08





















while true

do


sleep 1

done


















13 Issue--

while true

do


sleep 1

done
















parseDetailcsv


do



done

































































access

Comments




done

Student Notes













Lab Time

60 minutes

Description




Procedures















Comments


Notes Student area













Lab Time

60 minutes

Description





Procedures














Comments


Notes Student area







To bind a LUN

EXAMPLE command



8)




ACTION ITEM 4


NOTE -d debug_level






[ 822065827] sd 3000 [sdb] Mode Sense 77 00 00 08





















while true

do


sleep 1

done


















13 Issue--

while true

do


sleep 1

done
















parseDetailcsv


do



done

































































access

Comments




done

Student Notes













Lab Time

60 minutes

Description




Procedures















Comments


Notes Student area













Lab Time

60 minutes

Description





Procedures














Comments


Notes Student area


















while true

do


sleep 1

done


















13 Issue--

while true

do


sleep 1

done
















parseDetailcsv


do



done

































































access

Comments




done

Student Notes













Lab Time

60 minutes

Description




Procedures















Comments


Notes Student area













Lab Time

60 minutes

Description





Procedures














Comments


Notes Student area







13 Issue--

while true

do


sleep 1

done
















parseDetailcsv


do



done

































































access

Comments




done

Student Notes













Lab Time

60 minutes

Description




Procedures















Comments


Notes Student area













Lab Time

60 minutes

Description





Procedures














Comments


Notes Student area




























































access

Comments




done

Student Notes













Lab Time

60 minutes

Description




Procedures















Comments


Notes Student area













Lab Time

60 minutes

Description





Procedures














Comments


Notes Student area
















































access

Comments




done

Student Notes













Lab Time

60 minutes

Description




Procedures















Comments


Notes Student area













Lab Time

60 minutes

Description





Procedures














Comments


Notes Student area







































access

Comments




done

Student Notes













Lab Time

60 minutes

Description




Procedures















Comments


Notes Student area













Lab Time

60 minutes

Description





Procedures














Comments


Notes Student area













access

Comments




done

Student Notes













Lab Time

60 minutes

Description




Procedures















Comments


Notes Student area













Lab Time

60 minutes

Description





Procedures














Comments


Notes Student area













Lab Time

60 minutes

Description




Procedures















Comments


Notes Student area













Lab Time

60 minutes

Description





Procedures














Comments


Notes Student area











Comments


Notes Student area













Lab Time

60 minutes

Description





Procedures














Comments


Notes Student area













Lab Time

60 minutes

Description





Procedures














Comments


Notes Student area










Comments


Notes Student area

Technology

Course Notes: Know enterprise storage performance Hands-on Lab