Upload
kamathamey
View
107
Download
0
Embed Size (px)
DESCRIPTION
EMC Data Collection Guide
Citation preview
EMC Global Services Technology Implementation Services
EMC Modeling Tools Data Collection Guide v1.17
Date: November 24, 2013
Abstract
This document defines the data collection methodology for the EMC modeling tools.
The target audience is composed of EMC GS/TS Technology Consultants and TS Solutions Architects.
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P2/68
Technology Implementation Services
Copyright 2007, 2013 EMC Corporation. All rights reserved.
EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS IS." EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.
EMC2, EMC, EMC ControlCenter, AlphaStor, ApplicationXtender, Captiva, Catalog Solution, Celerra, CentraStar, CLARalert, CLARiiON, ClientPak, Connectrix, Co-StandbyServer, Dantz, Direct Matrix Architecture, DiskXtender, DiskXtender 2000, Documentum, EmailXaminer, EmailXtender, EmailXtract, eRoom, FLARE, HighRoad, Input Accel, Navisphere, NetWorker, nLayers, OpenScale, Powerlink, PowerPath, Rainfinity, RepliStor, ResourcePak, Retrospect, Smarts, SnapShotServer, SnapView/IP, SRDF, Symmetrix, TimeFinder, VisualSAN, VSAM-Assist, WebXtender, where information lives, Xtender, and Xtender Solutions are registered trademarks and EMC Developers Program, EMC OnCourse, EMC Proven, EMC Snap, EMC Storage Administrator, Acartus, Access Logix, ArchiveXtender, Authentic Problems, Automated Resource Manager, AutoStart, AutoSwap, AVALONidm, C-Clip, Celerra Replicator, Centera, CLARevent, Codebook Correlation Technology, Common Information Model, CopyCross, CopyPoint, DatabaseXtender, Direct Matrix, EDM, E-Lab, Enginuity, FarPoint, Global File Virtualization, Graphic Visualization, InfoMover, Infoscape, Invista, Max Retriever, MediaStor, MirrorView, NetWin, OnAlert, PowerSnap, RepliCare, SafeLine, SAN Advisor, SAN Copy, SAN Manager, SDMS, SnapImage, SnapSure, SnapView, StorageScope, SupportMate, SymmAPI, SymmEnabler, Symmetrix DMX, UltraPoint, UltraScale, Viewlets, and VisualSRM are trademarks of EMC Corporation.
For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com
All other trademarks used herein are the property of their respective owners.
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P3/68
Technology Implementation Services
TABLE OF CONTENTS
Document Control ................................................................................................................................................ 7
Contributors ...................................................................................................................................................................... 7
Build Status ...................................................................................................................................................................... 7
1 Introduction to Data Collection ..................................................................................................................... 9
1.1 Recommended 3 Day Minimum at 10 Minute Intervals ............................................................................................ 9
1.2 Minimum Data Collection Interval is 1 Minute......................................................................................................... 9
1.3 Data Collection Interval Must Be the Same............................................................................................................. 9
1.4 Standardized Statistics ......................................................................................................................................... 9
1.4.1 Required Fields ........................................................................................................................................ 9
1.4.2 Optional Fields ........................................................................................................................................ 9
1.5 Supported Array and Host Statistics ..................................................................................................................... 10
1.6 General Assumptions ........................................................................................................................................... 10
1.7 Sample Data ........................................................................................................................................................ 11
1.7.1 EMC Internal Users .................................................................................................................................. 11
1.7.2 External Users......................................................................................................................................... 11
2 AIX iostat .................................................................................................................................................... 12
2.1 Data Collection Considerations ............................................................................................................................ 12
2.2 Data Collection Procedure .................................................................................................................................... 12
2.3 Command Syntax ................................................................................................................................................. 12
2.4 Record Layout ...................................................................................................................................................... 13
2.5 Sample Output .................................................................................................................................................... 13
3 BCSD Export ................................................................................................................................................ 14
3.1 Data Collection Procedure .................................................................................................................................... 14
3.2 Record Layout ...................................................................................................................................................... 14
3.3 Sample Output .................................................................................................................................................... 14
4 BTP ............................................................................................................................................................. 15
4.1 Collection Considerations .................................................................................................................................... 15
4.1.1 RAID-10 ................................................................................................................................................. 15
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P4/68
Technology Implementation Services
4.1.2 Time Zone .............................................................................................................................................. 15
4.2 Data Collection Procedure .................................................................................................................................... 15
4.2.1 Performance Manager ............................................................................................................................ 15
4.2.2 STP ........................................................................................................................................................ 16
5 HP-UX sar .................................................................................................................................................... 17
5.1 Data Collection Considerations ............................................................................................................................ 17
5.2 Data Collection Procedure .................................................................................................................................... 17
5.2.1 Command Syntax ................................................................................................................................... 18
5.3 Record Layout ...................................................................................................................................................... 19
5.4 Sample Output ................................................................................................................................................... 20
6 iSeries (System i, IBM i) WRKDSKSTS .......................................................................................................... 21
6.1 Data Collection Considerations ............................................................................................................................ 21
6.2 Data Collection Procedure .................................................................................................................................... 21
6.3 Command Syntax ................................................................................................................................................. 21
6.4 Record Layout ..................................................................................................................................................... 22
6.5 Sample Output ................................................................................................................................................... 24
7 Linux iostat ................................................................................................................................................ 25
7.1 Data Collection Considerations ............................................................................................................................ 25
7.2 Data Collection Procedure .................................................................................................................................... 25
7.3 Command Syntax ................................................................................................................................................. 25
7.4 Record Layout ..................................................................................................................................................... 26
7.5 Sample Output ................................................................................................................................................... 28
8 Mainframe (SMF/CMF) ................................................................................................................................ 29
8.1 Data Collection Considerations ........................................................................................................................... 29
8.2 Mainframe Assembler Kit .................................................................................................................................... 30
8.2.1 Additional notes .................................................................................................................................... 30
8.3 Onsite Processing ................................................................................................................................................ 31
8.4 Offsite Processing ................................................................................................................................................ 31
8.5 Record Layout ...................................................................................................................................................... 33
8.6 Sample Output .................................................................................................................................................... 35
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P5/68
Technology Implementation Services
8.7 Accessing EMCs E01 LPAR .................................................................................................................................. 36
9 MirrorView/A LOR (locality of reference)..................................................................................................... 39
9.1 Data Collection Considerations ........................................................................................................................... 39
9.2 Acquiring the MirrorView/A LOR Tool ................................................................................................................... 39
9.3 Data Collection Procedure ................................................................................................................................... 39
9.4 Command Syntax ................................................................................................................................................ 39
9.5 Sample Output ................................................................................................................................................... 40
10 Navisphere Analyzer (NAR) .......................................................................................................................... 41
10.1 Data Collection Considerations for metaLUNs and Thin LUNs ................................................................................. 41
10.1.1 RAW files ............................................................................................................................................... 41
10.1.2 CSV files ................................................................................................................................................ 41
10.2 Data Collection Procedure .................................................................................................................................... 41
10.2.2 Sample batch file to convert all NAR files in a directory to CSV files ......................................................... 42
10.3 Command Syntax ................................................................................................................................................ 42
10.4 Record Layout ......................................................................................................................................................44
10.5 Sample Output .................................................................................................................................................... 45
11 Solaris iostat ............................................................................................................................................. 46
11.1 Data Collection Considerations ........................................................................................................................... 46
11.2 Data Collection Procedure ................................................................................................................................... 46
11.3 Command Syntax ................................................................................................................................................ 46
11.4 Record Layout ...................................................................................................................................................... 47
11.5 Sample Output .................................................................................................................................................... 47
12 VMware esxtop/resxtop ............................................................................................................................. 48
12.1 Data Collection Considerations ........................................................................................................................... 48
12.2 Data Collection Procedure ................................................................................................................................... 48
12.3 Command Syntax ................................................................................................................................................ 49
12.4 Record Layout ..................................................................................................................................................... 49
12.5 Sample Output ....................................................................................................................................................50
13 VPLEX Stats ................................................................................................................................................. 51
13.1 Data Collection Considerations ............................................................................................................................ 51
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P6/68
Technology Implementation Services
13.2 Data Collection Procedure .................................................................................................................................... 51
13.3 Command Syntax ................................................................................................................................................. 51
13.4 Record Layout ...................................................................................................................................................... 52
13.5 Sample Output .................................................................................................................................................... 53
14 Windows Perfmon ...................................................................................................................................... 54
14.1 Data Collection Considerations ............................................................................................................................ 54
14.2 Data Collection Procedure .................................................................................................................................... 54
14.3 Conversion from BLG to CSV ................................................................................................................................ 62
14.4 Record Layout ..................................................................................................................................................... 62
14.5 Sample Filtered Output ....................................................................................................................................... 64
Appendix A TroubleShooting and Support ...................................................................................................... 65
Appendix B Feature Enhancements and Feedback ........................................................................................... 66
B.1 Feature Enhancements .............................................................................................................................................. 66
B.2 Feedback .................................................................................................................................................................. 68
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P7/68
Technology Implementation Services
DOCUMENT CONTROL
Contributors
Role Name
Author(s) Mike Lawrence
Build Status
Version Date of issue Author Reason
1.10 10-Mar-2011 Mike Lawrence 1. Rewrite data collection procedure for VMware esxtop.
2. FLARE 30 requires a change to the collection procedure for NAR CSV.
1.11 13-Dec-2011 Mike Lawrence 1. Clarify how to use metaLUNs when using NAR files and include/exclude lists.
2. Removed NaviCLI commands for pre-FLARE 24.
3. Add VPLEX stats support 4. As the SVC no longer exists, removed
the section on the SVC.
1.12 30-May-2012 Mike Lawrence Correct some typos.
1.13 25-Nov-2012 Mike Lawrence 1. Change NAR section to include raw. 2. Change - archivedump to-
archivedump (removed the space). 3. For all data imported into the same
application, the data collection interval must be the same.
4. Change WRKDSKSTS total read and write IOs to IOPS.
1.14 29-Jan-2013 Mike Lawrence Dropped CLARiiON/Unified NAR CSV and Simulated (Workload Generator) performance data.
1.15 29-May-2013 Mike Lawrence 1. Updated Windows Perfmon data collection for Windows 2008 and above.
2. Added Linux iostat considerations for partitioned devices and emcpower devices.
3. Removed references to the now retired ET tool.
4. Added T option as default parameter for AIX iostat as newer versions of the OS support it.
5. Added the collection consideration, the read/write ratio is more of a "best guess" based on all of the read/write activity for all of the volumes in any given interval. In other words, it is not accurate, for HP-UX sar.
6. Added CLARiiON/Unified NAR CSV as it is temporarily back in BCSD.
7. Added to HP-UX sar Data Collection Considerations.
8. Added to Minimum Data Collection
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P8/68
Technology Implementation Services
Interval is 1 Minute.
1.16 03-Jul-2013 Mike Lawrence 1. Correct Windows Perfmon relog syntax. 2. Added consideration for AIX iostat
partitioned records.
1.17 14-Nov-2013 Mike Lawrence Removed comments about iSeries RAID-1 volumes as the parser now correctly handles them.
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P9/68
Technology Implementation Services
1 INTRODUCTION TO DATA COLLECTION EMCs remote replication modeling tools require host or array based statistical data to determine data change activity. This enables modeling based on real world data rather than a best guess based on highly averaged data.
1.1 Recommended 3 Day Minimum at 10 Minute Intervals
The minimum data capture interval should be 3 days, while a week is quite common as it will include the various daily peaks and valleys. The actual times for the peaks should be the responsibility of the customer as they know when their peak processing occurs. Ideally, the data capture will include the peak workload period such as month-end or year-end processing. By capturing at least 24 hours, a distinct online and batch processing profile is observed. This is important because the average IO size of write I/Os tends to be smaller for online workloads and larger for batch, which has a direct impact on bandwidth requirements.
Data capture should be in 10 minute intervals or less, but no less than 1 minute. If longer intervals are used you may be impacted by the flattening out of the true peaks due to longer averages, thus skewing the results. Even at 10 minute intervals your true peaks could be twice as much as the average peak recorded in the interval. This can cause a significantly under-configured solution. If a collection interval of less than 10 minutes is desired, note the tools require a minimum of 1 minute.
1.2 Minimum Data Collection Interval is 1 Minute
The minimum data collection interval is 1 minute. Using a data collection interval of less than 1 minute will result in unpredictable results such as corrupted data, invalid error messages, and invalid modeling results. This is due to the fact that the time rounding routine rounds to the nearest data collection interval in minutes.
1.3 Data Collection Interval Must Be the Same
For all data imported into the same application, the data collection interval must be the same.
1.4 Standardized Statistics
Performance data is imported into the tool where all statistics are normalized into a single format.
1.4.1 Required Fields
Field Description
Timestamp Date and Time of the performance data (converted to GMT based on user specified offset). The timestamp is rounded to the nearest collection interval.
Read I/Os Total read I/Os
Read I/O Size Average read I/O size (bytes)
Write I/Os Total write I/Os
Sequential WriteIOs Total sequential write I/Os (If not contained in the data, set to zero)
Write IO Size Average write I/O size (bytes)
Volume ID LUN, device name, volume ID or volser
1.4.2 Optional Fields
These fields are used exclusively for BCSDs VPLEX modeling.
Field Description
RRH I/Os Random read hit IO type
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P10/68
Technology Implementation Services
Field Description
RRM I/Os Random read miss IO type
SR I/Os Sequential read IO type
RW I/Os Random write IO type
SW I/Os Sequential write IO type
1.5 Supported Array and Host Statistics
Array Description
Symmetrix EMC Symmetrix
CLARiiON EMC CLARiiON
Statistics Description
AIX iostat IBM AIX iostat command output
Symmetrix BTP EMC Symmetrix data in binary format
HP-UX sar HP-UX sar command output
iSeries WRKDSKSTATS IBM iSeries WRKDSKSTATS
Linux iostat Linux iostat command output
Mainframe Assembler Kit v2 IBM mainframe SMF data processed by EMCs Mainframe Assembler Kit v2 and output in CSV format
MirrorView/A LOR Tool EMCs MirrorView/A LOR tool output in CSV format
CLARiiON/Unified Block NAR CSV CLARiiON and Celerra Navisphere Analyzer data in CSV format
CLARiiON/Unified Block NAR (raw) CLARiiON and Celerra Navisphere Analyzer data in raw format
Solaris iostat Sun Solaris iostat command output
Windows Perfmon CSV Microsoft Windows Perfmon data in CSV format (raw BLG data is not supported)
VMware esxtop VMware esxtop command output (BCSD only)
VPLEX stats VPLEX statistics output (BCSD only)
1.6 General Assumptions
All headings are in English, so non-English headings need to be translated to English.
Missing data such as timestamps and collection intervals will be supplied by the user during importing.
For BCSD, after determining the IO rates and/or counts:
o All records with zero I/O's for reads, writes and sequential writes will be skipped and not imported.
o All reads and writes with more than 250,000 IOPS will be flagged and excluded from modeling.
o Any write IO size larger than 20MB will be flagged and excluded from modeling.
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P11/68
Technology Implementation Services
1.7 Sample Data
1.7.1 EMC Internal Users
Sample data can be found in the EMC GS Tools eRoom in the GS Tools Common Components folder at http://ctseroom02.corp.emc.com/eRoom/tsfuncprac/TSTools/0_47cc6. Internal EMC users who do not have access may request it by emailing [email protected]. but all other users must request copies of the data.
1.7.2 External Users
You may request copies of the sample data by emailing [email protected].
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P12/68
Technology Implementation Services
2 AIX IOSTAT
2.1 Data Collection Considerations
AIX iostat formats may vary from version to version with some versions providing timestamps while others do not. If timestamps are not present then you will need to know when the data was collected and what collection interval was used and enter it on import.
Partitioned records are redundant data and will be ignored.
sd* device driver names (as an example, sddj and then sddj1) may possibly be multiple entries for the same basic host device. In the example previously, sddj will be totally encompassing for all subcomponent partitions (i.e. sddj will show total IOPs for sddj1 and sddj2 and any other partitions). Because of this, you will need an include list to strip out all numbers after the primary device driver name.
The command can be scheduled in cron or other scheduling products for the desired capture period.
2.2 Data Collection Procedure
Follow these steps to collect iostat data:
1. Identify the servers and the specific LUNS (disks) that will be replicated.
2. Run iostat for a short duration and verify the output matches the required format (see Record Layout and Sample Output below).
3. On each host, run iostat from the command line or optionally create a cron entry.
4. Collect the iostat output file(s).
2.3 Command Syntax
The command syntax should be similar to the following and be issued on each host:
iostat d T interval count >> hostname.iostat.date.txt
Options Description
-d Displays only the device utilization report
-T Adds a column containing the time, but no date (AIX 5.2 and above)
interval Collects data every specified seconds
Suggested setting is 600 (10 minutes)
count Collects statistics for specified intervals
Suggested setting is 432 (3 days of 10 minute intervals)
CRON Syntax:
* * * * * iostat d T interval count >> hostname.iostat.date.txt - - - - - | | | | | | | | | + day of week (1 - 7) (Monday = 1) (Sunday = 0 or 7) | | | +------- month (1 - 12) | | +--------- day of month (1 - 31) | +----------- hour (0 - 23) +------------- min (0 - 59)
Sample CRON Entry: 0 0 * * 5-7 iostat d T 600 144 >> zeus.iostat.04142009.txt
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P13/68
Technology Implementation Services
This entry schedules iostat to start Friday at 00:00 and end Sunday at 00:00. The iostat command pulls statistics every 600 seconds (10 minute) for 24 hours creating 144 intervals. The UNIX redirect >> appends the output to the same file.
2.4 Record Layout
Heading Description
Disks: Disk name
% tm_act Percentage of time device was busy servicing a request
Kbps Amount of data transferred (both reads and writes) to the drive in KB per second
tps Total I/Os per second
Kb_read Total amount of data read from the drive in KB
Kb_wrtn Total amount of data written to the drive in KB
time Time the statistics were written (5.2+)
2.5 Sample Output
Without Time
System configuration: lcpu=32 drives=976 paths=2 vdisks=0 Disks: % tm_act Kbps tps Kb_read Kb_wrtn hdisk0 1.1 27.3 2.2 0 16430 hdisk1 1.1 27.3 2.2 0 16430 hdisk263 0.8 30.0 3.5 17632 408 hdisk259 0.9 29.9 3.4 17704 288 hdisk256 0.8 30.8 3.5 17504 1027 hdisk261 0.8 29.6 3.4 17008 752 Disks: % tm_act Kbps tps Kb_read Kb_wrtn hdisk0 1.3 24.2 2.4 8 14532 hdisk1 1.2 24.2 2.4 0 14532 hdisk263 0.5 15.2 1.9 9040 80 hdisk259 0.6 18.1 2.2 10832 48 hdisk256 0.7 17.2 2.1 9912 400 hdisk261 0.7 17.7 2.1 10184 424
With Time
System configuration: lcpu=4 drives=161 paths=233 vdisks=0 Disks: % tm_act Kbps tps Kb_read Kb_wrtn time hdisk0 0.2 4.0 0.5 0 2408 09:50:39 hdisk1 0.0 0.0 0.0 0 0 09:50:39 hdisk5 0.0 0.0 0.0 0 0 09:50:39 hdisk3 0.0 0.0 0.0 0 0 09:50:39 hdisk0 0.2 4.0 0.5 0 2392 10:00:39 hdisk1 0.0 0.0 0.0 0 0 10:00:39 hdisk5 0.0 0.0 0.0 0 0 10:00:39 hdisk3 0.0 0.0 0.0 0 0 10:00:39
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P14/68
Technology Implementation Services
3 BCSD EXPORT
3.1 Data Collection Procedure
Data can be exported from BCSD by going to the replication/consistency/volume group and using the export performance data to CSV feature. This will export only the mapped data for that group.
3.2 Record Layout
Field Description
Performance Data Tag Project level user tag used to identify a group of performance data that will be accessed by one or more BCSD projects
Load Tag Performance data level user tag used to identify specific performance data files
File Name Name of the performance data file for these specific records
Timestamp Date and Time of the loaded data after being normalized to GMT based on the user selected offset and rounded to the nearest collection interval
Data Type Type of performance data
Seconds per Interval Calculated time interval in seconds between two statistic records
Volume ID LUN, device name, volume ID or volser
Read IOs Total read IOs
Read IO Size Average read IO size in bytes
Write IOs Total write I/Os
Sequential Write IOs Total sequential write I/Os (If not contained in the data, set to zero)
Write IO Size Average write IO size in bytes
Response Time Host IO response time in ms (If not contained in the data, set to zero)
Random Read Hit Total random read hit IOPS
Random Read Miss Total random read miss IOPS
Sequential Read Total sequential read IOPS
Random Write Total random write IOPS
Sequential Write Total sequential write IOPS
3.3 Sample Output
Performance Data Tag,Load Tag,File Name,Timestamp,Data Type,Seconds per Interval,Volume ID,Read IOs,Read IO Size,Write IOs,Sequential Write IOs,Write IO Size,Response Time,Random Read Hit,Random Read Miss,Sequential Read,Random Write,Sequential Write, Test,PeakFiles,T1_20120414_180004_000292600925.btp,2012-04-14 18:25:00,BTP,300,027A,124.584716796875,999.4239501953125,2.9900331497192383,0.9966776967048645,5802.66650390625,0.0,0.37541529536247253,0.039867103099823,0.0,0.006644518114626408,0.003322259057313204 Test,PeakFiles,T1_20120414_180004_000292600925.btp,2012-04-14 19:45:00,BTP,300,027A,64.21404266357422,880.0,0.0,0.0,0.0,0.0,0.1872909665107727,0.02675585448741913,0.0,0.0,0.0 Test,PeakFiles,T1_20120414_180004_000292600925.btp,2012-04-14 20:05:00,BTP,300,027A,67.60562896728516,944.0,0.0,0.0,0.0,0.0,0.19718310236930847,0.02816900610923767,0.0,0.0,0.0 Test,PeakFiles,T1_20120414_180004_000292600925.btp,2012-04-14 21:05:00,BTP,300,027A,64.21404266357422,816.0,0.0,0.0,0.0,0.0,0.18394649028778076,0.030100330710411072,0.0,0.0,0.0
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P15/68
Technology Implementation Services
Test,PeakFiles,T1_20120414_180004_000292600925.btp,2012-04-14 21:25:00,BTP,300,027A,64.0,1024.0,0.0,0.0,0.0,0.0,0.19333332777023315,0.02000001072883606,0.0,0.0,0.0 Test,PeakFiles,T1_20120414_180004_000292600925.btp,2012-04-14 21:45:00,BTP,300,027A,64.21404266357422,720.0,0.0,0.0,0.0,0.0,0.20066890120506287,0.013377919793128967,0.0,0.0,0.0 Test,PeakFiles,T1_20120414_180004_000292600925.btp,2012-04-14 18:05:00,BTP,300,027B,0.0,0.0,64.21404266357422,0.0,512.0,0.0,0.0,0.0,0.0,0.21404682099819183,0.0 Test,PeakFiles,T1_20120414_180004_000292600925.btp,2012-04-14 18:10:00,BTP,300,027B,0.0,0.0,128.0,0.0,512.0,0.0,0.0,0.0,0.0,0.4266666769981384,0.0 Test,PeakFiles,T1_20120414_180004_000292600925.btp,2012-04-14 18:15:00,BTP,300,027B,0.0,0.0,64.0,0.0,512.0,0.0,0.0,0.0,0.0,0.2133333384990692,0.0 Test,PeakFiles,T1_20120414_180004_000292600925.btp,2012-04-14 18:20:00,BTP,300,027B,0.0,0.0,128.0,0.0,680.0,0.0,0.0,0.0,0.0,0.4266666769981384,0.0
4 BTP
4.1 Collection Considerations
If your data will include metas, then you will need to specify both the meta-heads and meta-members for any include or exclude list used during modeling.
Access to the Symmetrix BIN file or symapi_db.bin file is needed to accurately identify the meta-members. Use the Symmetrix Meta Member Expansion & Validation tool to expand your list of meta-heads for use as an include or exclude list.
4.1.1 RAID-10
A fix has been implemented in 5671, 5772, 5773 and 5874 for the issue where the RAID-10 volume statistics are written in total to all four meta-members, thus if processed as is would result in a 4x increase in the actual number of IOs. For additional information, see Primus EMC174047.
When importing data containing RAID-10 volumes the EMC remote replication modeling tools scan through all of the imported records (during modeling) and for each RAID-10 group with the same read and write statistics, will divide those read and write statistics by 4. This is similar to the fix provided in SymmMerge.
4.1.2 Time Zone
BCSD does not request a GMT offset and will always reflect GMT with zero offset, unless you specify a different offset at modeling time.
There are several factors that can impact the accuracy of the BTP time-zone information when viewed by any of the EMC tools:
The actual time-zone set on the Symmetrix/DMX service processor. The actual time-zone of the Agent used to collect the BTP data. The actual time-zone of where the data is being viewed.
4.2 Data Collection Procedure
Follow these steps to collect BTP data:
1. Identify the arrays and the specific LUNS (disks) that will be replicated
2. For each array, use Performance Manager or STP to start BTP data collection
3. Collect the BTP or TTP output file(s)
4. For TTP files, convert them to BTP files using STPNavigator
4.2.1 Performance Manager
To create BTP files, follow the same collection process as defined in ECC. As a recommendation, use the Analyst Method as it creates a single BTP file for any interval and duration specified by the user.
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P16/68
Technology Implementation Services
4.2.2 STP
Coordinate STP data collection with your Customer Engineer (CE) as they will need to setup a script to run on the Service Processor.
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P17/68
Technology Implementation Services
5 HP-UX SAR
5.1 Data Collection Considerations
The read/write ratio is a "best guess" based on all of the read/write activity for all of the volumes in any given interval. In other words, it is not accurate. If you can collect and use data from another source, that will be more accurate.
The command can be scheduled in cron or other scheduling products for the desired capture period.
5.2 Data Collection Procedure
Follow these steps to collect sar data:
1. Identify the servers and the specific LUNS (disks) that will be replicated.
2. Run sar for a short duration and verify the output matches the required format (see
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P18/68
Technology Implementation Services
3. Record Layout and Sample Output below).
4. On each host, run sar from the command line or optionally create a cron entry.
5. Collect the sar output file(s).
5.2.1 Command Syntax
The command syntax should be similar to the following and be issued on each host:
sar d b interval count >> hostname.sar.date.txt
-d Displays the device activity
-b Displays buffer I/O statistics
interval Collects data every specified seconds
Suggested setting is 600 (10 minutes)
count Collects statistics for specified intervals
Suggested setting is 432 (3 days of 10 minute intervals)
CRON Syntax:
* * * * * sar d b interval count >> hostname.sar.date.txt - - - - - | | | | | | | | | + day of week (1 - 7) (Monday = 1) (Sunday = 0 or 7) | | | +------- month (1 - 12) | | +--------- day of month (1 - 31) | +----------- hour (0 - 23) +------------- min (0 - 59)
Sample CRON Entry: 0 0 * * 5-7 sar d b 600 144 >> zeus.sar.04142009.txt
This entry schedules sar to start Friday at 00:00 and end Sunday at 00:00. The sar command pulls statistics every 600 seconds (10 minute) for 24 hours creating 144 intervals. The UNIX redirect >> appends the output to the same file.
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P19/68
Technology Implementation Services
5.3 Record Layout
First Header
Field Description
Day of Week Day of week truncated to first three characters
Month Month truncated to first thre characters (MMM)
Day Day (dd)
Time Time (hh:mm:ss)
? Unknown
Year Year (YYYY)
Second Header
Field Description
OS Constant HP-UX
Server Name Name of the server where sar was run
OS Version Version of the OS where sar was run
? Unknown
Hardware Family
Hardware family where sar was run
Date Date (MM/DD/YY)
Statistics
Heading Description
Time (HH:MM:SS). Only on first record in set of statistics.
device Device name
%busy Percentage of time device was busy servicing a request
avgque Average number of I/Os queued
r+w/s Total I/Os per second
blks/s Total blocks/sec in 512 byte blocks
avwait Average I/O queue time (ms)
avserv Average I/O service time (ms). AKA response time.
Buffer statistics
bread/s Total physical reads per second from disk
lread/s Total reads per second from buffer cache
%rcache Buffer cache hit ratio for reads
bwrit/s Total physical writes per second from disk
lwrit/s Total writes per second from buffer cache
%wcache Buffer cache hit ratio for writes
pread/s Total physical reads per second from character device using raw I/O
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P20/68
Technology Implementation Services
pwrit/s Total physical writes per second from character device using raw I/O
5.4 Sample Output
HP-UX pshqu1 B.11.00 U 9000/800 02/15/06 07:51:01 device %busy avque r+w/s blks/s avwait avserv bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s 08:01:01 c1t6d0 16.33 0.52 21 167 4.79 10.12 c2t6d0 9.56 0.52 16 146 4.87 7.56 c8t0d1 0.05 0.50 0 1 4.37 3.87 c8t0d2 7.05 0.50 16 66 3.47 6.59 0 5927 100 14 63 77 169 21 08:11:01 c1t6d0 12.64 0.50 16 134 4.70 10.29 c2t6d0 7.90 0.50 13 115 4.82 7.78 c8t0d1 0.03 0.50 0 0 4.08 4.02 c8t0d2 3.99 0.50 13 59 3.11 4.16
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P21/68
Technology Implementation Services
6 ISERIES (SYSTEM I, IBM I) WRKDSKSTS
6.1 Data Collection Considerations
Overflow fields indicated by +++++ must manually be converted to 999.9 in order to successfully import. There is no way to specify the actual value so 999.9 will be lower than the actual value, but it is the largest value you can specify.
6.2 Data Collection Procedure
Follow these steps to collect WRKDSKSTS data:
1. Identify the servers and the specific disks that will be replicated.
2. FTP the source code to the iSeries (System i, IBM i) that would be the SRDF source host.
3. Compile the CL (Control Language) program.
4. Add the library containing the compiled program to the library list (if not already there)
5. Run the program.
6. If the program is run multiple times, rename or copy the resulting file (QGPL/DSKSFOREMC) before each time it is run.
7. Collect the QGPL/DSKSFOREMC output file(s).
6.3 Command Syntax
The command syntax should be similar to the following and be issued on each host:
SBMJOB CMD(CALL PGM(WRKDSKSTS) PARM('Count' 'Interval')) Options Description
count Collects statistics for specified intervals
Suggested setting is 432 (3 days of 10 minute intervals)
interval Collects data every specified minutes
Suggested setting is 10
WRKDSKSTS program to write output to spool and then to physical file
WRKDSKSTS: PGM PARM(&PARM1 &PARM2) /******************************************************************************/ /* Author: Russ Lewis, EMC Corporation */ /* Date: 3/9/01 */ /* Desc: This program is provided as an example of writing output from the */ /* WRKDSKSTS command to a spool file. The spool file(s) output will */ /* then be copied & combined into physical file QGPL/DSKSFOREMC. */ /* The purpose is to allow capturing of KB per I/O, separated by */ /* reads operations versus write operations, which is available */ /* from the WRKDSKSTS cmd in OS/400. The statistics for WRKDSKSTS */ /* are reset between each iteration. 2 character format parameters */ /* must be passed to this program, 4 bytes for the number of */ /* iterations to run, and 4 bytes for the number of minutes to wait */ /* between each iteration. If blanks or zeroes are passed as */ /* parameter 1, the default is to run for 24 iterations. If blanks */ /* or zeroes are passed for parameter 2, the delay time between each */ /* iteration will be 60 minutes. */ /* */ /* Example - SBMJOB CMD(CALL PGM(WRKDSKSTS) PARM('0012' '0005')) */ /* will run 12 iterations of the WRKDSKSTS command with a 5 minute */ /* delay between each interval. */ /* */
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P22/68
Technology Implementation Services
/* This program is not part of any EMC product & may be used and/or */ /* modified as desired. */ /* */ /* Modification History: */ /* */ /* 11/01/04 RJL - Changed &PARM1, &ITERATIONS, &COUNT and &SPOOLCOUNT from */ /* 2 to 4 or 5 byte variables to allow > 99 iterations to run. */ /* Also changed file size of QGPL/DSKSFORMEMC to *NOMAX to */ /* allow for more iterations with large configurations. */ /******************************************************************************/ DCL VAR(&PARM1) TYPE(*CHAR) LEN(4) DCL VAR(&PARM2) TYPE(*CHAR) LEN(4) DCL VAR(&ITERATIONS) TYPE(*DEC) LEN(4 0) VALUE(24) DCL VAR(&ITVMINUTES) TYPE(*DEC) LEN(4 0) VALUE(60) DCL VAR(&DELAYSECS) TYPE(*DEC) LEN(6 0) VALUE(0) DCL VAR(&COUNT) TYPE(*DEC) LEN(4 0) VALUE(0) DCL VAR(&SPOOLCOUNT) TYPE(*DEC) LEN(5 0) VALUE(0) IF COND(&PARM1 *GT '0000') + THEN(CHGVAR VAR(&ITERATIONS) VALUE(&PARM1)) IF COND(&PARM2 *GT '0000') + THEN(CHGVAR VAR(&ITVMINUTES) VALUE(&PARM2)) CHGVAR VAR(&DELAYSECS) VALUE(&ITVMINUTES * 60) LOOP1: IF COND(&COUNT *GE &ITERATIONS) + THEN(GOTO CMDLBL(WRITEFILE)) WRKDSKSTS RESET(*YES) /**** The reset creates a spool file that we don't use so delete it. ****/ DLTSPLF FILE(QPWCDSKS) SPLNBR(*LAST) DLYJOB DLY(&DELAYSECS) WRKDSKSTS OUTPUT(*PRINT) CHGVAR VAR(&COUNT) VALUE(&COUNT + 1) GOTO CMDLBL(LOOP1) WRITEFILE: DLTF FILE(QGPL/DSKSFOREMC) MONMSG MSGID(CPF2105) CRTPF FILE(QGPL/DSKSFOREMC) RCDLEN(132) + TEXT('Temporary Physical File for + WRKDSKSTS output') SIZE(*NOMAX) CHGVAR VAR(&SPOOLCOUNT) VALUE(&ITERATIONS * 2) CHGVAR VAR(&COUNT) VALUE(2) LOOP2: IF COND(&COUNT *GT &SPOOLCOUNT) + THEN(GOTO CMDLBL(ENDPGM)) CPYSPLF FILE(QPWCDSKS) TOFILE(QGPL/DSKSFOREMC) + SPLNBR(&COUNT) MBROPT(*ADD) /**** Add 2 to counter each time through to skip over the "extra" spool ****/ /**** file created by the statistics reset during each pass. ****/ CHGVAR VAR(&COUNT) VALUE(&COUNT + 2) GOTO CMDLBL(LOOP2) ENDPGM: ENDPGM
6.4 Record Layout
First Header
Field Description
Licensed program name for the OS
OS version
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P23/68
Technology Implementation Services
Unknown
Title Constant Work with Disk Status
Date Date (MM/DD/YY)
Time Time (HH:MM:SS)
Statistic pre-header
Field Description
Constant Elapsed time
Interval Elapsed time (HH:MM:SS)
Constant System name
Server Name Server name
Statistics
Heading Description
Unit System assigned number identifying a specific disk
Type The disk type
Size (M)
Disk space (MB)
% Used
Percentage of disk space allocated
I/O Rqs
Total I/Os
Request Size (K)
Average I/O size (KB)
Read Rqs
Total read IOPS
Write Rqs
Total write IOPS
Read (K)
Average Read I/O size (KB)
Write (K)
Average Write I/O size (KB)
% Busy
Percentage of time device was busy servicing a request
ASP Auxiliary Storage Pool ID
Protection Type
Protection Status
Compression
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P24/68
Technology Implementation Services
6.5 Sample Output
5722SS1 V5R2M0 020719 Work with Disk Status 10/13/04 11:07:28 Page 1 Elapsed time . . . . . . . : 01:00:00 System name . . . . . . . : A005 Size % I/O Request Read Write Read Write % --Protection-- Unit Type (M) Used Rqs Size (K) Rqs Rqs (K) (K) Busy ASP Type Status Compression 1 6717 6442 68.9 22.6 6.9 3.1 19.4 7.6 6.7 3 1 DPY ACTIVE 2 4326 30769 88.8 33.6 8.1 3.0 30.6 9.6 8.0 1 1 DPY ACTIVE 3 4326 30769 91.9 40.9 7.0 2.1 38.7 7.9 7.0 1 1 DPY ACTIVE 4 4326 30769 92.4 31.1 8.1 2.0 29.0 10.0 7.9 1 1 DPY ACTIVE 5 4326 30769 92.2 35.6 7.9 2.6 32.9 9.1 7.8 2 1 DPY ACTIVE 34 6717 6442 69.0 41.3 8.3 4.5 36.8 8.5 8.2 5 1 DPY ACTIVE 35 6717 6442 68.9 38.1 8.0 4.2 33.8 7.3 8.1 5 1 DPY ACTIVE 36 6717 6442 68.9 41.4 6.8 4.4 36.9 7.5 6.8 5 1 DPY ACTIVE 38 4326 30769 92.4 32.1 8.1 2.3 29.8 9.9 7.9 2 1 DPY ACTIVE
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P25/68
Technology Implementation Services
7 LINUX IOSTAT
7.1 Data Collection Considerations
Linux iostat formats may vary from version to version with some versions providing additional fields while others do not.
There are generally three types of UNIX OS device driver name types internal type drives (the cciss/* types), the sd* types, and the emcpower* types.
If PowerPath is installed on the host, then only the emcpower* (PowerPath) device driver names have valid iostat perf data and all of the lower level device driver metrics are not reliable and/or not even valid. (This is because of how the Linux/UNIX kernel works with the PowerPath kernel device driver.)
Multiple emcpower devices
For the sd* device driver names, then just like with Solaris you will see (as an example, sddj and then sddj1) possibly multiple entries for the same basic host device. In this example, sddj is totally encompassing for all subcomponent partitions (i.e. sddj shows total IOPs for sddj1 and sddj2 and any other partitions). Because of this, we need to strip out all numbers after the primary device driver name (supported with BCSD v1.6.2 parsers and above) as they are overlaps of the primary and if you dont, then the total sum of metrics like IOPs will be incorrect (doubled/tripled/etc.) from what is really being requested. (In other words, only the primary sd entry will be inclusive and if subsequent partition entries are also included in any totals, then the totals will be wrong.)
The command can be scheduled in cron or other scheduling products for the desired capture period.
7.2 Data Collection Procedure
Follow these steps to collect iostat data:
1. Identify the oservers and the specific LUNS (disks) that will be replicated.
2. Run iostat for a short duration and verify the output matches the required format (see Record Layout and Sample Output below).
3. On each host, run iostat from the command line or optionally create a cron entry.
4. Collect the iostat output file(s).
7.3 Command Syntax
The command syntax should be similar to the following and be issued on each host:
#iostat d x t interval count >> hostname.iostat.date.txt
Options Description
-d Displays only the device utilization report
-x Displays extended format
-t Displays the time of each report interval
interval Collects data every specified seconds
Suggested setting is 600 (10 minutes)
count Collects statistics for specified intervals
Suggested setting is 432 (3 days of 10 minute intervals)
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P26/68
Technology Implementation Services
CRON Syntax:
* * * * * iostat d x t interval count >> hostname.iostat.date.txt - - - - - | | | | | | | | | + day of week (1 - 7) (Monday = 1) (Sunday = 0 or 7) | | | +------- month (1 - 12) | | +--------- day of month (1 - 31) | +----------- hour (0 - 23) +------------- min (0 - 59)
Sample CRON Entry: 0 0 * * 5-7 iostat d x t 600 144 >> zeus.iostat.04142009.txt
This entry schedules iostat to start Friday at 00:00 and end Sunday at 00:00. The iostat command pulls statistics every 600 seconds (10 minute) for 24 hours creating 144 intervals. The UNIX redirect >> appends the output to the same file.
7.4 Record Layout
Header1 (may not exist)
Field Description
Date Date statistics were started
Time Time statistics were started
Year Year statistics were started
Header2
Field Description
OS Constant Linux
OS Version Version of the OS where iostat was run
Server Name Name of the server, in parenthesis, where iostat was run
Date Date preceded by a tab (MM/DD/YYYY)
CPU Info CPU type and quantity
Time
Field Description
Date Date (MM/DD/YYYY) May not exist
Time Time (HH:MM:SS AM/PM) or (HH:MM:SS)
Extended Statistics (Old format)
Heading Description
Device: Device name
rrqm/s Total read requests merged per second
wrqm/s Total write requests merged per second
r/s Total reads per second
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P27/68
Technology Implementation Services
w/s Total writes per second
rsec/s Total sectors read per second
wsec/s Total sectors written per second
avgrq-sz Average I/O size (KB)
avgqu-sz Average queue length
await Average I/O queue and service time (ms). AKA response time.
svctm Average I/O service time (ms)
%util Percentage of CPU time I/O requests were issued to the device
Extended Statistics (New format)
Heading Description
Device: Device name
rrqm/s Total read requests merged per second
wrqm/s Total write requests merged per second
r/s Total reads per second
w/s Total writes per second
rsec/s Total sectors read per second
wsec/s Total sectors written per second
rkB/s Total kilobytes read per second
wkB/s Total kilobytes written per second
avgrq-sz Average I/O size (KB)
avgqu-sz Average queue length
await Average I/O queue and service time (ms). AKA response time.
svctm Average I/O service time (ms)
%util Percentage of CPU time I/O requests were issued to the device
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P28/68
Technology Implementation Services
7.5 Sample Output
Old Format
Mon Aug 4 09:06:09 EDT 2008 Linux 2.6.18-53.1.4.el5 (bio201.agencourt.com) 08/04/2008 Time: 09:06:09 AM Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 2.51 215.32 6.42 5.97 1209.95 1770.33 240.60 0.27 21.57 3.09 3.83 sdb 0.02 6.47 0.05 0.58 3.60 56.39 94.95 0.03 42.96 1.21 0.08 sdc 29.51 191.91 89.37 5.69 5460.28 1585.99 74.12 0.28 13.58 1.52 14.42 dm-0 0.00 0.00 8.88 228.01 1212.62 1824.05 12.82 0.51 2.13 0.16 3.88 dm-1 0.00 0.00 0.12 0.33 0.92 2.66 8.00 0.04 99.84 0.40 0.02 Time: 09:16:09 AM Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.58 0.00 1.11 0.00 13.49 12.16 0.01 4.53 0.49 0.05 sdb 0.00 1.88 0.00 0.50 0.00 19.09 37.93 0.00 0.40 0.28 0.01 sdc 0.00 0.00 0.06 0.01 0.44 0.07 8.22 0.00 4.46 4.46 0.03 dm-0 0.00 0.00 0.00 4.07 0.00 32.59 8.00 0.01 2.83 0.16 0.07 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
New Format
Mon Aug 4 09:07:09 EDT 2008 Linux 2.6.9-67.ELsmp (bio202.agencourt.com) 08/04/2008 Time: 09:07:09 AM Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.10 112.46 2.82 4.57 229.87 162.64 114.94 81.32 53.10 0.31 41.26 3.54 2.62 dm-0 0.00 0.00 2.90 116.91 229.66 161.66 114.83 80.83 3.27 0.19 1.58 0.22 2.62 dm-1 0.00 0.00 0.02 0.12 0.18 0.98 0.09 0.49 8.00 0.04 290.93 0.61 0.01 Time: 09:17:09 AM Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 1.31 0.00 1.02 0.00 18.60 0.00 9.30 18.30 0.00 0.63 0.43 0.04 dm-0 0.00 0.00 0.00 2.32 0.00 18.60 0.00 9.30 8.00 0.00 0.61 0.19 0.04 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
EMC Modeling Tools Data Collection Guide
EMC GS/TIS
Technology Implementation Services
8 MAINFRAME (SMF/CMF)
8.1 Data Collection Considerations
Mainframes write SMF and/or CMF data. This data contains binary information and much more data than needed for modeling. A program to extract and analyze this data is available and called the Mainframe Assembler Kit. This program is written in assembler and is dependant on specific SMF record types being recorded on each LPAR.
The use of Mainframe SMF requires the following SMF records:
70 subtype 1, 73 74 subtype 1 and 5 78 subtype 3
The shortest SMF interval recording time the assembler program will accept is five minutes. In a multi-frame, multi-LPAR environment, it may not be possible for all systems to complete the writing of SMF interval data within the confines of a smaller recording interval, so five minutes is the smallest practical interval allowed by the program.
The assembler program, and the EMC remote replication modeling tools, do not support mixed SMF recording intervals. If there are systems that use different SMF recording intervals, the SMF data with like intervals are processed together, creating a mainframe CSV file for each different recording interval. As these CSV files have different collection intervals, the EMC remote replication modeling tools require separate projects for each of these as the data can not be combined into composite views.
The assembler program must have SMF (RMF record 74 subtype 5) data for ALL DEVICES. The cache counters contained in the RMF 74 subtype 5 records are maintained in the control unit and are universal to all systems. Users therefore often only record the RMF 74 subtype five records on one LPAR to avoid recording redundant data. The LPAR that records the RMF 74 subtype 5 records must have all disk devices varied online or cache statistics for the offline devices will not be recorded. This will result in incomplete input data and inaccurate results.
The CMF records written to the SMF dataset are a superset that includes the CMF-specific records and the RMF records used by the assembler program. Users often however, suppress the recording of the RMF records because they use CMF reports in their place and do not want to record data they are not using. The assembler program does not support the native CMF record formats. If the required RMF record types are not being recorded, the user must temporarily enable RMF recording during the data collection period.
EMC Modeling Tools Data Collection Guide
EMC GS/TIS
Technology Implementation Services
Figure 1. Mainframe SMF and CMF Data Collection Process
Process SMF/CMF Onsite or Offsite?
If possible, run the Mainframe Assembler kit at the customer site to create the CSV file necessary for the EMC remote replication modeling tools. This is usually the quickest and easiest way to process the data and minimizes the amount of data being transferred across the network. Alternatively, the required SMF/CMF records can be sent to EMC corporate via FTP or 3490 tape, processed on the Hopkinton mainframe, thus creating the CSV file for the EMC remote replication modeling tools.
8.2 Mainframe Assembler Kit
The Mainframe Assembler Kit is available in one of several ways:
5. Packaged in the BCSD install zip file as Mainframe Assembler Kit vX.zip.
6. For EMC internal users only, in the EMC GS Tools eRoom in the GS Tools Common Components folder. It is packaged in the file called Mainframe Assembler Kit vX.zip which can be downloaded from http://ctseroom02.corp.emc.com/eRoom/tsfuncprac/TSTools/0_53238. If you do not have access, please email [email protected] to request access.
The kit contains all necessary programs, sample JCL and a Users Guide.
8.2.1 Additional notes
Inputs
Control Unit FICON Channel Paths Data File, user parameters
Are there FICON channels?
Customer Onsite
Approach
EMC Offsite Approach
Get Mainframe Assembler Kit
Install Assembler Kit Onsite
Customize and run RMFCUGEN
Customize and run ET000CSV
Retrieve flat-file from Mainframe
(as CSV file)
TRSMAIN SMF/CMF data
FTP data to EMC
SMFDUMP SMF/CMF data
Dump to Tape and post to EMC
Obtain TSOid for Remote MF
access
Restore FTP or Tape data to E01
Yes
No
EMC Modeling Tools Data Collection Guide
EMC GS/TIS
Technology Implementation Services
Sorted RMF (and CMF) record types 70 subtype 1, 73, 74 subtype 1, 74 subtype 5, and 78 subtype 3 (74 subtype 1 and 74 subtype 5 are the only record types needed if all channels are ESCON)
The CSV file is sorted by DATE, TIME, and descending WRITES.
Volume Inclusion/Exclusion
If not replicating all volumes you can include or exclude using SYSIN control cards in ETALLCSV or during data import into the EMC remote replication modeling tools. The difference is:
If done in ETALLCSV, the CSV will strip out the volsers you specified and cannot be re-included in the remote replication tool unless ETALLCSV is rerun. The advantage is a smaller output file and quicker processing by the EMC remote replication modeling tools.
If done in the remote replication tool, the import include/exclude filters can be used against the same input file multiple times depending on the date to be modeled. The disadvantage is a larger output file and slower processing by the remote replication tool.
8.3 Onsite Processing
Follow these steps to collect mainframe data:
1. Identify the LPARs and the specific volumes that will be replicated
2. Install the Mainframe Assembler Kit
3. Modify and run the appropriate JCL
4. Collect the output file(s)
Optionally Upload Output to EMC FTP Site
Follow these steps to upload the CSV file(s) to EMCs FTP site:
1. Optionally zip the CSV file to reduce transfer times
2. Use FTP via the ISPF Command Shell
3. Connect to FTP.EMC.COM
4. Logon as user anonymous, password should be your email address
5. Remain in ASCII Transfer Mode
6. cd incoming/customer 7. put CSV-OUTPUT-FILE CSV FILENAME ON FTP SITE
8. Confirm that the CSV file was transmitted and QUIT
8.4 Offsite Processing
Step 1 Extract the Required SMF/CMF Records
The TRSMAIN program does not support tape, move the dataset to disk first!!! Edit and run the below SMFDUMP JCL to dump the Type 70, 73, 74 and 78 records to an output dataset.
//SMFDUMP JOB (EMC),,CLASS=S,MSGCLASS=X,MSGLEVEL=(1,1),
// NOTIFY=&SYSUID
//**************************************************
//* This job will collect SMF - Type 70, 73, 74 and 78 (dataset
//* level statistics) for I/O performance analysis
//**************************************************
EMC Modeling Tools Data Collection Guide
EMC GS/TIS
Technology Implementation Services
//*
//SMFDUMP EXEC PGM=IFASMFDP
//DUMPIN DD DISP=SHR,DSN=INPUT SMF DATASET
// DD DISP=SHR,DSN=EXTRA INPUT SMF DSN IF NEEDED
//OUTDD1 DD DISP=(,CATLG),DSN=TEMP OUTPUT DATASET,
// UNIT=SYSDA,SPACE=(CYL,(25,5),RLSE),
// VOL=(,,,5),DCB=(RECFM=VBS,LRECL=32760)
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
INDD(DUMPIN,OPTIONS(DUMP))
OUTDD(OUTDD1,TYPE(70,73,74,78))
Step 2 Terse the SMFDUMP output
Documentation on the TRSMAIN program can be found at:
http://techsupport.services.ibm.com/390/trsmain.html
Use the following JCL to Terse the output dataset prior to FTP transmission:
//TRSMAIN JOB (EMC),,CLASS=A,MSGCLASS=X,MSGLEVEL=(1,1),
// NOTIFY=&SYSUID,REGION=4M
//************************************
//* TRSMAIN with PACK option *
//*************************************
//TRSMAIN EXEC PGM=TRSMAIN,PARM=PACK
//SYSPRINT DD SYSOUT=A
//INFILE DD DISP=SHR,DSN=TEMP OUTPUT DATASET
//OUTFILE DD DISP=(NEW,CATLG,DELETE),SPACE=(CYL,(50,10),RLSE),
// DSN=TERSED DATASET NAME,
// DCB=(RECFM=FB,LRECL=1024,DSORG=PS)
Step 3 FTP the Tersed File to EMC
Use the following instructions to transmit the tersed file to EMCs FTP site.
1. Use FTP via the ISPF Command Shell. Connect to FTP.EMC.COM
2. Logon as user anonymous, password should be your email address.
3. BIN to change to BINARY Transfer Mode.
4. cd incoming/customer
5. put TERSED DATASET NAME TERSED FILENAME ON FTP SITE
6. Confirm that the tersed file was transmitted and QUIT.
EMC Modeling Tools Data Collection Guide
EMC GS/TIS
Technology Implementation Services
Step 4 Login to the Hopkinton Mainframe
See Accessing EMCs E01 LPAR for details.
Step 5 Retrieve the Tersed SMFDUMP file
Transmit the data from our FTP site to our analytics LPAR (E01).
1. Email the TS_Tools_Support mailbox for a TSO userid and password.
2. Logon to the E01 LPAR (IP ADDR=10.243.142.41)
3. Ensure that the target dataset name that you are going to use to receive the tersed file from the FTP site has been pre-allocated with the following DCB:
a. LRECL =1024 b. DSORG = PS c. RECFM = FB d. The dataset name should match TERSED INPUT FILE as per below.
4. Use FTP via the ISPF Command Shell. Connect to FTP.EMC.COM
5. Logon as user anonymous, password should be your email address.
6. BIN to change to BINARY Transfer Mode.
7. cd incoming/customer
8. get TERSED FILENAME ON FTP SITE TERSED INPUT FILE (REPLACE)
9. Confirm that the tersed file was received from the FTP site and QUIT.
Step 6 Unterse the SMFDUMP file
Use the following JCL to unpack the tersed file:
//TRSMAIN JOB (EMC),,CLASS=A,MSGCLASS=X,MSGLEVEL=(1,1),
// NOTIFY=&SYSUID,REGION=4M
//****************************************
//* TRSMAIN with UNPACK option *
//****************************************
//TRSMAIN EXEC PGM=TRSMAIN,PARM=UNPACK
//SYSPRINT DD SYSOUT=A
//INFILE DD DISP=SHR,DSN=TERSED INPUT FILE
//OUTFILE DD DISP=(NEW,CATLG,DELETE),SPACE=(CYL,(50,10),RLSE),
// DSN=TARGET UNPACKED FILE ON EMC E01
NOTE: The TARGET UNPACKED FILE should have the same attributes as the original TEMP OUTPUT DATASET.
8.5 Record Layout
Field Description SITE Site/Customer Identifier
IDATE Interval Start Date (MM/DD/YYYY)
ITIME Interval Start Time (HH:MM)
ISEC Interval Elapsed Seconds (SSSS)
EMC Modeling Tools Data Collection Guide
EMC GS/TIS
Technology Implementation Services
VOLSER Volser
DEVNAME Device Type Name
SGNAME SMS Storage Group Name
UNIT CCUU
SSID SSI
OEM Vendor
CUSER# Control Unit Serial #
IREADS All Device Reads (random and sequential)
IWRITES All Device Writes (random and sequential)
ISEQREADS Sequential Device Reads
ISEQWRTS Sequential Device Writes
IWRTBLKL Average Write Block Length (Same as IREADBLKL if ESCON)
IREADBLKL Average Read Block Length (Same as IWRTBLKL if ESCON)
IDEVB True Device Busy All LPARs (PP.PP)
IOPS Device I/O/Sec All LPARs ((NNNN.N)N)
IRESP Device Response All LPARs (NNNN.N)
IQUE Device Avg Q Time All LPARs (NNNN.N)
ISERV Device Svc Time All LPARs (NNNN.N)
ICNN Device Connect All LPARs (NNNN.N)
IDIS Disconnect Time All LPARs (NNNN.N)
IPEND Device Pend Time All LPARs (NNNN.N)
CARHR Cache Read Hit % X 10 (PPP.P)
CARWR Read Percentage X 10 (PPP.P)
CAWHR Cache Write Hit % X 10 (PPP.P)
CASRR Cache Seq Read % X 10 (PPP.P)
CASWR Cache Seq Write % X 10 (PPP.P)
CASRH Cache Seq Read Hit % X 10 (PPP.P)
CASWH Cache Seq Write Hit % X 10 (PPP.P)
CADSR Cache Destage Count
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P35/68
Technology Implementation Services
8.6 Sample Output
VERIZON STE FICON ,12/25/2006,23:59,0900,E0M7B1,33909 , ,4262,4200,HTC,000000022352, 4065, 26, 20, 0, 24886, 61863, .60, 4.54, 1.9, 0.0, 1.9, 1.1, 0.3, 0.5, 98.0, 99.3,100.0, 0.5, 0.0, 81.0, 0.0, 8 VERIZON STE FICON ,12/25/2006,23:59,0900,ADBPJT,33903 ,SGDB2PTS,4200,4200,HTC,000000022352, 496, 162, 0, 0, 24886, 61863, .31, .73, 4.9, 0.0, 4.9, 1.4, 2.9, 0.6, 50.9, 75.3,100.0, 0.2, 0.0,100.0, 0.0, 272 VERIZON STE FICON ,12/25/2006,23:59,0900,EDI1F1,33903 ,SGDB2PIX,4220,4200,HTC,000000022352, 679, 17, 1, 0, 24886, 61863, .37, .77, 5.4, 0.0, 5.4, 0.7, 4.2, 0.5, 49.1, 97.4,100.0, 0.2, 0.0,100.0, 0.0, 40 VERIZON STE FICON ,12/25/2006,23:59,0900,ENH1FD,33909 ,SGEVNHA ,4260,4200,HTC,000000022352, 98, 4, 2, 0, 24886, 61863, .06, .11, 6.0, 0.0, 6.0, 5.3, 0.5, 0.3, 95.6, 95.7,100.0, 2.2, 0.0,100.0, 0.0, 0
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P36/68
Technology Implementation Services
8.7 Accessing EMCs E01 LPAR
Before you attempt to use the z/OS TSO LPAR you need to be familiar with TSO, ISPF and JCL. Additionally, you need to email TS_Tools_Support in order to obtain access to a TSOid and Password for temporary use. Keep in mind that you must be behind EMCs Firewall in order to access this LPAR.
The first step in getting access to the LPAR, is obtaining and installing a 3270 emulator. EMC uses TN3270 which can be downloaded from http://mfe01.lss.emc.com. The site will look something like this:
Now click on Get Vista TN3270. This will download the install program to your PC. Once downloaded, install and then start it.
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P37/68
Technology Implementation Services
This will start a new terminal session:
When prompted, type the host IP name mfe01.lss.emc.com, then click Connect.
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P38/68
Technology Implementation Services
Now you can login to TSO:
Just type in the supplied TSOid and password to bring up ISPF. As there is only a small pool of TSOids, your assigned TSOid could be shared, so please do not acquire a session if it is already in use.
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P39/68
Technology Implementation Services
9 MIRRORVIEW/A LOR (LOCALITY OF REFERENCE) MirrorView/A LOR (locality of reference) provides host-based (Windows only) I/O statistics for modeling MirrorView/A replication solutions with increased accuracy. This tool attempts to mimic what MirrorView/A will actually do if this same workload was run on a CLARiiON array with MirrorView/A active. This is done by determining the locality of reference and how that impacts COFW for snapshots and data movement (SnapView and Incremental SAN Copy).
9.1 Data Collection Considerations
There are two steps that need to be run. The first step creates a trace file and the second step uses that trace file as input and creates an analysis file for import.
Collection will cease when one of three conditions occurs: when a non-zero duration is complete, when the output file reaches a size of max-output-size, or when the user executes locality with the stop operation.
When importing LOR files into the EMC remote replication modeling tools, the Time Zone must be set to GMT.
Warning! This tool can produce many GBs of trace data depending on the number of volumes attached to the server and, the sub-period and duration specified.
9.2 Acquiring the MirrorView/A LOR Tool
The Mainframe Assembler Kit is available in one of two ways:
1. Packaged with the EMC remote replication modeling tools in the tools install subfolder called MV/A LOR Tool.
2. In the EMC GS Tools eRoom in the GS Tools Common Components folder. It is packaged in the file called MirrorView-A LOR Tool.zip which can be downloaded from:
http://ctseroom02.corp.emc.com/eRoom/tsfuncprac/TSTools/0_5499c
If you do not have access, please email [email protected] to request access. If you are a non-EMC user, email the support mailbox above and provide the contact details of your EMC account team after which the kit will be made available to you.
The kit contains the program and a Users Guide.
9.3 Data Collection Procedure
Follow these steps to collect the LOR tool data:
1. Identify the servers and the specific LUNS (disks) that will be replicated 2. On each host, run locality trace from the command line
3. Once the locality trace command has completed, run locality analyze from the command line
4. Collect the LOR output file(s)
9.4 Command Syntax
Create Trace File
The command syntax should be similar to the following and be issued on each host:
locality trace file max-output-size duration
Options Description
file File name for trace data (should end with the file extension .elt)
max-output- Specifies a number [modifier] where modifier may be k or K (kilobytes), or m or M (megabytes),
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P40/68
Technology Implementation Services
size and must be at least 1 MB.
duration
The duration is specified as number [modifier] where modifier can be s or S (seconds, default) m or M (minutes), h or H (hours), or d or D (days)
Suggested setting is 3d (3 days)
Analyze Trace File
The analyze must run on same server where .elt file was created. The command syntax should be similar to the following and be issued on each host:
locality analyze file period sub-period chunk-size grain-size output-prefix
Options Description
file File name for trace data (should end with the file extension .elt)
period MirrorView/A update period (must match what is specified in the EMC remote replication modeling tools or you may get erroneous results)
sub-period
Collection interval (1 minute default and must be a divisor of the MirrorView/A update period specified)
Suggested setting is 5 minutes to minimize file size and allow period changes in increments of 5 minutes
chunk-size Should always be 64K (used to determine reserved LUN pool size and max COFW operations/sec)
grain-size FLARE 14/16 should be 16KB and FLARE 19+ should always be 2KB (used to determine network throughput)
output-prefix Prefix applied to CSV output files
Example
A typical use of the locality program:
locality trace q:\logs\JetStress.elt 200 24h
locality analyze q:\logs\JetStress.elt 1h 5m 64k 2k 1hour
The first command begins a trace into q:\logs\JetStress.elt. The trace will run for 24 hours, and will stop collecting data once 24 hours is reached or q:\logs\JetStress.elt reaches 200 MB.
The second command will analyze the data for a 1 hour MirrorView/A update cycle (which would result in a 2 hour RPO), reporting incremental CoFW data every 5 minutes, using the default 64KB SnapCopy chunk size and default 2KB MirrorView/A transfer size and will create the files 1hour.csv and 1hour_log.csv.
9.5 Sample Output
LoR version,0.7 Timestamp,7/20/2007 16:26:05 PM disk,seconds,period,# reads,# writes,bytes read,bytes written,chunk size,modified chunks,overwritten chunks,cofw chunks,chunk size,modified chunks,overwritten chunks,cofw chunks 0,60,1,0,33,0,107008,65536,5,28,0,2048,50,25,0 0,120,0,0,9,0,32256,65536,3,7,2,2048,18,6,12 0,120,1,0,9,0,32256,65536,3,7,2,2048,18,6,12 0,180,1,0,24,0,93696,65536,5,20,2,2048,50,19,12 0,240,0,0,34,0,125440,65536,4,31,2,2048,43,50,12 0,240,1,0,34,0,125440,65536,4,31,2,2048,43,50,12 0,300,1,0,9,0,36864,65536,2,8,1,2048,19,8,1 0,360,0,0,28,0,100864,65536,3,26,0,2048,32,43,0 0,360,1,0,28,0,100864,65536,3,26,0,2048,32,43,0 0,420,1,0,31,0,108032,65536,4,27,2,2048,47,34,12 0,480,0,0,16,0,56320,65536,3,14,2,2048,20,22,12 0,480,1,0,16,0,56320,65536,3,14,2,2048,20,22,12
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P41/68
Technology Implementation Services
10 NAVISPHERE ANALYZER (NAR)
10.1 Data Collection Considerations for metaLUNs and Thin LUNs
10.1.1 RAW files
With raw NAR file support, conversion to CSV is no longer necessary. Note that loading of FLARE 28 and older raw NAR files may fail due to a bug in FLARE which causes an incorrect earliest polltime or Error parsing MetaLUN Stats. Converting these files to CSV and loading the NAR CSV files into BCSD solves this problem.
10.1.2 CSV files
For FLARE 28 and above and you can specify metaLUNs and thin LUNs in the NAR to CSV conversion allowing you to create an include or exclude list based on meta-heads, otherwise you must specify all of the meta-members within a metaLUN for the meta to be properly included in the modeling.
10.2 Data Collection Procedure
Step 1 Start Analyzer Performance Logging
As the data is shared between BOTH SPs, you only need to collect and process a single SPs files, but it may be desired to retrieve NAR files from both SPs in case one file is corrupt. For a MirrorView/A analysis, the collection period should be specified as a divisor of the desired cycle time which is half of the desired RPO time. If you expect an RPO of 2 hours, then the data should be collected in 1 hour intervals or less, such as 30 minutes or even 10 minutes. For any other analysis, the time interval should be specified as 10 minutes or less.
Changing the narinterval, which affects both SPs, while logging is active will result in a file with multiple collection intervals. To prevent this, you can first determine if logging is active and what the current collection interval is. If archiving is enabled, then you must stop Analyzer, change the Archive Interval, and then start Analyzer. When you stop analyzer, the current archive data will be saved in a new archive. Use the analyzer STOP command, then the START command to start Analyzer running with the new narinterval.
1. Check if logging is active naviseccli -address -user -password -scope analyzer -status
2. Display the archive interval naviseccli -address -user -password -scope analyzer -get
3. If necessary, change the narinterval (remember to enable Periodic Archiving) and restart logging naviseccli -address -user -password -scope analyzer stop naviseccli -address -user -password -scope analyzer -set -narinterval -periodicarchiving 1 naviseccli -address -user -password -scope analyzer start
Step 2 Copy NAR archive file(s) to your local hard drive
List all available NAR files stored on the array
naviseccli -address -user -password -scope analyzer archive -list
Retrieve a specific NAR file
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P42/68
Technology Implementation Services
naviseccli -address -user -password -scope analyzer -file -location -overwrite -retry
Retrieve all Files
naviseccli -address -user -password -scope analyzer all -location -overwrite -retry
For raw NAR files, no further processing of the files is required.
Step 3 Convert NAR file(s) to CSV file(s)
FLARE 30 and Above CLI Host LUNs (allows you to specify meta-heads only)
You can use the latest version of Navisphere CLI unless your NAR file is pre-FLARE 26.
For include/exclude lists, meta-heads must be specified and not the meta-members (meta-heads and not meta-members are presented to the host)
naviseccli analyzer archivedump data -out -object hl -format on,pt,rb,rs,rio,wb,ws,wio
FLARE 28 and Above NAR FLARE LUNs, Meta-Members and Thin LUNs
For include/exclude lists, meta-members must be specified and not the meta-heads
naviseccli analyzer archivedump data -out -object l tl -format on,pt,rb,rs,rio,wb,ws,wio
FLARE 28 and Above NAR Meta-Heads and Thin LUNs
For include/exclude lists, meta-heads must be specified and not the meta-members
naviseccli analyzer archivedump data -out -object ml tl -format on,pt,rb,rs,rio,wb,ws,wio
FLARE 24 and 26 NAR
For include/exclude lists, meta-members must be specified and not the meta-heads
naviseccli analyzer archivedump data -out -object l -format on,pt,rb,rs,rio,wb,ws,wio
10.2.2 Sample batch file to convert all NAR files in a directory to CSV files
Replace the naviseccli command with any of the Step 3 Convert NAR file(s) to CSV file(s) commands. The example is using FLARE 30 and Above CLI Host LUNs (allows you to specify meta-heads only)
path "C:\Program Files\EMC\Navisphere CLI";%path% for %%f in (*.nar) do naviseccli analyzer archivedump data -out -object hl -format on,pt,rb,rs,rio,wb,ws,wio
10.3 Command Syntax
Options Description
-address Specifies the IP address or network name of the targeted SP on the desired storage system
-user Specifies the username on the storage system you want to login to
-password Specifies the password on the storage system you want to login to
-scope Specifies whether the user account on the storage system you want to log in to is local, global, or lightweight directory access protocol (LDAP). 0 (default) indicates
EMC Modeling Tools Data Collection Guide
EMC GS/TIS P43/68
Technology Implementation Services
global, 1 indicates local, and 2 indicates LDAP.
analyzer Issue an Analyzer command
archiveretrieve Creates and Retrieves a single archive file
archivedump Dumps Analyzer archive files to a CSV file
-all Retrieves all files
-data Dumps data from archive files
-file Specifies the name of the archive file stored on the array to be retrieved
-format
Specifies which performance characteristics to output and the order in which they appear. The -format switch requires one or more of the codes listed below (not all are listed):
ml MetaLUN
l LUN
tl Thin LUN
-get Returns the current values of the performance logging properties
-list Lists all the archive files
-location Specifies t