RMF WTO 2006 - FILE/RMF_WTO_2010.pdf · How to Generate WTO Messages from RMF Performance Exceptions This document can be found on the web, ... 4 STEP 1: Start an interactive Monitor

How to Generate WTO Messages from RMF

Performance Exceptions

This document can be found on the web, www.ibm.com/support/techdocs

Search for document number WP100402 under the category of “White Papers”.

Version Date: March 2010

IBM Advanced Technical Support

Mike Bonett

[email protected]

© Copyright 2010, IBM Corporation Version 3/8/2010

http://www.ibm.com/support/techdocs

How to Generate WTO Messages from RMF Performance Exceptions Page 2 of 15

Trademarks

The following terms are registered trademarks of International Business Machines Corporation in the United States

and/or other countries: AIX, AS/400, DB2, IBM, Micro Channel, MQSeries, Netfinity, NUMA-Q, OS/390, OS/400,

Parallel Sysplex, PartnerLink, POWERparallel, RS/6000, S/390, Scalable POWERparallel Systems, Sequent, SP2,

System/390, ThinkPad, WebSphere.

The following terms are trademarks of International Business Machines Corporation in the United States and/or

other countries: DB2 Universal Database, DEEP BLUE, e-business (logo), ~, GigaProcessor, HACMP/6000,

Intelligent Miner, iSeries, Network Station, NUMACenter, POWER2 Architecture, PowerPC 604,pSeries, Sequent

(logo), SmoothStart, SP, xSeries, zSeries. A full list of U.S. trademarks owned by IBM may be found at

http://iplswww.nas.ibm.com/wpts/trademarks/trademar.htm. NetView, Tivoli and TME are registered trademarks

and TME Enterprise is a trademark of Tivoli Systems, Inc. in the United States and/or other countries.

Oracle, MetaLink are registered trademarks of Oracle Corporation in the USA and/or other countries.

Microsoft, Windows, Windows NT and the Windows logo are registered trademarks of Microsoft Corporation in the

United States and/or other countries.

UNIX is a registered trademark in the United States and other countries licensed exclusively through The Open

Group.

LINUX is a registered trademark of Linus Torvalds.

Intel and Pentium are registered trademarks and MMX, Pentium II Xeon and Pentium III Xeon are trademarks of

Intel Corporation in the United States and/or other countries.

Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States and/or

other countries.

Other company, product and service names may be trademarks or service marks of others.




Introduction................................................................................................................................................................. 4 Architecture and Flow Overview .............................................................................................................................. 4 STEP 1: Start an interactive Monitor III session..................................................................................................... 5 STEP 2: Define the desired performance exception(s). ........................................................................................... 5 Step 3: Customize the ERBRMF3B JCL to run Monitor III as a started task or batch job.............................. 11 General Observations and Considerations ............................................................................................................. 14 Summary ................................................................................................................................................................... 14 Further Reference and Detailed Information......................................................................................................... 15




Introduction

IBM Resource Measurement Facility (RMF) provides the online RMF Monitor III session. Using

ISPF, a user can invoke an interactive Monitor III session to access reports; one of these is the

Workflow/Exceptions Report. This report presents information about system activity and system

resources. Within this report, system and resource performance threshold exceptions can be

defined. Exceptions (conditions that violate the thresholds) will be displayed on the ISPF screen

during a Monitor III monitoring cycle (for example, a job using a certain amount of CPU

processing, or a DASD volume at a certain I/O activity rate). Details of the

Workflow/Exceptions report options, and the performance metrics available for use in defining

exceptions, are documented in the RMF Report Analysis manual (SC33-7991).

It may be desirable to surface these exceptions as Write to Operator (WTO) messages, where

they can logged in the SYSLOG, observed by human operators, or (best of all) detected and

acted upon by automation products such as Tivoli NetView for z/OS or System Automation for

z/OS. This paper describes one technique to accomplish this, using functions and capabilities

provided by RMF. The steps to perform for this technique are:

1. Starting an interactive RMF Monitor III session.

2. Creating the desired performance exceptions.

3. Customizing JCL to run the RMF Monitor III session as a started task or batch job.

Architecture and Flow Overview

When RMF is installed, a set of CLISTS and REXX procedures are copied into the

SYS1.SERBCLS library (an installation may have changed the library name to meet local

naming standards). Details on the CLISTS and execs are documented in the RMF Users Guide

(SC33-7990), in the “RMF Client/Server Enabling” Chapter. The following diagram shows the

key CLISTS and procedures that are used to generate the WTO messages:

1. JCL (the RMF supplied procedure RMFM3B can be used) invokes the ERBM3B REXX

procedure, which controls the Monitor III background session.

ERBM3B ERB3RPH3

ERBM3BWX

ERBR3WFX

JCL

Started task

or batch job

Monitor III

Batch Control

Table Switch

Reporter Exit

WFEX Exit




2. ERBM3BWX allocates the Monitor III table dataset used to hold the report information.

3. ERB3RPH3 determines the report type (which is passed as a parameter in the JCL) and

calls the corresponding report processing exit handler.

4. ERBR3WFX is the WFEX report processing exit handler. It checks for exceptions in the

WFEX report, and issues any found as WTO messages. There are three exit handlers

contained in ERBR3WFX, each performing different WTO processing.

These CLISTs and REXX procedures can be easily modified to suit specific needs. For this

process a specific WFEX report process is used. ERBR3WFX exit handler 2 produces a WTO

message for each exception, and will be used. There are 2 ways accomplish this:

1. Make a copy of the ERBRPH3 REXX procedure, and modify its call to ERBR3WFX to pass

the desired exit handler number. Within the REXX code locate the line containing

SELECT CMD(ERBR3WFX)and change it to SELECT CMD(ERBR3WFX 2).

2. Make a copy of the ERBR3WFX REXX program and set exit handler 2 to always be

invoked. Within the REXX code locate the line Arg handler . and add the line

handler=2 right after it.

Any modified REXX procedure must be placed in the dataset at the top of the SYSPROC DD

concatenation in the JCL used within this process (documented later in this paper).

The rest of this paper will walk through the steps to define a DASD device activity rate threshold

and will show the resulting WTO messages that are generated.

STEP 1: Start an interactive Monitor III session.

The SERBCLS library (or its customized equivalent) must be allocated to the SYSPROC

concatenation of the RMF Monitor III user’s TSO session, to allow a Monitor III session to be

started from ISPF. Most installations have customized ISPF and added RMF as an application

that can be invoked directly via an ISPF command.

When a Monitor III ISPF session is started, a user ISPF output table dataset is allocated (it is

created if it does not exist) by the SERBCLS(ERBRMF3X) CLIST. This output table contains

the defined exceptions. The default name is <TSOUSERID>.RMFOSLEVEL.ISPTABLE (this

can be changed by modifying the ERBRMF3 and ERBRMF3X CLISTs in the SERBCLS

library). In this example, the dataset is named BONETT.RMFZOS12.ISPTABLE.

STEP 2: Define the desired performance exception(s).

From the RMF ISPF Main Panel, select the Monitor III Session (Option 3):




From the RMF Monitor III Primary Menu, select OVERVIEW

(Option1):




From the OVERVIEW Report Selection Menu, Select the Workflow/Exceptions (WFEX) Report

(Option 1)

The Workflow/Exceptions Report shows the current status of system activity; the bottom portion

of the screen shows current exceptions that have violated existing defined thresholds. These will

be issued as WTO messages when this process is completed.

For this example an exception to report any DASD device with a Device Activity Rate greater

than 5 will be created. Type in RO at the command prompt, to access the Report Options:




The Report Options Action Panel displays the defined exceptions. To add an exception

definition, type AD on the first line under the Action column:




The Definition and Criteria Screen appears. The information on this screen is defined in detail in the

RMF Report Analysis Guide. The following entries are created to define the I/O rate exception:

- CLASS ==> DEV

- QUALIFIER ==> ALL

- INIDICATOR ==> EX-ANY

- LABEL ===> (optional – left blank here)

- ALERT ===> NONE (relevant to screen display only)

Under Criteria Set 1, on the first line, DAR > 5 is entered. DAR is the metric representing

Device Activity Rate (Details on creating more sophisticated criteria definitions using resource

filtering, multiple metrics and Boolean logic are documented in the RMF Report Analysis

manual):

After completing the screen, press F3. The exception that has just been defined is now listed on

the Report Options Action Panel:




Press PF3 to exit the action panel. Any DASD devices that, in the latest cycle, have an activity

rate greater than the threshold are shown in the exceptions section:




Press PF3 several times to exit the RMF Monitor III session. The exception has now been saved

in the RMF ISPF user table.

Step 3: Customize the ERBRMF3B JCL to run Monitor III as a started task or batch job.

When RMF is installed, a started task procedure RMFM3B is provided. This runs the

Monitor III session and will invoke the appropriate RMF CLISTS to capture any surfaced

exceptions and produce related WTO messages (RMFM3B can be modified to run as a batch job,

if desired).

The JCL invokes the functions (described in the Overview section of this document) to do the

following:

- Accept parameters for the HLQ of the table dataset with the exceptions, and the type of

report to run.

- Create an ISPF table dataset in which exceptions surfaced by the report will be placed.

- Read the ISPF table dataset that the exception definitions were stored in Step 2 above, to

apply to the performance data.

- Invoke the appropriate report exit (for the workflow/exception report, this is the

ERBR3WFX RMF REXX procedure) to process the report at each cycle and create the WTO

messages for any exceptions.

The RMFM3B JCL will likely have to be modified for specific environments. Here is an

example of a working modification. Note that the JCL shown is for this specific environment and

will have to be modified for any other installation environment. It is based on the RMFM3B

sample provided with RMF (the example changes the task name to RMFWTO, the dataset names

must be adjusted for the target environment):




//RMFWTO PROC ISPF='SYS1.ISPF',REPORT=WFEX,HLQ=TEST,CLASS=O //DELETE EXEC PGM=IEFBR14 //ERBPHDS3 DD DSN=&HLQ..ERBPHDS3.ISPTABLE,DISP=(SHR,DELETE,DELETE) //SYSPRINT DD SYSOUT=* //ALLOC EXEC PGM=IEFBR14 //ERBPHDS3 DD DISP=(NEW,CATLG),DSN=&HLQ..ERBPHDS3.ISPTABLE, // UNIT=SYSDA,VOL=SER=DSX304, // SPACE=(TRK,(1,1,1)), // DCB=(LRECL=80,RECFM=FB,BLKSIZE=3120) //SYSPRINT DD SYSOUT=* //ERBM3B EXEC PGM=IKJEFT01,REGION=4M,DYNAMNBR=90, // PARM='ERBM3B &HLQ &REPORT' //ISPPROF DD DSN=&&TEMP,UNIT=SYSDA,SPACE=(TRK,(2,1,2)), // DCB=(DSORG=PO,LRECL=80,BLKSIZE=3120,RECFM=FB) //ISPLOG DD DSN=&&TEMP,UNIT=SYSDA,SPACE=(TRK,(5,5)), // DCB=(DSORG=PS,LRECL=125,BLKSIZE=129,RECFM=VA) //SYSPROC DD DISP=SHR,DSN=BONETT.CLIST // DD DISP=SHR,DSN=SYS1.COMBINED.ISPFCLIB // DD DISP=SHR,DSN=&ISPF..SISPEXEC // DD DISP=SHR,DSN=&ISPF..SISPCLIB //ISPPLIB DD DISP=SHR,DSN=&ISPF..SISPPENU // DD DISP=SHR,DSN=SYS1.COMBINED.ISPFPLIB //ISPSLIB DD DISP=SHR,DSN=&ISPF..SISPSENU //ISPMLIB DD DISP=SHR,DSN=&ISPF..SISPMENU // DD DISP=SHR,DSN=SYS1.COMBINED.ISPFMLIB //ISPTLIB DD DISP=SHR,DSN=&ISPF..SISPTENU // DD DISP=SHR,DSN=&HLQ..ERBPHDS3.ISPTABLE //***THE FOLLOWING IS THE DATASET CONTAINING THE WFEX DEFINITIONS // DD DISP=SHR,DSN=BONETT.RMFZOS12.ISPTABLE //***THE FOLLOWING 2 DATASETS ARE THE REQUIRED RMF ISPF TABLE DATASETS //***THEY MAY HAVE BEEN MOVED ELSEWHERE IN YOUR ENVIRONMENT // DD DISP=SHR,DSN=SYS1.RMF.SERBTENU

// DD DISP=SHR,DSN=SYS1.RMF.SERBT // DD DISP=SHR,DSN=SYS1.COMBINED.ISPFTLIB /ISPTABL DD DISP=SHR,DSN=&HLQ..ERBPHDS3.ISPTABLE //SYSTSPRT DD SYSOUT=&CLASS //SYSTSIN DD DUMMY //SYSPRINT DD SYSOUT=&CLASS

The key aspects of the JCL are highlighted in bold:

- The DELETE and ALLOC steps simply delete and recreate the ISPF table

(&HLQ.ERBPHDS3.ISPTABLE) that will hold active exceptions detected by the task. This

is done to ensure that any exceptions from previous runs are deleted.

- The ERBM3B step invokes the TSO environment. The parameters passed are the initial

CLIST to be run (ERBM3B), the High-Level qualifier of the ISPF table for the active

exceptions (&HLQ), and the Report to be run in batch (&REPORT). Since this example

focuses on the Workload/Exceptions report, the default &REPORT value is set to WFEX.

- BONETT.CLIST is a custom dataset that is the first one allocated to SYSPROC. It contains

copies of the RMF CLISTS and REXX execs from the SERBCLS library, so that they can be

modified as needed. Either ERB3RPH3 or ERBR3WFX will have to be modified as




described earlier and placed in a custom dataset to get a WTO messages for each

exception3B.

- &HLQ.ERBPHDS3.ISPTABLE is allocated to both ISPTLIB and ISPTABL DD statements.

- BONETT.RMFZOS12.ISPTABLE was the dataset used in the online RMF Monitor III

session in Step 2, and contains the saved exception definitions; this is allocated to ISPTLIB.

- The SERBT and SERTENU datasets are required in the ISPTLIB allocation.

- The COMBINED datasets for this environment contain the required RMF members for that

DD. For use in another environment ensure that all RMF related ISPF datasets are in the

appropriate ISPFxLIB DD concatenation.

When the started task is submitted, the RMF Monitor III session runs in the address space. At the

Monitor III cycle interval, it will output WTO messages for exceptions occurring during the

interval. The following is an example from the SYSLOG of the output produced:

17:38:27.37 BONETT 00000290 S RMFWTO 17:38:27.74 STC09747 00000090 $HASP100 RMFWTO ON STCINRDR 17:38:27.84 STC09747 00000290 IEF695I START RMFWTO WITH JOBNAME RMFWTO IS ASSIGNED TO USER STCRACF 17:38:27.84 STC09747 00000090 $HASP373 RMFWTO STARTED 17:38:27.88 STC09747 00000090 IEF403I RMFWTO - STARTED - TIME=17.38.27 17:38:36.89 STC09747 00000090 +RMF100I 3B: PROCESSING WFEX REPORT... 17:38:36.89 STC09747 00000090 +RMF100I 3B: NAME REASON CRITICAL VAL. 17:38:36.89 STC09747 00000090 +RMF100I 3B: --------------------------------------- 17:38:36.89 STC09747 00000090 +RMF102I 3B: IAM200 DAR > 5 7.900 /SEC 17:38:36.89 STC09747 00000090 +RMF102I 3B: WEBHFD1 NOT AVAIL 17:38:36.89 STC09747 00000090 +RMF102I 3B: WEBHFD2 NOT AVAIL 17:38:36.90 STC09747 00000090 +RMF102I 3B: ZO2RS1 DAR > 5 8.100 /SEC 17:38:36.90 STC09747 00000090 +RMF102I 3B: HSM NOT AVAIL 17:38:36.90 STC09747 00000090 +RMF103I 3B: END OF WFEX REPORT... 17:39:02.50 STC09747 00000090 +RMF100I 3B: PROCESSING WFEX REPORT... 17:39:02.50 STC09747 00000090 +RMF100I 3B: NAME REASON CRITICAL VAL. 17:39:02.50 STC09747 00000090 +RMF100I 3B: --------------------------------------- 17:39:02.51 STC09747 00000090 +RMF102I 3B: WEBHFD1 NOT AVAIL 17:39:02.51 STC09747 00000090 +RMF102I 3B: WEBHFD2 NOT AVAIL 17:39:02.51 STC09747 00000090 +RMF102I 3B: WRK28C DAR > 5 7.500 /SEC 17:39:02.51 STC09747 00000090 +RMF102I 3B: ZO2RS1 DAR > 5 10.50 /SEC 17:39:02.51 STC09747 00000090 +RMF102I 3B: HSM NOT AVAIL 17:39:02.51 STC09747 00000090 +RMF103I 3B: END OF WFEX REPORT... 17:39:32.92 STC09747 00000090 +RMF100I 3B: PROCESSING WFEX REPORT... 17:39:32.92 STC09747 00000090 +RMF100I 3B: NAME REASON CRITICAL VAL. 17:39:32.92 STC09747 00000090 +RMF100I 3B: --------------------------------------- 17:39:32.93 STC09747 00000090 +RMF102I 3B: WEBHFD1 NOT AVAIL 17:39:32.93 STC09747 00000090 +RMF102I 3B: WEBHFD2 NOT AVAIL 17:39:32.93 STC09747 00000090 +RMF102I 3B: HSM NOT AVAIL 17:39:32.93 STC09747 00000090 +RMF103I 3B: END OF WFEX REPORT... 17:40:02.93 STC09747 00000090 +RMF100I 3B: PROCESSING WFEX REPORT... 17:40:02.93 STC09747 00000090 +RMF100I 3B: NAME REASON CRITICAL VAL. 17:40:02.93 STC09747 00000090 +RMF100I 3B: --------------------------------------- 17:40:02.93 STC09747 00000090 +RMF102I 3B: WEBHFD1 NOT AVAIL 17:40:02.94 STC09747 00000090 +RMF102I 3B: WEBHFD2 NOT AVAIL 17:40:02.94 STC09747 00000090 +RMF102I 3B: HSM NOT AVAIL 17:40:02.94 STC09747 00000090 +RMF103I 3B: END OF WFEX REPORT...

Each Monitor III cycle (on this system the cycle time is 30 seconds) produces the WTO lines that

start with the RMF100I message and end with the RMF103I message. Exceptions are shown as

RMF102I messages. In this example Device Activity Rate exceptions occurred for volumes




IAM200 and ZO2RS1 in the first cycle and for WRK28C and ZO2RS1 in the second cycle. The

message also shows the actual device activity rate. In subsequent cycles the rate for all volumes

was below the thresholds, so there were no device activity rate exceptions for those cycles. In all

depicted cycles there were availability exceptions (specific monitored address spaces not found).

For the WFEX report, message IDs and message contents can be modified in the ERBR3WFX

REXX procedure. In this example, modifications were made to output the “Processing WFEX

Report…” and “End of WFEX Report…” messages around the exceptions for each cycle.

The message IDs shown in the output above are specific to the customization in the ERBR3WFX

REXX procedure for the example used in this paper. A specific installation should choose

message IDs that are unique for their environment, if using this procedure.

General Observations and Considerations

- Exceptions will be driven based on the Monitor III cycle time. If an exception occurs during

the cycle, it is not reported until the end of the cycle. The smaller the cycle, the closer the

exception message will be reported to the occurrence time of the exception situation.

- The format of the messages is the same as the exception format on the Monitor III display.

Automation products that process these messages will have to understand the message format

and parse the message contents to extract the desired information.

- There is no “clearing message” for exception conditions that return to normal. The exception

simply does not show up in the next cycle. Automation of these messages must take this into

account, and further customization of the ERBR3WFX procedure may be required. For this

example the RMF100I and RMF103I messages were added in the ERBR3WFX exit handler

2 for use by the automation product to identify the start and end of a monitoring cycle. Each

exception automation found in a cycle was saved, and was analyzed to determine if this was

a new or existing exception, or if an exception saved in a previous cycle was still reported.

This method determined if a previously detected exception did not reoccur – which indicated

the condition causing the exception had returned to normal.

Summary

The RMF Monitor III Workload/Exception report provides the ability to define workload and

performance based thresholds that an installation might want to provide notification. By using

the functions provided by RMF, as described in this paper, these exceptions can be sent as WTO

messages, and allow RMF performance indications to be integrated into operational monitoring

and automation efforts. Information from other Monitor III reports can be accessed in a similar

fashion.




Further Reference and Detailed Information

- RMF Newsletter 14, December 19, 2001 (contains a short article on applying this technique

to capture CPC Capacity Report information). It is available at

http://www.ibm.com/servers/eserver/zseries/zos/rmf/rmfhtmls/rmfnws14.html

- RMF Product Manuals - available online at http://www.ibm.com/servers/eserver/zseries/os

(select the operating system, then the library option to locate them):

- RMF Users Guide – details on the RMF Monitor III batch functions (SC33-7990).

- RMF Report Analysis - details on the Workflow/Exception report, defining, exceptions,

and available metrics (SC33-7991).

Documents

RMF WTO 2006 - FILE/RMF_WTO_2010.pdf · How to Generate WTO Messages from RMF Performance Exceptions This document can be found on the web, ... 4 STEP 1: Start an interactive Monitor