IMS Problem Investigator

© 2010 IBM Corporation

Information Management

Uncovering the Hidden Information in Your IMS Application Using IMS Problem Investigator

2010 – Information Management Tech Conference - Brazil




Uncovering the Hidden Information in Your IMS Application Using IMS Problem Investigator - Agenda

� This presentation will share an approach to quickly learn how to navigate through the IMS Problem Investigator online ISPF environment to enable you to see IMS log record flows related to IMS applications and their contents

� Section 1– IMS PI Overview

� Section 2 – Starting the IMS PI Dialog Panels

� Section 3– IMS PI ISPF Dialog Panel Overview

� Section 4– IMS PI Setup

� Section 5– Navigating Through a Log File in IMS PI

� Section 6– Examining IMS Application Log Record Flows

� Section 7– IMS PI Summary




Section 1:

IMS Problem Investigator Overview




IMS Problem Investigator Overview, Continue

� IMS PI provides a unified logical view of transactions spanning:

– A single IMS system

– IMS Sysplex system

– IMS/DB2/MQSeries

– IMS Connect




IMS Problem Investigator Overview, Continued

� IMS PI uses log records from a single source or across many sources to create a composite view of the entire transaction flow

– IMS PI supports the following log types as input: • IMS log• IMS Transaction Index created by IMS Performance Analyzer• IMS monitor and DB monitor files• Common Queue Server (CQS) log stream• IMS Connect event data collected by IMS Connect Extensions• OMEGAMON® Transaction Reporting Facility (TRF) log and extract• DB2 log• WebSphere® MQ log extract• SMF – IRLM Long Lock records

IMSPI




IMS PI Overview, Continued

� IMS PI reports enable you to see transaction/application interaction with the IMS system and across other systems as additional input sources are added to the

input selection.




Section 2:

Starting the IMS PI Dialog

Panels




IMS Problem Investigator Setup, Installation Libraries

� IMS Problem Investigator libraries– The components of the IMS Problem Investigator dialog are delivered in the following

libraries:• SALZEXEC REXX EXECs• SALZLINK Executable load modules• SALZMENU ISPF messages• SALZPENU ISPF panels• SALZSENU ISPF skeleton JCL• SALZTENU ISPF input tables

– These libraries will be installed on your MVS system under a specific high level qualifier

prefix

– Once installed, note the HLQ prefix that was used




IMS Problem Investigator Setup, Starting IMS PI

� IMS PI runs on your MVS system as an ISPF panel dialog

– You can start IMS PI by • Installing IMS PI libraries statically within your ISPF library setup • Add IMS PI to an ISPF menu• Start IMS PI dynamically

� The simplest way to start IMS PI is dynamically.– We’ll walk through the steps to dynamically start IMS PI in the following




IMS Problem Investigator Setup, Starting IMS PI

� Starting IMSPI dynamically – From your TSO session READY prompt, start your ISPF panel environment

• Example: READY• ISPF




IMS Problem Investigator Overview, Starting IMSPI , Continued

– Navigate to your TSO command processor panel • From the ISPF Primary Option Menu select (option 6),




IMS Problem Investigator Overview, Starting IMSPI , Continued

– From the ISPF Command Shell, enter the following command to start the IMS PI dialog• EXEC ‘hlq.prefix.SALZEXEC(ALZOREXX)' ‘hlq.prefix‘• Where hlq.prefix is the MVS dataset name High Level Qualifier and prefix assigned to the IMSPI

libraries




IMS Problem Investigator Overview, Starting IMS PI , Continued

� The IMS PI Dialog will appear as follows:




Section 3:

IMS PI ISPF Dialog Panel Overview





� From the IMS PI primary option menu, the main options can be selected.– A brief navigation map is presented in the following pages for each option





� Option 0: Profile – The profile section is used to define datasets and control cards necessary for IMS PI to

run in your environment

0





� Option 1: Process– Main processing center of IMSP PI where you can explicitly specify log files for

interactive analysis or batch processing.• Interactively browse and analyze formatted log files and extract data sets.• Merge log files from various sources to enable end-to-end analysis of transactions.• Generate and submit JCL to produce reports of formatted log records or to create log file extracts

or CSV files.

1

w s

s

v

SUB





� Option 2: Filter– Create or list filters which may be used to control the selection of log records for online

viewing or batch report and extract processing..

2 S/E





� Option 3: Forms– Create or list forms

• Forms control which fields from a given record type to display. You can use forms to customize the display for certain problem scenarios.

• A form is also used to specify the record format of a CSV file..

S/E3





� Option 4: Specify IMS systems and files• Used to identify the input data for reporting and analysis • You can define your IMS subsystems, their log and monitor files and your Connect systems

4 S

J





� Option 5: Groups– Specify groups of IMS and Connect systems (Sysplexes)

• IMS and IMS Connect systems can be grouped using this option for reporting and analysis purposes.

• IMS Problem Investigator merges the log files from each system in the group.• Existing groups can also be edited

5

S

J





� Option 6: Connect– Process IMS Connect Extensions archive journals

• Used to process IMS Connect event records using pre-defined systems and journal data sets maintained in the IMS Connect Extensions definitions repository

6

S

SUB





� Option 7: User Log– Define and prepare User Log records for formatting

• Used to process IMS User Log records using the same formatting and navigation features available to system log records.

1. Register the User Log Code, used to identify User Log records in IMS log files that require special processing.

2. Create the Knowledge Module used to format the User Log record.

7 S/E




Section 4:

IMS PI Setup




IMS PI Setup � Allow IMS PI to allocate your Control Data Set which stores filter and form

information using your High Level Qualifier determined by your TSO prefix and userid

Your personal HLQ “yourhlq”




IMS PI Setup, Define Your Dialog Profile

� Select option 0 (Profile) from the IMS Problem Investigator - Primary Option menu




IMS PI Setup, Define Your Dialog Profile

� Select option 1, (IMS PI Settings) from the IMS PI Profile Options Menu




IMS PI Setup, Continued

� Use the IMS PI Settings shown below as a guideline to quickly set up your IMS PI environment

Substitute your personal HLQ for all “yourhlq”

Replace “identify” with any unique identifier to represent your current

investigation

Replace this jobcard with your

local options. Large REGION

and TIME values are good.

Substitute the IMSPI HLQ for all

“IMSTOOL.IPS11“

*

*

*

**





� Hit PF3 to return to the “IMS PI Profile Options Menu”– Select option 2, (Extract Allocation Settings) from the IMS PI Profile Options Menu





� Enter the following values for your log extract and CSV datasets




IMS PI Setup, Key User Datasets

� Personal Profile Library:– IMS PI keeps your list of log file dataset names and IMS system definitions in this

library. – Defined in option 0.1 IMS PI Settings. – Consider creating a new dataset for each problem analysis

• Come up with a usable naming convention• ‘yourhlq.IMSPI.Dyymmdd.ABUxxxx.ALZPROF'

– Control Data Set (CDS)• IMS PI keeps your Filter and Form definitions in this VSAM KSDS.• Defined on the IMS Problem Investigator - Primary Option Menu

Personal Profile LibraryInput Log File and System Definitions

Control DatasetFilter and Form Definitions




Section 5:

Navigating Through a Log File in IMS PI




Navigating Through a Log File in IMS PI

� In this section, we will discuss an overall approach to accessing the function of

IMS PI.




Navigating Through a Log File , Summary of steps

Summary of steps presented in this section:

� Define your input log data to IMS PI

� View the contents of your input log data

� Synchronize GMT STCK and UTC Time

� Position yourself within the input log data– Find a time, record, or string

� Merging sysplex logs




Navigating Through a Log File in IMS PI, Define Input Datasets

� Define your input log data to IMS PI– Select option 1 (Process)





� Define your input log data to IMS PI in the “Process Log Files” panel– Initially, this panel will be empty.





� Define your input log data to IMS PI in the “Process Log Files” panel – Use a “/” in the line command field to list available line commands

• The “/” may also be used in other line command fields for action prompts

Enter a “/”to see

available line

commands





� Define your input log data to IMS PI in the “Process Log Files” panel– Enter your log dataset names here

– IMS release should match the IMS system that created the logs

Enter the IMS log dataset name

Enter the IMS release





� Specify time zone used to display STCK (Store Clock) timestamps inside

individual log records– Options are: GMT|LOCAL|+hhmm|-hhmm

– Caution: LOCAL converts the time to the local time of the MVS system currently running

IMS PI and may not match the local time of the system where the records were

produced.

Enter the time zone for timestamp presentation





� Begin analysis of the log dataset– Place an “S” in the line command area and hit <enter>

Begin log analysis by entering “S” here

S




Navigating Through a Log File in IMS PI, Browse Expanded/TIME View

� Individually formatted log records appear on the IMS PA BROWSE panel– The log records appear with their code and description

• The view below is the Expanded/Time view of the log records with information from global fields. such as transaction code, LTERM, program name, database, tracking unit of work ID, recovery token, as well as performance and status information.

– The PF11 key will rotate between 4 views, Expanded/Time view (shown below)

Expanded global field values

Log record code and description

Log Record STCK GMT timestamp




Navigating Through a Log File in IMS PI, Browse Compact Time View

� Individually formatted log records appear on the IMS PA BROWSE panel– Hit the PF11 key to rotate through the next of 4 views

– The Compact/Time view is shown below

Log record code and description only – no global data included





Navigating Through a Log File in IMS PI, Browse Expanded LSN View

� Individually formatted log records appear on the IMS PA BROWSE panel– Hit the PF11 key to rotate through the next of 4 views – The Expanded/LSN (Log Record Sequence Number) view is shown below

• The LSN uniquely identifies an IMS log record. • For merged files, the LSN is prefixed by a number to identify the input file from which it came

Log record code and description

Log Record LSN

Expanded global field values




Navigating Through a Log File in IMS PI, Browse Compact LSN View

� Individually formatted log records appear on the IMS PA BROWSE panel– Hit the PF11 key to rotate through the next of 4 views – The Compact/LSN (Log Record Sequence Number) view is shown below

• The LSN uniquely identifies an IMS log record. • For merged files, the LSN is prefixed by a number to identify the input file from which it came

Log record code and description – No

global data included

Log Record LSN




Navigating Through a Log File in IMS PI, Synchronize STCK UTC Time

� It is best to synchronize time values between the log record time stamps on the far right and the UTC timestamps within the expanded records

� Enter IMS PI command “Time” on the command line

UTC Global Time






� Select “2 . GMT” with leap second adjustment “3. Derive from log”– GMT STCK time and UTC times will be compatible when these options are chosen





� Notice tha the UTC and GMT times are now more closely representing the same

time zone





� However, the leap second conversion does not occur until IMS PI encounters log records to make the leap second calculation clear.

� Page forward (PF8) or enter command “Down 500” until the time stamp adjustment message is seen





� Notice that the Log Record STCK GMT timestamp now includes leap seconds

� The STCK and UTC timestamps will now follow each more closely during analysis

UTC Global Time

Log Record STCK GMT timestamp adjusted 23 seconds




Navigating Through a Log File in IMS PI, Positioning to a point in a file

� Log datasets can contain many records

� Positioning within the log files is an important time saving feature of log analysis





� Point and shoot navigation fields– “Forwards / Backwards . . HH.MM.SS.THMIJU Example“

• Navigate forwards and backwards in the record using time increments.• Enter the amount of time to move forward in the file in the modifiable HH.MM.SS.THMIJU field

– “Time of Day . . HH.MM.SS. THMIJU “• Go to an exact point of time in a log file with microsecond precision.• Enter the timestamp of the time you wish to position to in the file

– After adjusting the navigation value, place the cursor under the field label and hit enter

Time Increment

Point of time

Cursor Here Cursor Here




Navigating Through a Log File in IMS PI, Positioning to a Point in a File

� Point and shoot navigation fields– “Time of Day . . 23.37.00.000000 “ Example:

– Enter the time value in the modifiable field

– Place cursor under “Time of Day” field label

– Hit EnterEnter point

of time

Cursor Here




Navigating Through a Log File in IMS PI, Positioning to a Point in a File

� Point and shoot navigation fields– “Time of Day . . 23.37.00.000000 “ Example:

– The file will now be positioned at the first record after the chosen time

First log record after chosen time





� Finding values within the brows screen view– FIND|F search-string NEXT | PREV | FIRST | LAST PREFIX | SUFFIX | WORD

• If the search-string contains embedded spaces, enclose the string in quotes

� Example to find first IMS 08 (Application Start) log record after our current time position

Enter Find command





� Finding values within the browse screen view– FIND|F search-string NEXT | PREV | FIRST | LAST PREFIX | SUFFIX | WORD

• If the search-string contains embedded spaces, enclose the string in quotes

� Example to find first IMS 08 (Application Start) log record after our current time position– Results…..

IMS Application

Start




Navigating Through a Log File in IMS PI, Remembering Position

� Remembering file position using labels– Assign the label by entering it in the command line.

• The label, prefixed by a '.‘ (period), consists of 1 to 7 characters

– Re-establish position to that label using the LOCATE .label command

Assign the label

Label assigned

msg





� Remembering file position using labels– Re-establish position to your label using the LOCATE .label command

– Example:• Note: Labels are lost after leaving the IMS PI session and must be re-established.

Locate your label

Label located msg





� Remembering file position using a screen print and “Time of Day”– Another good way to remember the position of a log record is to copy the screen image

into a notepad.

– Later, use point and shoot on “Time of Day” to return to the same time and record• Note: This technique works across IMS PI sessions

Type the time of desired recordCursor here,hit

enter




Navigating Through a Log File in IMS PI, Merging Log Files

� Tracking a transaction in a sysplex environment will require merging log datasets

from different IMSPlex– Return to the IMS PI Primary Option Menu

– Select Option 1 (Process)





� Use the R (Repeat) or I (Insert) line command to create a new line to enter your

additional log file from your IMSPlex partner system.





� Ener the new IMSPlex member log file on the new line

� Use the WW (Merge Timeslicing) or SS (Merge Select) to select the log files

Type the additional log file name





� Specify the time slice– Make sure the time slice is complete for all files, if possible

Enter time duration (5 minutes in this case)

Coverage complete for both log files





� Specify the time slice– Had we specified a 13 minute time slice, IMS PI would warn us that the file merge is

partial for one of the log files

Requested duration of 13 minutes

Coverage is partial for one log file





� Specify the time slice– Make sure the time slice is complete for all files, if possible

– Select the specified time slice to merge and browse specified log files

Re-specify time duration to 5 minutes

Coverage complete for both log files

Select to merge log records for the specified interval and

enter Browse mode





� View the merged IMSPlex log files from the IMS PI Browse– Position to the desired application schedule record using time of day point and shoot

Enter saved record time of day

Cursor here, hit enter





Specified record found

� Time of Day point and shoot brings us back to our application schedule log record

inside our merged log files




Section 6:

Examining IMS Application Log Record Flows




Examining IMS Application Log Record Flows – Application Records

� IMS Log records related to application processing:

IMS log Record code and description

– 08 An application program was scheduled

– 07 An application program terminated

– 09 SB (Sequential Buffering) statistics

– 0A08 A CPI (Common Programming Interface) communications

driven application program was scheduled

– 0A07 A CPI communications driven application program

terminated

– 11 A conversational program started

– 12 A conversational program terminated

– 41 A batch program or BMP program issued a checkpoint

– 55 External subsystem information (DB2, MQSeries, etc.)

– 56xx IMS external subsystem support recovery




Examining IMS Application Log Record Flows – Transaction Records

� IMS Log records related to transaction processing:


– 01 IMS message placed on message queue– 03 IMS output message destined for lterm or transaction

– 30 Message prefix was changed– 31 A GET UNIQUE was issued for a message– 32 A message was rejected; it was presumed to have been

the cause of an application program ABEND

– 33 The queue manager released a record– 34 A message was cancelled– 35 A message was enqueued or re-enqueued

– 36 A message was dequeued or saved or deleted

– 37 Records marked as NO INPUT and NO OUTPUT are written bythe syncpoint coordinator when all resource managers have

completed Phase 1

– 38 An input message was put back on the input queue when the application abnormally terminated

– 39 The output queue was freed during cleanup processing of a RELEASE call




Examining IMS Application Log Record Flows – Database Records

� IMS Log records related to full function database processing


– 20 A database was opened

– 21 A database was closed

– 24 The buffer handler detected an I/O error

– 25 An EEQE was created or deleted

– 26 An I/O toleration buffer was created

– 27 A data set was extended

– 49 RSR Tracking

– 50 The database was updated. This log record contains the

new data on an insert and update call as well as the

old data and FSE updates on a delete call

– 57 Database updates in an RSR environment




Examining IMS Application Log Record Flows – IMSPI Tracking

� Locating all the application records related to a single IMS application schedule

would require a great deal of knowledge about IMS log record contents and interdependencies

� Tying these records together would also be difficult due to the volume of simultaneous transaction processing that typically takes place on and IMS production region.

� IMS PI provides a transaction/application tracking function that enables you to

view all records associated with a given transaction or application instance




Examining IMS Application Log Record Flows – IMSPI Tracking

� Track Schedule - Tracks all the application related log records associate witha single application schedule from the start of the application

to the end and includes the original input message

– Scheduling Tracking is invoked when a TX or TU is entered on the IMS 08 (application

start) or 07 (application terminate) record• TX ignores quick reschedules and displays the entire OASN program schedule sequence; for

Example 08-07-08-07-08-07 where the middle 07-08 records are quick reschedules.• TU strictly honors the 08-07 boundaries, regardless of whether it is a quick reschedule of not.

� Track Transaction TX - Tracks the entire lifecycle of the transaction/application,

including all its program switches – Enter from any transaction or application related record other than 08 or 07

� Track Unit of Recovery TU - Tracks the transaction/application events associated

with a single unit of recovery – Enter from any application or transaction related record other than 08 or 07




Examining IMS Application Log Record Flows – Track Schedule

� Enter TX in the line command area of the log record browse for an IMS 08

(application start) or 07 (application terminate) log record

Transaction tracking for entire application

schedule





� The Browse panel now shows the extended tracking for all log records associated

with the instance of the application associated with the IMS 08 record





� You will need to scroll backwards to find the top of the schedule sequence

including the initial input message creation





� Change the view by using PF 11 to show the compact time view of the records





� Depending upon the complexity of the application, including number of DB updates, number of MSG queue GUs, number of message switches, this tracking view could extend for many pages





� You may drill down into any record to see a complete formatting of the record with

extended field definitions – Enter “S” in the line command to format a record





� This is the formatted record view of the IMS 07 log record





� Note the detailed DLI call counts given in the accounting portion of the IMS 07 log

record





� Extended field definitions are available by placing the cursor under a field label

and hitting the enter key
















Examining IMS Application Log Record Flows – Filtering Records

� The IMS PI Filter function can be used to facilitate use of the IMS PI Tracking by

only showing IMS 08 (application start) or 07 (application terminate) records

Use the Filter command Or move cursor to Filter menu option and hit enter




Examing IMS Application Log Record Flows – Filtering Records

� Enter the filter information for the record type you’d like to see in browse mode

� Example: Enter Log type: IMS Code: 07 and hit enter (Selects application terminate records)

� Then hit PF3 to activate the filter

Indicate IMS log typeEnter the desired log record

code




Examing IMS Application Log Record Flows – Filtering Records

� Example: Enter Log type: IMS Code: 07 to see the application terminate records– Only IMS Type 07 (application terminate) records are presented

– Use “FIND ABEND PREFIX” command to find applications that abended

Enter “F ABEND PREFIX”




Examing IMS Application Log Record Flows – Use TRACK Schedule TU or TX for ABEND� Once you’ve identified an abended transaction, you can see all the IMS records

produced by the application program leading to the ABEND

Enter “TU” or “TX” Schedule tracking

ABEND code presented here




Examing IMS Application Log Record Flows – Use TRACK Schedule TU for ABEND� Once you’ve identified an abended transaction, you can see all the IMS records

produced by the application program leading to the ABEND – Hit PF7 to see the initial input message for this application schedule

Schedule UOR tracking active




Examing IMS Application Log Record Flows – Use TRACK Schedule TU for ABEND� The initial input log record could be formatted by selecting with an “S” in the line

command field to see the details of the input message

Initial input message

s




Examing IMS Application Log Record Flows – Use TRACK Schedule TU for ABEND� Formatted initial input message text




Examing IMS Application Log Record Flows – Use TRACK Schedule TU for ABEND� Upon return to the Browse panel, you can change the view to the compact view to

obtain a condensed summary of the log records involved in the application abend




Examing IMS Application Log Record Flows – Use TRACK Schedule TU for ABEND� To assist with performance related issues, use the “Elapsed Time” view as show

below to show processing delays. – Enter “E” in the line command




Examing IMS Application Log Record Flows – Use TRACK Schedule TU for ABEND

� The elapsed time between records is shown to the right

Elapsed time between records




Section 7:

Uncovering the Hidden Information in Your IMS Application Using IMS Problem

Investigator - Summary




Uncovering the Hidden Information in Your IMS Application Using IMS Problem Investigator - Summary

� We’ve shared key techniques to get you up and running in the IMS Problem Investigator environment. You will now be able to start using IMS PI with IMS log data to display and interpret key application processing flows within IMS

� Section 1– IMS PI Overview

� Section 2 – Starting the IMS PI Dialog Panels

� Section 3– IMS PI ISPF Dialog Panel Overview

� Section 4– IMS PI Setup

� Section 5– Navigating Through a Log File in IMS PI

� Section 6– Examining IMS Application Log Record Flows

� Section 7– Uncovering the Hidden Information in Your IMS Application Using IMS Problem Investigator -

Summary

Technology

IMS Problem Investigator