*
Migrate changes to back in to production
Promote your changes to production
Schedule and re-run the cycle
After having completed this unit on Production Support/Application
Testing/Software Defect and IBM Mainframe COBOL ABEND Research, you
should now be able to:
Define the steps in a generalized methodology of ABEND
resolution
List the various sources of ABEND inputs, including:
PD Tools documents
Fault Analyzer reports
Module 4 - Development of Local COBOL Programs
*
ABEND Completion Codes and some typical causes
While there is a wide variety of reasons for ABEND conditions
("WHYs") in production systems, it is possible (and useful) to
categorize and organize HOW certain conditions often lead to
certain types of ABEND completion codes - in order to expedite or
streamline your analysis and research (an 80/20 approach to
analysis).
The following information on a few common z/OS ABEND completion
codes, and the conditions which generated them is included for you
to make effective use of PD Tools/Fault Analyzer listings and the
above debugging, research and analysis process.
Notes:
This information is available to some degree within the RDz product
in the Lookup View
There is another source of ABEND/Debug information you might want
to take a peek at:
Reason(s)
S001-0: Conflict between record length specifications (program vs.
JCL vs. dataset label)
S001-2: Damaged storage media or hardware error
S001-3: Fatal QSAM error
S001-5: Attempt to read past end-of-file
Instructions:
S001-3: Internal z/OS problem
S001-4: Forgot to code BLOCK CONTAINS 0 RECORDS in FD (default
Block is 1)
S001-5: Logic error (either forgot to close file, or
end-of-file-switch not set, overwritten or ignored)
Tools to debug/RDz equivalent return codes:
S001-0: Cannot occur on RDz with Local ASCII/Windows (Line
Sequential) files
S001-2: Norton Utilities – if on Workstation/COBOL
application
S001-4: Cannot occur on Workstation/COBOL (no blocking for Line
Sequential files)
S001-5: Logic error: Use RDz's Perform Hierarchy or RAA's Program
Flow Diagram to detect
Dynamic:
S001-0: During Debug – set a Watch Monitor on the 01 record
S001-2: Need to have PC/IT technician investigate (may need to
reformat disk)
S001-4: Always code BLOCK CONTAINS 0
*
Reason(s)
S013-10: Dummy data set needs buffer space; specify BLKSIZE in
JCL
S013-14: DD statement must specify a PDS
S013-18: PDS member not found
S013-1C: I/O error search PDS directory
S013-20: Block size is not a multiple of the LRECL
S013-34: LRECL is incorrect
S013-50: Tried to open a printer for Input of I/O
S013-60: Block size not equal to LRECL for unblocked file
S013-64: Attempted to Dummy out indexed or relative file
S013-68: Block size > 32K
S013-A8: Invalid RECFM for SYSIN/SYSOUT
S013-D0: Attempted to define PDS with RECFM FBS or FS
S013-E4: Attempted to concatenate > 16 PDSs
Instructions:
Frequent Coding Causes:
Most of these ABENDs occur running und z/OS (some may not even
occur under z/OS, although older modules running OSVS or VS COBOL
II code that have not been recompiled can produce them). Most are
due JCL/COBOL FD inconsistencies.
Tools to debug:
Static
*
- SYSOUT DD statement missing
- The value in an AFTER ADVANCING clause is < 0 or > 99
- And Index or Subscript is out of range
- An I/O verb was issued against an unopened dataset
Instructions:
OPEN, CLOSE, READ, WRITE, Table handling routines
Note also that during Debug SYSOUT-DISPLAYs are written to the
"console", recall how to view
Frequent Coding Causes:
- Incorrect logic in setting AFTER ADVANCING variable (or failure
to understand 0-99 limits)
- Incorrect logic in table handling code, or number of table
entries has outgrown PIC of variable (e.g. PIC 99, but 100
entries)
Tools to debug:
Static
SYSOUT problem: Open multiple windows on RAA Batch Job Diagram and
program Environment Division - SELECT ASSIGN.
In RAA: Double-click on GO TO verb, or PERFORM chain, or paragraph
name.
In RDz: Select Paragraph name/Perform chain and select: Open
Declaration
Dynamic:
Set Watch Breakpoint and Monitor on table index or AFTER ADVANCING
variable.
*
Reason(s)
The program is attempting to access a memory address that is not
within the applications z/OS Address Space
Frequent Coding Causes:
- JCL DD statement is missing or incorrectly coded
- Incorrect logic in table handling code (referencing a table
subscript < 1 or > max-table-size),
- Number of table entries has outgrown PIC of variable (i.e. PIC
99, but 100 entries).
- In IMS/TM systems, an MFS LL (length) field value is smaller than
the actual input MSG length.
Tools to debug:
Static
- DD statement problem: Open multiple windows on RAA Batch Job
Diagram and program Environment Division - SELECT ASSIGN
- IMS LL problem: Analyze through multiple Edit Windows (same
solution as DD).
- Incorrect linkage problem:
- Open multiple windows on CALLing and CALLed programs - verify
linkage declarations.
Dynamic
- Set conditional advanced break point on subscript (i.e. IDX <
100).
RAA - Utilize Run-Unit View to view all modules in Debug run-stream
(both active and inactive).
Incorrect logic
*
Instructions:
Arithmetic, IF-THEN-ELSE, MOVE (if receiving field is numeric -
)
Note: RDz will also S0C7 if sending field is numeric and contains
non-numeric (MOVE pic9field TO picXfield)
Frequent Coding Causes:
- Incorrectly initialized, or uninitialized variable
- Missing or incorrect data edit
- 01 to 01 level MOVE if sending field is shorter than receiving
field
- Move of Zeros to Group-level numeric fields
- MOVE CORRESPONDING incorrect
Tools to debug:
RAA report with options data selector on MOD or ALL
Dynamic
Run through to S0C7.
Solutions:
*
Instructions:
- Incorrectly initialized, or un-initialized variable
- Missing or incorrect data edits (i.e. failed to check divisor for
zero value)
Tools to debug:
Static
RAA report on all DIVIDE and COMPUTE instructions – or using RDz
double-click on these verbs and select Filter from the Context
Menu
Dynamic
Locate to field definitions of the offending fields
Solution:
IF divisor > ZERO
*
Reason:
Timeout due to program logic caught in "loop" through instruction
set with no exit.
Note: RDz Workstation binaries (Run-Time System) does not honor the
concept of timeout.
Frequent Coding Causes:
- Invalid end-of-file logic
- End-of-file switch overlaid
- Subscript not large enough
- Perform Thru wrong Exit
- PERFORM UNTIL "End-Of-File", but not performing "READ" routine to
reach EOF condition
Tools to debug:
RAA Display Perform Thru
Analyze counts (color)
Set an Advanced Break Point - Conditional on count
Solution:
From within Debug, use RAAi to identify logic which could cause
looping.
Select and click on PERFORM THRU, PERFORM UNTIL, GO TO.
Place break points on potential error lines.
*
Reason:
CALL made to program which could not be located along normal search
path
(STEPLIB top-to-bottom, JOBLIB top-to-bottom, LINKPACK)
Instructions:
Frequent Coding Causes:
- Module name spelled incorrectly
- STEPLIB does not contain load library with module
- I/O error occurred while z/OS searched the directory of the
library
Tools to debug:
Dynamic
Set Program Advanced Break Point (Entry) to set program break
before entry to system.
Solution:
D37: Primary space exceeded, no secondary extents defined.
E37: Primary and secondary extents full. In TSO, PDS directory
needs compress.
E37-04: Disk volume table of contents (VTOC) is full.
Reason:
MVS could not find space for output WRITE.
Note: RDz Local COBOL Run-Time System does not honor the concept of
Primary/Secondary space "Out of space" condition will not occur
until disk actually fills up
Instructions:
WRITE
- Not enough space initially allocated to output file(s).
- (more likely) Logic error - program in (infinite) loop writing
output file(s) - see S222/S322 reasons.
Tools to debug:
Static - if (unlikely as this may be) you're debugging locally (on
your Workstation) you can find the out-of-space file and statistics
on it as follows:.
Do directory list on the file you are writing to:
Go to DOS Window
Type "DIR fname.ext"
On the host the JCL will show the DDNAME and z/OS filespec of the
dataset in question
Dynamic
Set an advanced conditional break point to break on a certain
number on iterations
See S222/S322 reasons and solutions
Also, set break point on file WRITE statements
*
Database Abends
DB2 application access rarely abends, they "die gracefully" due to
return code processing.
DB2:
SQLCODE
A unique integer which describes DB2's reaction to your
request.
SQLCA
A large 01 block, which contains several other fields pertinent to
debugging, particularly the SQLWARNs.
Suggestion:
Set Line Breakpoint and/or Variable Monitor on SQLCODE and other
key feedback areas
- or -
Double-click on field, Ctrl/F3
Notes on UDB Offloading and Database Abends
UDB handles SQLCODE processing in a manner consistent with DB2.
However, it should be noted that UDB does not handle SQLWARNO-n and
the SQLCA (SQLERRD, etc.) fields compatibly.
DB2 ABENDs
If DB2 tables have not been defined, you may get a -204 return code
on your SQL statement.
*
Topic Considerations
Note: In this topic you will learn how to use Fault Analyzer to
debug an ABEND. The screen captures all describe connecting to a
public z/OS machine that IBM makes available – during
classes.
If you are taking this course through standard IBM services
delivery you should be able to use the properties (I/P address,
port#s, etc.), logon IDs and passwords that your instructor
provides you with.
But you may also be taking this course standalone – and in that
case, you will need to speak to your company's Systems Programming
staff to learn how to connect and logon.
It goes without saying that the actual file names in the screen
captures of mainframe libraries and datasets will vary. So you
should focus on the process and steps and "how to" – and don't be
perplexed at differences in screen captures.
*
Work with ABEND analysis reports created by IBM Fault
Analyzer
Browse Report and Mini-Dump pages
Retrieve various Fault Analyzer view information
Browse and search ABEND codes
Use the various productivity features in the Fault Analyzer
perspective
Note:
This presentation and Lab is not a comprehensive IBM Fault Analyzer
unit. It is only intended to introduce you to the RDz/Fault
Analyzer interface. It is assumed that you are already working with
IBM Fault Analyzer on the z/OS host.
Connecting to a z/OS host from the workbench
Module 4 - Development of Local COBOL Programs
*
Face facts:
ABEND research ("shooting a dumps") is not how you want to spend
your week-nights
The good news? You don't have to.
Fault Analyzer:
Variables and variable values
Data and buffers
Gives you a serious head start on the What/Where and How of ABEND
debugging
The z/OS-based debugging engine is provided by the IBM Debug Tool
for z/OS product. It runs on the z/OS.
"Perspectives" - Recall that a "Perspective" is a convenient
grouping for a collection of views organized around a given role or
task
So far in this class, you've used the:
z/OS Projects Perspective – to develop local COBOL
applications
Data Perspective – to develop local COBOL applications
*
What is Fault Analyzer?
IBM Fault Analyzer is a tool that helps you determine the cause of
an application ABEND. It is used to determine:
What happened, how it happened, what program, what line/statement,
which variables, what files, were involved, etc.
Fault Analyzer provides the necessary information to perform root
cause analysis on an application ABEND.
You do not have to interpret low-level, system dumps and wade
through HEX data. Information is presented in report format
IBM Fault Analyzer for z/OS® gathers information about an
application and the surrounding environment at the time of an
abnormal end (ABEND), providing you with the valuable information
you need to work through
After analyzing information about your application and its
environment, Fault Analyzer generates an analysis report
(IDIREPORT) that describes the problem in terms of
application/program statements and variables
Connecting to a z/OS host from the workbench
Module 4 - Development of Local COBOL Programs
*
Fault Analyzer for z/OS Overview
Fault Analyzer is part of IBM’s Problem Determination Tool family
of products:
http://www-01.ibm.com/software/awdtools/deployment/
It runs in both test and production with very little overhead,
and:
Provides support for analyzing***:
IMS and CICS® online application and system failures - with
debugging facilities for all of the IBM-mainstream online files and
databases
IMS-DL/I, DB2, VSAM, IDMS, etc.
WebSphere® Application Server for z/OS system failures
WebSphere MQ application failures
Batch (QSAM/VSAM/DB2®) application failures
Helps you analyze failures when they occur or reanalyze them after
the fact
Expands error messages and codes that apply to your failure with
interactive reanalysis and includes a feature for using
application-specific messages and codes to supplement those
supplied by IBM
Creates a fault history file with an interactive display that helps
you track and manage application failures
Starts automatically when an application fails, eliminating the
need to recompile programs or change the job control language
(JCL)
Integrates with Rational Developer for System z enables developers
to diagnose application problems without changing user
interface
For Fault Analyzer product information, see:
http://www-01.ibm.com/software/awdtools/faultanalyzer/
***See Notes
Module 4 - Development of Local COBOL Programs
Fault Analyzer for z/OS supports applications running under z/OS
and OS/390 in the following environments:
COBOL
PL/I
Assembler
C/C++
Java
*
Fault Analyzer – Operational Process
The purpose of Fault Analyzer (FA) is to determine the cause of any
ABENDs in an application program.
You do not have to read through application or system dumps,
because the product has the ability to isolate the exact
instruction that caused a particular error.
The Analysis engine provides automatic analysis when the
application fails.
When an ABEND occurs, Fault Analyzer activates automatically, and
then records details in a fault history file (see screen capture
below)
Fault History files contain information about the faults analyzed
by Fault Analyzer for z/OS.
Using Fault History files, re-analysis is available when real-time
ABEND analysis isn’t enough (you can extract additional information
in batch or interactive mode)
ABEND happens
Salient details (points of interest) written and stored
Connecting to a z/OS host from the workbench
Module 4 - Development of Local COBOL Programs
*
Fault Analyzer
An IDIREPORT is produced that provides ABEND analysis
information
*
Reviewing ABENDs in the Fault Analyzer Perspective
Besides the IDIREPORT, you may also wish to use RDz's Fault
Analyzer perspective, to analyze and debug an ABEND
situation.
To do that you'll need to:
Switch to the Fault Analyzer perspective in RDz
Specify the history file to connect with, that populates a Default
ABEND view with failed online and batch job IDIREPORTs and other
outputs
Learn how to navigate the Fault Analyzer perspective, to make use
of the information contained therein
The next slides contain the step details
*
Steps:
Connecting to a z/OS host from the workbench
Module 4 - Development of Local COBOL Programs
*
2. Enter: FAULTANL.<version>.HIST
Ex. FAULTANL.V10R1.HIST
Module 4 - Development of Local COBOL Programs
*
Content Area shows Fault Analyzer IDIREPORT – detailed information
on ABEND
IDIREPORT
Additional Fault Analyzer views exist for other ABEND documentation
including history data
Connecting to a z/OS host from the workbench
Module 4 - Development of Local COBOL Programs
*
From the Default tab
Scroll up and down – to find a particular ABEND
Double-click an ABEND history file, to bring up its IDIREPORT and
other stats
Sort the list by any of the column headings
Can also work with options of the Context Menu – with each ABEND
entry
Connecting to a z/OS host from the workbench
Module 4 - Development of Local COBOL Programs
*
Fault Analyzer – Browse Report – S0CB
The IDIREPORT presents a formatted, high-level summary of the
points of interest necessary to debug ABEND conditions in your
application.
Specifically, to answer the questions:
What happened?
How did it happen?
Program line where the S0CB occurred
Module 4 - Development of Local COBOL Programs
*
Fault Analyzer – Browse Report – S0C4
Here's an example of an IDIREPORT which shows that RPT-REC is
"not addressable"
…which is a euphemism for: "There's something really wrong with the
file/FD/JCL DD connection"
Connecting to a z/OS host from the workbench
Module 4 - Development of Local COBOL Programs
*
Fault Analyzer – Browse Report – S0C7
The IDIREPORT and supporting text varies from ABEND to ABEND
depending on:
Type of ABEND
Run-time platform
Note:
CUST-ACCT-BALANCE's value is shown in hex (as it represents invalid
data)
Connecting to a z/OS host from the workbench
Module 4 - Development of Local COBOL Programs
*
Fault Analyzer – Browse Report – S0C9
A S0C9 is like a S0CB (divide by zero) except that a S0C9 occurs
because of an excessively large fixed-point number obtained as the
result of a decimal division operation
Connecting to a z/OS host from the workbench
Module 4 - Development of Local COBOL Programs
*
Here is information on an IMS (DL/I) S0C1 ABEND
Connecting to a z/OS host from the workbench
Module 4 - Development of Local COBOL Programs
*
IDIDREPORT information on a module-not-found (S806) ABEND
Most likely SAM2 is either a typo, or
the program did not successfully compile/link into the Load
Module
Connecting to a z/OS host from the workbench
Module 4 - Development of Local COBOL Programs
*
The Lookup view shows a great deal of background information
on:
ABEND codes
DB2 SQLCODE
etc.
You can use the view, or double-click on the ABEND code shown in
the IDIREPORT
Connecting to a z/OS host from the workbench
Module 4 - Development of Local COBOL Programs
*
Fault Analyzer Integration – Mini-Dump Reading 1 of 2
Fault Analyzer also provides for the reading/browsing of System
Dump data – in Hex/Character format.
Select an ABEND
Scroll through the dump – Issue navigation commands: Show nnn, +nn,
etc.
Connecting to a z/OS host from the workbench
Module 4 - Development of Local COBOL Programs
*
1. Right-click over the storage address
2. Add your note (click OK)
3. Your note becomes highlighted text inside the dump
Connecting to a z/OS host from the workbench
Module 4 - Development of Local COBOL Programs
*
1
2
3
Define views which contain a set of history files.
Display fault entries from all history files as if they were in one
dataset.
Connecting to a z/OS host from the workbench
Module 4 - Development of Local COBOL Programs
*
A History File
Connecting to a z/OS host from the workbench
Module 4 - Development of Local COBOL Programs
*
Available Line Commands
Batch and/or Online may be in the same list
Connecting to a z/OS host from the workbench
Module 4 - Development of Local COBOL Programs
*
What RDz Perspective is used to view Fault Analyzer reports?
How does RDz obtain Fault Analyzer Information? Where does the
information originate?
RDz Fault Analyzer interface has a Lookup View. What is it used
for?
How can you jump to the program statement where the ABEND occurred
with the RDz Fault Analyzer interface?
Connecting to a z/OS host from the workbench
Module 4 - Development of Local COBOL Programs
The Fault Analyzer Perspective
Fault History File (Add New, Fault History file). It originates
when a program abend occurs on the host. The complied source is
saved in PDS and matched against the Fault History file to obtain
the program statement (line number) where the abend occurs.
The Lookup View is used to search / look up the abend code. For
example, what is a SOC4, SOC7, etc.....
If the correct LANGX file was available during the real-time
analysis, you can click on the line number to jump to the source of
the abending program.
Checkpoint Solutions:
2. In the Remote Systems view, under New Connection:
i. Click the ‘+’ in front of z/OS …
ii. Right-click z/OS … then select New Connection from the pop-up
dialog
*
Having completed this unit, you should now be able to:
Work with ABEND analysis reports created by IBM Fault
Analyzer
Browse Report and Mini-Dump pages
Retrieve various Fault Analyzer view information
Browse and search ABEND codes
Use the various productivity features in the Fault Analyzer
perspective
Connecting to a z/OS host from the workbench
Module 4 - Development of Local COBOL Programs
*
In this workshop you are going to:
Copy a several datasets from your instructor's zServerOS TSO ID to
your ID
Details on the next slide
Modify JCL dataset names (and high-level qualifiers) to match your
Sandbox ID
Compile a program named: HOSPCALC – which contains different types
of COBOL ABENDs generated from invalid COBOL logic in different
parts of the program
Run HOSPCALC (it will ABEND) – and from the Fault Analyzer
IDIREPORT:
Find the error in the COBOL source, and use the IDIREPORT ABEND
analysis data to fix the error
After you've solved the problem, you will save your edits, and
re-compile HOSPCALC. Then run the program until you either get the
next ABEND … or get a zero return code
HOSPCALC
Module 4 - Development of Local COBOL Programs
*
If you are working on the Sandbox, do the following:
In Remote Systems, create a new MVS Filter named: RDZFA for:
DDS0001.POT.*
Expand the RDZA filter, and:
Copy DDS0001.POT.COBOL member: HOSPCALC to <EM4Zxx>.POT.COBOL
PDS
Copy DDS0001.POT.JCL member: HOSPCALC to <EM4Zxx>.POT.JCL
PDS
This is the compile JCL for HOSPCALC
Copy DDS0001.POT.JCL member: HOSPCRUN to <EM4Zxx>.POT.JCL
PDS
This is the run JCL, for HOSPCALC
Right-click over DDS0001.POT.DATA and use Allocate Like to create a
new PDS named: <yourEM4Zxx>.POT.DATA - with the same dataset
characteristics
Example with ID: EM4Z07
Module 4 - Development of Local COBOL Programs
*
Edit HOSPCALC.
Save your edits
Save your edits
5. Submit your modified HOSPCALC JCL. When the job finishes, check
the results in your JES My Jobs filter (it should compile
successfully – but check that the LKED step ran)
6. Submit your modified HOSPCRUN JCL. The job should ABEND. In JES
- My Jobs, view the IDIREPRT step, to see the specific system
completion code and surrounding Fault Analyzer ABEND analysis
information
Note that HOSPCALC contains four separate ABEND errors – that you
will hit, one at a time: an 0C7, an 0CB, an 0C4 and a combined
0C7/0C4 ABEND (Sounds strange ... Essentially Fault Analyzer will
find two different problems on one statement)
The comments and descriptions in the IDIREPRT combined with your
COBOL knowledge should be sufficient to debug and fix these
problems, but feel free to elicit help from your instructor if
you're stuck.
7. Fix the COBOL error that caused the ABEND. Then repeat steps 5
and 6 until you find and fix all of the ABENDs in HOSPCALC. You're
done when HOSPCRUN finishes with a zero return code.
Connecting to a z/OS host from the workbench
Module 4 - Development of Local COBOL Programs
However – If you need any help, from within edit on HOSPCALC, do a
Find ALL on: *RDZ