IBM - SAP Java Monitoring & Tuning Guide

IBM Americas Advanced Technical Support

© 2007 International Business Machines, Inc.

IIBBMM SSAAPP TTeecchhnniiccaall BBrriieeff

IIBBMM -- SSAAPP JJaavvaa MMoonniittoorriinngg && TTuunniinngg GGuuiiddee

DDaammiirr RRuubbiicc

IIBBMM AAmmeerriiccaass AAddvvaanncceedd TTeecchhnniiccaall SSuuppppoorrtt SSAAPP && OOrraaccllee SSoolluuttiioonnss

VVeerrssiioonn:: 11..00 DDaattee:: DDeecceemmbbeerr 2211,, 22000077



Page 2

1. PREFACE ............................................................................................................................................. 3

1.1 ACKNOWLEDGEMENTS .................................................................................................................... 3 1.2 DISCLAIMERS .................................................................................................................................. 3 1.3 COPYRIGHTS ................................................................................................................................... 3 1.4 FEEDBACK ...................................................................................................................................... 3 1.5 VERSION UPDATES .......................................................................................................................... 3 1.6 WHO SHOULD READ THIS DOCUMENT .............................................................................................. 4

2. JAVA TECHNOLOGY IN THE SAP APPLICATION SERVER ................................................. 5 2.1. SAP J2EE ENGINE SYSTEM ACHITECTURE ..................................................................................... 5 2.2. JAVA CLUSTER ACHITECTURE ...................................................................................................... 10

3. SAP JAVA SERVER ADMINISTRATION .................................................................................... 13

3.1. SAP J2EE ENGINE ADMINISTRATION TOOLS ................................................................................ 13 3.2. SOFTWARE DEPLOYMENT MANAGER (SDM) ............................................................................... 19

4. SAP JAVA SERVER MONITORING............................................................................................. 21 4.1. TECHNIQUES FOR PROBLEM DETECTION ....................................................................................... 22

4.1.1 Monitoring of the Log and Trace files .................................................................................... 22 4.1.2 Availability Monitoring of SAP Java engine and Java based applications (GRMG) ............. 26 4.1.3 Distributed Statistics Records (DSRs) .................................................................................... 28 4.1.4 State Monitoring ..................................................................................................................... 31

4.2. TECHNIQUES FOR PROBLEM ANALYSIS ......................................................................................... 34 4.2.1 Performance Trace ................................................................................................................. 34 4.2.2 Single Activity Trace / JARM .................................................................................................. 37 4.2.3 Single Activity Trace ............................................................................................................... 40 4.2.4 SQL Trace ............................................................................................................................... 42 4.2.5 Application Trace.................................................................................................................... 44 4.2.6 Conclusion .............................................................................................................................. 47

5. IBM JAVA TECHNOLOGY FOR SAP .......................................................................................... 49 5.1 IBM JVM GARBAGE COLLECTION ............................................................................................... 50 5.2 IBM JVM PARAMETERS ............................................................................................................... 51 5.3 MONITORING THE JVM ACTIVITY ................................................................................................. 51

PUBLICATIONS: ...................................................................................................................................... 54



Page 3

1. Preface

1.1 Acknowledgements I would like to thank Mark Gordon and Jim Dilley for all their help regarding this and many other Oracle & SAP & pSeries performance-related subjects.

1.2 Disclaimers IBM has not formally reviewed this paper. While effort has been made to verify the information, this paper may contain errors. IBM makes no warranties or representations with respect to the content hereof and specifically disclaim any implied warranties of merchantability or fitness for any particular purpose. IBM assumes no responsibility for any errors that may appear in this document. The information contained in this document is subject to change without any notice. IBM reserves the right to make any such changes without obligation to notify any person of such revision or changes. IBM makes no commitment to keep the information contained herein up to date.

1.3 Copyrights SAP and R/3 are copyrights of SAP A.G. RS/6000, pSeries, and AIX are copyrights of IBM Corporation. Oracle is a copyright of Oracle Corporation.

1.4 Feedback Please send comments or suggestions for changes to [email protected].

1.5 Version Updates • Version 1.0 – initial version



Page 4

1.6 Who should read this document This paper is primarily intended for pSeries Sales Specialists who would like to know more about the new SAP Technology Application Framework – NetWeaver 2004s. The goal of this paper is to provide the audience with material that will help them understand the most important elements of the SAP NetWeaver Application Framework, and how to combine these elements with the strengths of the IBM pSeries platform. The SAP NetWeaver Application Platform presents a new framework consisting of several different elements. This paper will focus on the J2EE engine, which is the central and most interesting element of the latest SAP Java announcement. This paper will provide the reader with an overview of the most important features of the J2EE engine. The J2EE engine also presents opportunities for extension into multiple areas. I would be happy to accept suggestions regarding exploring these additional areas for development.



Page 5

2. Java Technology in the SAP Application Server

2.1. SAP J2EE Engine System Achitecture As a system, the J2EE Engine consists of three logical layers: 1. Java Enterprise Runtime – comprised of low-level subsystems that provide functions such as class

loading, cluster communication, persistent configuration data management, etc. 2. J2EE Engine components – consists of interfaces, libraries and services components that provide

various runtime functions and programming APIs 3. Applications – refers to the applications that are deployed and run on the J2EE Engine

The J2EE Engine architecture is based on the following general rule: components from the higher level can use components from lower level; whereas components from the lower level are not aware of the APIs of the components from the higher level and therefore cannot use them. This rule is reflected by the starting order of the system modules: runtime is first, then services (libraries are loaded, and interfaces resolved at this phase), and lastly, applications. The system is considered functional when all runtime managers and core services components are started in the proper order. Components from the higher level use a set of defined APIs to utilize the functions of the components from the lower level. The J2EE Engine components use the Framework APIs to connect to the SAP Java Enterprise Runtime. Applications use the J2EE Engine components by the APIs that are defined by J2EE 1.3 Specification (and supporting specifications) and SAP-proprietary APIs.



Page 6

Figure 1-1 SAP J2EE Engine Architecture

• SAP Java Enterprise Runtime

The Java Enterprise Runtime provides the core functions of the system. It is built of several low-level subsystems called Managers. These are:

• Log Manager – manages the process of logging system events • Ports Manager – manages the process of opening various communication ports on the system • Pool Manager – manages the process of pooling Java objects (this does not refer to pooling JDBC

connections) • Application Thread Manager – handles threads in which client applications’ source code is

executed • Thread Manager – handles threads in which the J2EE Engine system operations are executed • Connections Manipulator Manager – manages the process of opening TCP/IP sockets from the

client to the J2EE Engine • Locking Manager – handles locking as interface to the Enqueue Service • Configuration Manager – manages the process of storing and reading persistent configuration

data (such as properties and deployment data) to and from a relational database



Page 7

• Class-loading Manager – controls activities for registering and unregistering of loaders and references between them, as well as for accessing loaders referenced by another loader. This is necessary for locating and loading Java classes

• Licensing Manager – handles SAP licenses • Cluster Manager – manages communication (in terms of cluster elements status and modules

available on it at runtime) between J2EE Engine cluster elements • Service Manager – represents a container of pluggable components. It provides runtime lifecycle

management of those components The Service Manager is the container of all J2EE Engine components of the system. Using the functions of other runtime subsystems, it loads and starts the components when the J2EE Engine starts up. This is why the Service Manager is the last runtime subsystem to be started. It also manages components at runtime, and stops them when the J2EE Engine is being stopped.

Figure 1-2 Service Manager



Page 8

The process of starting (loading) the J2EE Engine components is strictly defined. It is conducted exclusively by the Service Manager. It performs the following steps:

1. Reads the components persistent data from the database 2. Performs checks for data integrity. This includes checks for duplication of component names, for

availability and consistency of components deployment descriptors, and so on 3. Resolves components for which the previous steps are successfully completed 4. Checks whether all core components are resolved. If not, the Service Manager stops the entire

system 5. Creates the class loaders of the components and loads them 6. Starts service components 7. Performs a check on whether all core service components are started. If not, the Service Manager

stops all started components, unloads them and stops the whole system

• SAP J2EE Engine Components The J2EE Engine components build the second level of the system. Built on top of the runtime and able to communicate and use each other, these components form the complete system infrastructure to run both J2EE and SAP proprietary applications. Three types of J2EE Engine components are defined:

1. Interfaces – “contracts” that define how different components of the system work together. These components do not provide any runtime functions. At runtime, they provide the system with their name and classes (no objects). They are used by services components that provide their implementation. Interface components can also be implemented directly by applications.

2. Libraries – provide name, classes and objects to the system. These objects are created by the system when it loads the library, or when an object is first requested. Other library components or services components typically access those using static methods.

3. Services – more powerful than the other two types of components. They provide the system with their name, classes and runtime objects. The runtime objects are registered on the system once the components classes have been loaded. Service components can access and utilize functions of the runtime through the Framework API.



Page 9

• Applications The applications form the third level of the J2EE Engine architecture. The boundary between the applications level and the J2EE Engine components level is defined by the J2EE 1.3 APIs along with a few SAP proprietary APIs. Applications use these APIs to utilize the functions of the different types of components on the lower level in the system. An enterprise application consists of several types of application components. These components reside in different containers on the J2EE Engine as shown in Figure 1-3. Each of these containers provides runtime services and life cycle management for the application components.

Figure 1-3 SAP J2EE Engine Containers

Browser

mySAP component

J2EE Services Presentation Layer

Web Server J2EE Application Server

DB Server

Business Layer

JDBC JNDI JAAS

JMS JTA JCA

...

DB Server

EJB Server Legacy

EJB Container Session

Bean Entity Bean

Web Container Servlet Java

Server Page



Page 10

2.2. Java Cluster Achitecture A Java cluster installation consists of:

• One or more instances of the AS Java • The Central Services, which also form an instance • One or more databases

The dispatchers and servers can be split up among different physical servers. The Central Services (Message Service and Enqueue Service) are installed on a host that meets the requirements for high availability.

• Java Instance A Java instance is a unit in the AS Java cluster which can be individually started, stopped, and monitored. The cluster elements that form an instance run on one physical machine. It is also possible to run several instances on one physical machine. An instance is identified by the system ID (SID) and the instance number. Central Services are a special example of a Java instance. Another special instance is one that runs the SDM (Software Deployment Manager). This instance usually runs with the database and Central Services on the same machine, and is then indicated as the central instance. A Java instance normally is comprised of one Java Dispatcher and several server processes. A Java instance is started and stopped by the Java Startup and Control Framework. An HTTP request to execute a servlet runs through several layers in the J2EE engine. The Java dispatcher receives the request, selects a server process for the processing and establishes the connection from the client to the server.

Figure 1-4 Layers of the SAP J2EE Engine



Page 11

• Minimum Cluster Installation The following graphic shows the simplest installation of usage type AS Java. A system installed in this way can only process requests to Java applications.

Figure 1-5 Minimum Cluster Installation This minimal AS Java installation consists of:

• A Java central instance with a dispatcher, a server process, and Software Deployment Manager (SDM)

• The Central Services instance • The Database

A Java Instance consists of (with the exception of Central Services):

• a Java Dispatcher

• One or several server processes



Page 12

• Large Cluster Installation The graphic below shows a larger Java cluster installation. It consists of four instances, each of which has an instance number and can be started, stopped, and monitored separately.

Figure 1-6 Large Cluster Installation



Page 13

3. SAP Java Server Administration

3.1. SAP J2EE Engine Administration Tools The SAP J2EE Engine provides the following basic administration options:

o offline – enables the configuration and administration of J2EE Engine cluster elements when the server is not running

o online – enables the administration of J2EE Engine cluster elements at runtime o remote – enables configuration of the J2EE Engine from a distance; that is, the

administration tool runs on one machine and the server that must be managed is running on a different machine

These options are implemented in the administration tools for the J2EE Engine: Visual Administrator,

Shell Console Administrator, Config Tool and SAP Netweaver Administrator.

• Visual Administrator

SAP J2EE Engine Visual Administrator is a graphical user interface (GUI) that enables administration of the entire cluster, cluster elements, and modules running on them. It provides remote monitoring and management of managers, services, libraries, and interfaces working on each element in a single GUI. Visual Administrator enables:

o Obtaining general information about a service, manager, interface or library (for example, its name, group, and so on)

o Logging on to the SAP J2EE Engine via the Visual Administrator tool o Administrating and changing common properties or those specific to a service or manager o Configuring the global properties o Runtime administration and control



Page 14

Figure 1-7 Visual Administrator

To start the Visual Administrator, enter the following command: “/usr/sap/<SID>/<Instance Name>/j2ee/admin/go.sh” (or go.bat for Win based environment).

• Shell Console Administrator

The J2EE Engine Console Administrator is an alternative to the Visual Administrator. Unlike the Visual Administrator, it is not a GUI and the runtime control and administration is done using specific commands in the shell language. The commands are entered on the command line of the console where the cluster element is running. This tool enables monitoring of the processes started on the different elements of the cluster and provides an opportunity for prompt and adequate reaction whenever problems occur. The Console Administrator enables remote administration through Telnet clients or applets that simulate a Telnet client, as well as continuous deployment of applications and additional libraries. By default, the Telnet shell is opened on the dispatcher. Once connected, the user can access and use all shell commands available on the different J2EE Engine cluster elements. Use the LSC command to



Page 15

display all server components with their ID, component name, host, port, and type. The first component displayed is the current one. To pass over from one component to another, use the JUMP command and specify the ID of the target element. For example, executing jump 4001 enables the remote administration of a cluster element with ID 4001. The JUMP command is available for Telnet administration only. To close the Telnet connection type exit on the command line.

Figure 1-8 Shell (Telnet) Administration

To connect to the J2EE dispatcher, enter the following command: “telnet hostname telnet_provider_service_port” or, as an example: “telnet ibm_node_01 50008”.



Page 16

• ConfigTool

The Config Tool enables offline configuration of the J2EE Engine cluster elements. It is XML-based and allows the user to export for later use. When working offline, it is not required to have a SAP J2EE Engine running, as the Config Tool connects directly to database and applies the changes. When running the J2EE Engine Config Tool via a GUI or text-only interface, it connects to the database and scans the server configuration. The information provided is passed to the corresponding interface that is used for configuration. The J2EE Engine GUI Config Tool enables configuration of J2EE Engine cluster elements concerning properties, adding new elements, and exporting the system configuration to an XML file. After installing the J2EE Engine a configtool directory is created, containing a configtool script file. To start the GUI ConfigTool enter the following command: “/usr/sap/<SID>/<Instance Name>/j2ee/configtool/configtool.sh” (or configtool.bat for the Win based environment).

Figure 1-9 The GUI ConfigTool



Page 17

• SAP Netweaver Administrator SAP NetWeaver Administrator (NWA) is a new Web Dynpro-based tool for administration and monitoring, offering a central point of entry point into the SAP NetWeaver system landscape. The SAP NetWeaver Administrator can be used in a central scenario, where it is capable of operating an entire system landscape containing ABAP and Java systems as the application platform of SAP NetWeaver.

The NWA unifies the most important administration and monitoring tools both for Java and ABAP systems. The most important advantages of the NWA are:

• No longer need to switch between different tools for administration, troubleshooting and problem analysis of SAP NetWeaver system landscape

• Central administration tool available landscape-wide for both Java and ABAP systems, for starting and stopping instances, checking configuration settings and logs, and monitoring error-free functioning of components

• Interface follows the current guidelines for interface design, is easy-to-use, task-oriented and complete. Web Dynpro runs in a normal web browser

• Interface allows seamless navigation to other SAP NetWeaver administration tools (User Management Engine, System Landscape Directory)

• For Java, the NWA represents a crossover from various expert tools to an integrated, simple and clear solution. The NWA also completes the integration of the data sources for monitoring

• For ABAP, the NWA represents a crossover from many different expert transactions, which are difficult to use and integrate



Page 18

Figure 1-10 Netweaver Administrator

To connect to the Web Administrator, open a following http page: “http://<fully qualified hostname>:50000+(100 * instance number)/nwa” or in the following example “http://ibm_node_01:50010/nwa”.



Page 19

3.2. Software Deployment Manager (SDM) The Software Deployment Manager (SDM) is a tool to deploy and manage software packages from SAP. SDM resides in the SAP Web Application Server. The SDM is responsible for installing and updating software components into the SAP J2EE Engine. It also provides modification support on demand. Each time a new component version is deployed, the CBS consults the SDM on the target system. This process can be initiated via the SAP NetWeaver Developer Studio.

• Software Deployment Archive (SDA) The Software Deployment Archive (SDA) is the delivery format for SAP applications in programming languages other than ABAP. It is a ZIP-compatible archive format that can be used as a container for other archives. The SDA contains the manifest information - that is, package-related data - of its archives (such as jar, war) and an SAP manifest, which contains additional software logistics information. The EAR archive is a special case in the J2EE context. If an EAR archive contains a SAP manifest, it is also a SDA. The SDM recognizes the EAR archive as a SDA, but does not rename the archive extension as <archive_name>.sda. SDA is the smallest unit that can be deployed. Furthermore, the SDA is the smallest unit for which patches can be created and delivered.

• Software Component Archive (SCA) A Software Component Archive (SCA) is the physical representation of a version of a software component. It contains a specific number of SDAs, whose quantity describes a precisely-defined version level. A SCA update always results in a new version level of the software component.

• Deployment

The deployment is the final step in the software delivery process; it involves the deployment of available software packages - SDAs or SCAs - in the runtime environment of the SAP Systems.

When deploying SDAs/SCAs, the Software Deployment Manager stores the data in the SDM Repository, from where it then manages the installed archives. The SDM recognizes dependencies between archives and provides support when installing and maintaining shared applications.

The SDM uses the Deployment Manager to control the deployment of SDAs/SCAs for the following

o J2EE applications o File system content o Database content o SAP J2EE Engine Additional Libs



Page 20

Figure 1-11 Software Deployment Manager (SDM)

To start the SDM GUI tool, enter the following commands: “/usr/sap/<SID>/<Instance Name>/SDM/program/StartServer.sh” (if server not started), “/usr/sap/<SID>/<Instance Name>/SDM/program/sdm.sh remotegui”.



Page 21

4. SAP Java Server Monitoring The previous chapters have provided information about the basic architectural elements of the SAP Java landscape. This chapter will cover the different aspects of the Java monitoring techniques. The following tools and techniques should be utilized to achieve a robust and scalable environment through higher availability and optimized performance of the monitored applications. The basic Java monitoring architecture is developed and organized centrally, and in most of cases, will take advantage of the SAP CCMS monitoring architecture. Instead of investigating potential configuration problems or performance bottlenecks through individual system logging, the data are collected and presented in a central monitoring system (CEN). Problems will be displayed as soon as they occur, while log files and performance data can be accessed from one location to simplify administration.

Figure 1-12 Central Monitoring of the SAP NW04s System Landscape

To address the different aspects of Java monitoring, two different task groups have been defined: • Problem detection • Problem analysis



Page 22

4.1. Techniques for Problem detection

4.1.1 Monitoring of the Log and Trace files The first step in the problem detection phase is the search for information contained in the log files generated by the Java engine. All the important events that describe either the startup or typical workload of the engine are contained in this set of log files. SAP has done an excellent job of separating the log files in two groups, providing different levels of the information that support the needs of two different types of users: administrators and developers. The log files for administrators contain only the information needed for a broad overview of typical Java engine related activities. System administrators need to scan a wide range of engine related elements to determine current status (overall system availability, CPU and memory utilization, performance related to the DB, etc.) On the other hand, developers require detailed information about engine-related activities and applications they are working on. Typical elements of interest include status of the heap memory and program stack. Thus, during typical Java engine activity, two types of the files are created:

• Log files for the administrators • Trace files for the developers

The Visual Administrator contains all the necessary tools needed to configure and review Java engine generated files. Those three configuration elements are:

• Log Manager – manages the logging of system events (Figure 1-13)

Figure 1-13 Log Manager in Visual Administrator



Page 23

Each of the generated log files are positioned into a separate “home” directory of the specific node (dispatcher or server), for example: “/usr/sap/SID/JC<InstNo>/j2ee/cluster/dispatcher<No> or server<No>”. The location of the generated trace file, however, is defined by the values of the variables circled in the Figure 1-13. The functionality of the Log Manager is also provided in the Configtool. The benefit is that the Java engine doesn’t have to be started to use this tool. Once the specific set of the parameters has been changed and applied, the engine has to be restarted.

• Log Configurator – used to configure individual logging and tracing settings (Figure 1-14) This element enables the user to manage logs and traces for individual Java components. Even though it is possible to create customized elements (category objects for logs and location objects for traces), all SAP applications come with predefined categories and locations. There are two modes of work on this panel (circled on Figure 1-14): “simple mode” for use on production systems and “advanced mode” for use on development systems.

Figure 1-14 Log Configurator in Visual Administrator



Page 24

• Log Viewer – used to review Java engine generated logs and traces, the “Log Viewer” functionality is also provided by other tools:

o as a standalone Java application contained in the directory: “/usr/sap/SID/JC<InstNo>/j2ee/admin/logviewer-standalone” (Figure 1-15)

o Command line log viewer o Log Viewer functionality integrated in the SAP Netweaver (Figure 1-16)

Figure 1-15 Standalone Log Viewer

Attached is a list of the most important log & trace files to be reviewed in the problem resolution process. They are located in the: “/usr/sap/SID/JC<InstNo>/work” directory. Files providing basic information:

• INSTSTAT: o Information about J2EE instance startup

• log_bootstrap_ID_<InstNo>: o Information about the files that are synchronized during the engine startup o Every server process creates a separate log file

• std_dispatcher.out: o Information about the J2EE dispatcher process startup and shutdown

• std_server<ServNo>.out: o Information about the specific J2EE server process startup and shutdown

• std_sdm.out: o Information about the SDM server process startup and shutdown, exists only on the J2EE

central instance



Page 25

Files providing more detailed information: • dev_bootstrap_ID<InstNo>

o Detailed information about the J2EE bootstrap process including the corresponding JLaunch processes and startup parameters

• dev_dispatcher o Detailed information about the startup and shutdown of the J2EE dispatcher process

• dev_server<ServNo> o Detailed information about the startup and shutdown of the J2EE server process and

startup parameters • dev_sdm

o Detailed information about the SDM server process startup and shutdown, exists only on the J2EE central instance

• jvm_bootstrap_ID<InstNo>.out o Detailed information about the J2EE bootstrap process, files that are synchronized and

synchronization statistics

Figure 1-16 Log Viewer in SAP Netweaver Administrator



Page 26

4.1.2 Availability Monitoring of SAP Java engine and Java based applications (GRMG)

This technique is based on the so-called “Heartbeat” (GRMG) central infrastructure, which is used to check the availability of the Java engine and Java based applications. GRMG contains two parts:

• GRMG Infrastructure, to send, receive and display GRMG communication elements in the alert monitor

• GRMG application performs the actual application monitoring o Central monitoring system periodically calls a URLs – the GRMG applications o GRMG checks its component or scenario o GRMG application send a XML message back to the CEN o Message received is transferred into a CCMS monitoring tree and displays availability and

messages of the component

Figure 1-17 Availability Monitoring using GRMG

Most current Java based SAP applications (EP, XI, BI etc) use the GRMG infrastructure. The following figures show how to configure and activate the GRMG for the monitoring of the Enterprise Portal infrastructure.



Page 27

Required set of the steps are to be executed: • Maintain the GRMG customizing file in the Visual Administrator • Transfer the GRMG customizing file into the CCMS agent (use the “Upload” button, Figure 1-18)

Figure 1-18 GRMG customizing in Visual Administrator

• Customize GRMG information in the CEN system using the GRMG transaction (Figure 1-19) • Display the availability of the activated scenarios in the RZ20 sub-tree

Figure 1-19 GRMG customizing in CEN



Page 28

4.1.3 Distributed Statistics Records (DSRs)

SAP Performance Statistics Records are generated to monitor the performance of an ABAP system and its components. Generated statistics records provide information about the system workload and the resources used by a specific action. Recently, this functionality was extended by using Distributed Statistics Records to monitor the non-ABAP components, primarily the SAP J2EE engine. Figure 1-20 provides an overview of the elements and communication paths required for performance data collection, aggregation and display (Java Standalone System and Double-Stack System):

Figure 1-20 Collection, Aggregation and Display of the DSR’s

To activate the J2EE service to generate statistics records, a certain set of configuration activities must be executed in the Visual Administrator. Detailed explanation of these configuration steps can be found on the SAP Global Help site (http://help.sap.com).

Once the data collection agents are successfully activated (SAPCCMSR with the –j2ee option,) the generated statistics will be transferred to the CEN to which that agent was registered.



Page 29

There are several steps that DSRs go through before they are displayed in the CCMS of the CEN:

• Activated DSR service in the J2EE engine starts generating the records • Generated records are written to the specific location of the file system containing J2EE relevant

objects • Activated agent reads the records and sends them to the CEN • Raw transferred data can be collected and aggregated by the DSR collector (runs on CEN) and

then presented in the Global Workload Monitor (ST03G) (Figure 1-21)

Figure 1-21 Aggregated DSR presentation in ST03G

• Raw transferred data can be presented directly (without aggregation) in the CEN using the STATTRACE transaction; this single record data overview will be used as a functional trace to provide a finer granularity of statistic files (Figure 1-22)

• In the case of a double-stack system, the agent has to be registered twice, with the local ABAP system and the CEN

• In both types of activation, single or double J2EE stack, the SAPCCMSR agent is launched automatically after the J2EE engine is started successfully



Page 30

Figure 1-22 Detailed DSR presentation in STATTRACE



Page 31

4.1.4 State Monitoring The most important elements of a system management discipline are the diagnostic tools and procedures that monitor system performance. Within a global corporate IT infrastructure, the monitored systems can include computers, networks, storage, operating systems, databases and applications. This chapter will focus on the JMX (Java Management Extensions) monitoring infrastructure implemented in the J2EE engine. JMX is an open Java standard that can be used to monitor all Java-based web applications, in addition to SAP Netweaver applications. This infrastructure provides a consolidated and automated process to monitor the system state, along with the important parameters of SAP Netweaver and Java applications running on those systems. Using JMX will decrease the frequency and granularity of the tasks required to retrieve necessary information about the system status.

Figure 1-23 JMX Implementation in SAP Netweaver

Figure 1-23) provides an overview of the steps that are executed during the runtime data collection: o Monitored resource (Java application, service, etc.) creates a monitoring object (MBeans) o Developers are responsible for creating and registering MBeans in the application code o Data collection can be passive (polled by monitor) or active (pushed by resource) o Monitoring objects are registered with the JMX Agent (MBean Server) o MBean server in the SAP Java engine is presented through the JMX Adapter Service o MBean Server manages the MBeans information and makes it available to the JMX clients

through the monitor Bean o JMX clients are Monitoring Services in the Visual Administrator and RZ20 in CCMS



Page 32

The JMX Adapter service links the data providers implemented in the shape of MBeans and the JMX clients used to monitor the system (or Java applications). To control the behavior of this service, two different tools are available: Java engine ConfigTool and Visual Administrator (Figure 1-24). Properties of the service may be changed on the global and local level. If the properties need to be replicated across all nodes, the global configuration should be applied. If specific nodes need to be tested, the local configuration should be applied. The local change has a priority over the global change. Every time the property is changed, the affected nodes have to be restarted.

Figure 1-24 JMX Adapter Service Global Properties in the Visual Administrator

The data generated by the link established between the system (or Java application) and the MBeans are grouped, consolidated and presented in one of the administrator tools (Visual Administrator or CCMS). The monitoring tree consists of five separate “nodes:” Applications, Kernel, Performance, Services and System. Each of these nodes has a predefined group of sub-nodes that consolidates a monitoring structure for a specific application or system resource (MBean). Every resource has a predefined threshold which can be changed using the “Configuration” button (Figure 1-25). A color-coded scheme (red = problem; yellow = warning, green = OK) signals down to the root node once a certain threshold is reached.



Page 33

Figure 1-25 Visual Administrator Monitoring Service

Besides the Visual Administrator, the CCMS transaction RZ20 (Alert Monitor) can also be used to present data submitted by the source Java-related elements. Before this is established, agent registration and configuration has to be executed. One of the benefits of the Alert Monitor is that like Java, it can present the data generated by the ABAP instances. The unique value presented through the state monitoring infrastructure is a simplification of the anomoly detection generated in the system and Java applications.



Page 34

4.2. Techniques for Problem Analysis The previous chapter presented tools and procedures used to detect Java-based system performance and/or configuration issues. Once the cause of the problem is determined, the problem analysis phase will start. This section contains a set of tools that will provide detailed information about any system anomaly. At this point in time, since it presents such an important prerequisite, I will assume that the basic knowledge about the system architecture of the monitored environment and the Java programming language is already present.

4.2.1 Performance Trace Performance tracing is the “advanced” version of the activities discussed in the description of the DSR collection. The same performance data collection elements will be used. The source (monitored) system must have the Java engine “Distributed Statistics” service enabled, and the CCMS agent SAPCCMSR registered with the Central Monitoring System (CEN). The elements are presented in the Figure 1-26.

Figure 1-26 Data presentation in the STATTRACE and ST03G

The functional trace tool (transaction STATTRACE) is used to display raw statistical data (individual records) and traces from the non-ABAP systems.



Page 35

It is an extension of the following two transactions: o STAD, displaying only raw statistical data from the ABAP systems o ST03G (Global Workload Monitor), displaying aggregated statistics data collected by the

specific collector Data path (C) in Figure 1-26 represents a single record display option for the data, read directly by the functional trace tool from the statistics files accessed by the remote CCMS agent. The granularity of the information in the functional trace is finer than that of the data in the Global Workload Monitor. DSRs can only measure the duration of the processing between the entry points of the “local” J2EE service presented on Figure 1-27.

.

Figure 1-27 DSR’s across Multiple Components

It is possible to trace the actions of the business process across several components and system boundaries using the communication element called Passport. Every time the LUW is executed on the host, a new DSR with a GUID is generated and sent to the SAPCCMSR agent for presentation in CEN. If the business process spans multiple hosts with multiple LUWs, each time a remote call is made, a passport is sent with the previously generated GUID. Figure 1-28 provides an overview of the Passport function. To activate the performance trace, the following Visual Administrator menu is selected: Performance Tracing Service -> Runtime -> Trace Config. The chosen page will present a set of the functional modules that can be activated to generate a trace for a specific user. If no user is specified, a trace is activated for all users.



Page 36

Figure 1-28 DSR’s across Multiple Components



Page 37

4.2.2 Single Activity Trace / JARM The Java Application Response Time Measurement (JARM) is a method for collecting response time data from Java applications. The data delivers an overview of components passed by a thread, along with the time remaining in the corresponding component. The JARM displays not only the response time, but also the user who sent the request and the amount of data that was transferred. This function is designed to allow performance measurement in productive systems and to identify areas causing problems. JARM data is shown in the Performance Tracing service of the J2EE Engine Visual Administrator tool.

Figure 1-29 JARM Data in Visual Administrator

The JARM data are additionally available in the CCMS and can be displayed using the RZ20 transaction. JARM monitors must be implemented by the developer into the coding of the application. They are very low overhead and can be used during runtime without any restrictions. The purpose of this monitoring is to track the status of the Java system. The data are presented in several different views:

o Request summary This tab displays data accumulated since system start or data reset. The most

important counters are: Average Response Time of a Request and Number of Requests. If system is permanently under load, attention should be paid to the Requests per Second counter.

Figure 1-30 JARM Data – Request Summary



Page 38

o Request Overview Most important requests and components may be selected by different criteria such

as response time or amount of transferred data. The slowest response time gives an indication of where performance analysis may be necessary. If one of the requests is chosen, the detailed information in the pane below is displayed. The net time metric identifies the slowest components and the levels indicated how deep component is nested.

Figure 1-31 JARM Data – Request Summary

o Component Overview Shows the data collected for each component in a condensed way. The data can be

sorted by average gross time, outbound data, average net time or the number of calls.

Figure 1-32 JARM Data – Component Overview



Page 39

o User Overview

Shows the username with average response time, number of requests and time of the first and last request.

Figure 1-33 JARM Data – User Overview

o Thread Overview Lists all the threads currently running in applications instrumented with JARM. The

number of threads indicates the current load on the server. This tab also tracks the time a request is running and the active time of components. This helps in locating long running or blocked requests.

Figure 1-34 JARM Data – Thread Overview



Page 40

4.2.3 Single Activity Trace

The Single Activity Trace (SAT) is one usage of Java Application Response time Measurement (JARM). To write a Single Activity Trace, every component has to be instrumented with JARM. Use JARM to analyze a single user’s request to find performance problems in an application, or to understand the logical processing steps inside an application. The performance data collected are automatically written in a trace file using the SAP Logging API. This file can be viewed in the Log Viewer. The difference between SAT and JARM is that JARM only provides the top value data, while SAT provides all the trace data generated for the selected request.

Figure 1-35 SAT Functionality

To activate the Single Activity Trace choose: Server -> Services -> Performance Tracing -> Runtime -> Trace Config tab.

Figure 1-36 SAT Activation



Page 41

SAT log files are displayed in a format different from general logs:

o Steps are displayed in hierarchical manner o Username indicates the beginning of a request or an action o Displays within the structure how deep a component is nested (first column shows the

level)

Figure 1-37 SAT log files



Page 42

4.2.4 SQL Trace

SQL trace is a convenient tool for detecting problems when working with the persistent layer. It provides traces for JDBC methods that access the database or are otherwise important for database connectivity. SQL can trace the execution of SQL statements in both system and application threads. The function provides detailed information about traced statements such as a thread, application, method and session in which the statement was executed. The log format is database independent. Besides the SQL statement text, each log record contains information about the time when the statement was executed, its duration, input parameters and results, as well as context information.

To launch the SQL Trace enter the URL of the application in your browser:

“http://<hostname>:<http_port>/OpenSQLMonitors”

Only an administrator has authorization to use this tool.

Figure 1-38 Accessing the SQL Trace

SQL trace can be switched on and off dynamically. Follow the Web browser path to turn the SQL Trace on and off (it is turned off by default): “Main Page” -> “Switch on and off SQL Trace” -> “SQL Trace on or off”. SQL trace is more suitable tool for developers than for detecting general performance issues. It is not recommended to keep constantly activated as it consumes additional resources.

The SQL Trace can also be activated or deactivated using the Visual Administrator. Besides using the “OpenSQLMonitors” Web application, the results of the trace can be reviewed using the Log Viewer.

To evaluate the SQL Trace follow the Web path: “Main Page” -> “Trace Evaluation” -> Evaluate the selected trace for the specific node. The typical output is presented in the Figure 1-39.



Page 43

Figure 1-39 SQL Trace output

For a detailed description of the data columns in the presented sample output, please use the following link:

http://help.sap.com/saphelp_nw2004s/helpdata/en/24/d4ff0eccf33a49a5c26a280a119b14/content.htm

If additional information is needed, click on the specific SQL statement and the detailed output will be presented.

Figure 1-40 Detailed SQL Trace output

At this time, SQL Trace does not offer the functionality and flexibility of the equivalent ABAP ST05 transaction, but further improvements are expected, especially with the upcoming 7.10 kernel based NW.



Page 44

4.2.5 Application Trace

Application Tracing is aimed primarily for the developers. It allows the user to see all method invocations during a process flow. For every method, the used time is measured in microseconds. The application therefore must be started in the “bytecode-modified” mode. This can be done without putting the VM into the debug mode or redeploying the application. When the application is setup in the “bytecode-modified” mode, it is restarted and its new class loader knows there are changes to be made in the bytecodes. Once the debugging is over, the application can be started again in the “normal” mode. To use this tracing method, no instrumentation is required from the application developer. The runtime control provides everything needed for proper application tracing.

To access the runtime control in the Visual Administrator, follow the path: Server -> Services -> Performance Tracing -> Runtime -> Application tracing tab (Figure 1-41).

Figure 1-41 Application Trace Menu in Visual Administrator

Sub-tabs “Application”, “Classes”, “Invocations”, “Objects” and “Plug-ins” are used to control the tracing activities for the application.

“Application” sub-tab – area provides a list of the applications that are deployed on the server. Right mouse button click on the specific application will provide an option for the apps restart in the “bytecode-modified” or “normal” mode (Figure 1-42).



Page 45

Figure 1-42 Application Sub-tab

“Classes” sub-tab presents the Java bytecode-modified classes loaded. To check the content of the loaded classes, the action from the Figure 1-43 has to be executed (“View Source”).

Figure 1-43 Classes Sub-tab

“Invocations” sub-tab presents the methods invoked, combined with the length of the execution timed in microseconds, Figure 1-44.

Figure 1-44 Invocations Sub-tab



Page 46

“Objects” sub-tab provides a more detailed overview of the Java object graph on the target virtual machine. It can display the values of the fields of every object, show the inheritance of classes, and support paged views of arrays. Figure 1-45.

Figure 1-45 Objects Sub-tab

“Plug-ins” sub-tab provides the information about the “bytecode-modified” plug-ins, Figure 1-46.

Figure 1-46 Plug-ins Sub-tab



Page 47

4.2.6 Conclusion

Figure 1-47 When to use different performance tools (1)

After reviewing the various problem detection and problem analysis techniques, this paper will examine the following: When, how and which set of performance techniques should be utilized to properly locate and repair application anomalies on the system? A daily review of the system can be conducted using the following set of the tools:

o Application availability information is provided using the GRMG instrumentation (RZ20) o DSR statistics in the ST03G (CEN) provide a basic set of the system performance related

data o State Monitors in Visual Administrator (or CCMS in CEN, RZ20) present the state of the

Java engines, o Properly configured Log Monitoring is another valuable source of the system status related

information.



Page 48

If problem is detected using one of the previously mentioned steps, an initial set of the problem analysis techniques will be executed:

o Performance trace using the STATTRACE transaction o Single Activity Trace using the Visual Administrator and Log Viewer o SQL Trace if anomaly was discovered in the JDBC layer

Once the problem has been properly discovered and described, the application developers will determine the problem solution path primarily using the Application Trace and Application Debugging functionality in the Visual Administrator and SAP Netweaver Developer Studio. An overview of the previously described steps is presented in Figure 1-48.

Figure 1-48 When to use different performance tools (2)



Page 49

5. IBM Java Technology for SAP As mentioned before, SAP J2EE Engine utilizes IBM JDK 1.4.2 implementation as an underlying JVM mechanism. Following paragraphs provide detailed description of the IBM JDK prerequisites and configuration details for the successful Java based SAP Netweaver implementation (from the SAP Web AS perspective). It is important to state that the application of the recommendations from the following chapters should only be executed after careful review of the specific group of the SAP OSS notes. Most of these notes will be mentioned in this chapter, but the good starting point is the OSS note 716927. IBM recently exchanged the JDK version 1.4.2 for the application stacks of SAP Netweaver 2004 and SAP Netweaver 2004s. The new JDK version is called 1.4.2 Hybrid JVM and it is available for NW2004 and NW2004s on AIX 5L. It provides more consistent runtime behavior, enhances memory management and improves debugging capability. It is also 100% upward compatible for all SAP business applications. To check the details of the announcement, check the SAP OSS Note 1039948. Additionally, there is a set of the OSS notes that should be reviewed before the recommendations are applied:

o If planning to a new installation or an upgrade of SAP NW2004 SR1 or SAP NW2004s SR1, follow the instructions in SAP Note 1024700

o For new installations of SAP NW2004s SR2, follow the instructions in SAP Note 1008311 o For SAP NW2004 SP14 and above or SAP NW2004s SP6, it is recommended to use the

new IBM 1.4.2 Hybrid JVM with J9 technology, follow the instructions in SAP Note 1024539

Before describing the IBM JVM parameters, we will provide the overview for one of the most important activities executed inside the JVM working area. It is called a garbage collection and, in the simple words, it provides a “heap memory recycling” mechanism.



Page 50

5.1 IBM JVM Garbage Collection When the JVM cannot allocate an object from the current heap because of lack of space, a memory allocation fault occurs, and the Garbage Collector is invoked. The first task of the Garbage Collector is to collect all the garbage that is in the heap. This process starts when any thread calls the Garbage Collector, either indirectly as a result of allocation failure, or directly by a specific call to System.gc(). The first step is to retrieve all the locks the garbage collection process needs. This step ensures that other threads are not suspended while they are holding critical locks. All the other threads are then suspended. Garbage collection can then begin. It occurs in three phases:

o Mark o Sweep o Compaction (optional)

o Mark phase

In the mark phase, all objects that are referenced from the thread stacks, statics, interned strings and JNI references are identified. This action creates the root set of objects the JVM references. Each of those objects can, in turn, reference others. Therefore, the second part of the process is to scan each object for other references it makes. These two processes together generate a vector that defines live objects.

o Sweep phase After the mark phase, the mark vector contains a bit for every reachable object that is in the heap. The mark vector must be a subset of the allocbits vector. The task of the sweep phase is to identify the intersection of these vectors; that is, objects that have been allocated but are no longer referenced. The original technique for this sweep phase was to start a scan from the bottom of the heap, and visit each object in turn. The length of each object was held in the text that immediately preceded it on the heap. At each object, the appropriate allocbit and markbit was tested to locate the garbage. Now, the bitsweep technique avoids the need to scan objects that are in the heap and therefore avoids the associated overhead cost for paging. In the bitsweep technique, the mark vector is examined directly to look for long sequences of zeros (not marked), which probably identify free space. When such a long sequence is found, the length of the object at the start of the sequence is examined to determine the amount of free space to be released. Objects are not normally allocated from the heap itself but from thread local heap, which is allocated from the heap and later used by an individual thread to meet any allocation requirements.

o Compaction phase When the garbage has been removed from the heap, the Garbage Collector can consider compacting the resulting set of objects to remove the spaces between them. Because compaction can take a long time, the Garbage Collector tries to avoid it if possible. Compaction is, therefore, a rare event.



Page 51

5.2 IBM JVM parameters The following is an overview of the Hybrid JVM related parameters used in the previously mentioned OSS notes. This list is a subset of all parameters available. No parameters for the “old” JVM version will be discussed. If the default values of the presented parameters need to be changed, it must be done under the close supervision of either the SAP and/or IBM support service.

o -Xms<size>, sets the initial memory size. o -Xmx<size>, sets the maximum memory size. o -Xmn<size>, sets both the values of the parameters –Xmns (the initial size of the new

space) and –Xmnx (the max size of the new space.) o -Xgcpolicy:gencon, one of the four garbage collection policies. Handles short-lived objects

differently than objects that are long-lived. Applications that have many short-lived objects can experience shorter pause times with this policy while still producing good throughput.

o -Xj9, switch that tells the JVM not to run in the “classic” mode. o -Xdisableexplicitgc, disables the explicit calls to System.gc to request garbage collection.

In some cases, these calls can degrade performance time through premature garbage collection and compactions.

o -Verbose:gc, this option is used to obtain information about the Java Object heap in real time while running Java applications.

5.3 Monitoring the JVM activity The initial set of the steps suggested to be used to check the status of the JVM (or the SAP JLaunch process) are based on AIX system performance tools. Topas, nmon, vmstat, ps and svmon will provide a basic overview of process related activity. To view detailed information about the JLaunch process linked with the IBM JVM, the previously explained Verbose:gc JDK option must be utilized. When this option is used, all Garbage Collection activities are logged into a file named standard server out (or std_serverX.out,) where X is the number of the corresponding server process (Jlaunch ID.) Every server process has its own std_output file that resides in the “/usr/sap/<SID>/JCXX/work” directory. The content of the simplified sample std_output file is presented in Figure 1-49. This is a sample file simulating the content generated using a System.gc call. This call is used to manually initiate a Garbage Collection.

Figure 1-49 Verbose:gc content as a result of the System.gc call

Another example is included to show how the content of the std_output file is changed if heap memory allocation happens, Figure 1-50.



Page 52

Figure 1-50 Verbose:gc content as a result of the Heap memory allocation failure

Browsing the content of the sample file is a complex and time consuming process. Several different free tools exist that can help present the GC generated data in a much more understandable way. The tool that I use most often is called PMAT, “Pattern Modeling and Analysis Tools for Java Garbage Collector”. PMAT can be downloaded from the IBM AlphaWorks Web site: http://www.alphaworks.ibm.com/tech/pmat. PMAT is used for parsing verbose GC traces, analysis of Java heap usage, and for recommending key configurations based on pattern modeling of Java heap usage.

Figure 1-51 Initial screen of the PMAT tool



Page 53

One of the best options in the PMAT is the graphical overview of the JVM Garbage Collection related activities, presented in the Figure 1-52:

Figure 1-52 PMAT tool (Graphical overview of the GC activities)

This chapter provided a basic overview of the IBM JVM tuning steps around the Garbage Collection process. I didn’t cover the specific tuning challenges around the most popular SAP solution, Enterprise Portal since a document about it already exists and is mentioned in the Publications Appendix.



Page 54

Publications: The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this white paper.

1. Publications

• “SAP Enterprise Portal on AIX 5.3 and Power5”, Walter Orb (IBM) http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100740

• “Tuning SAP R/3 with Oracle on pSeries”, Mark Gordon (IBM) http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100377

2. Online Resources

• SAP Service Marketplace http://service.sap.com

• SAP Help Portal

http://help.sap.com

• Developer Works (IBM’s Resource for Developers) http://www-130.ibm.com/developerworks/java/

• AlphaWorks (Java Technology)

http://www.alphaworks.ibm.com/java

• IBM Infocenter for AIX: http://publib.boulder.ibm.com/infocenter/pseries/index.jsp

Documents

IBM - SAP Java Monitoring & Tuning Guide