What Your Jvm Has Been Trying To Tell You

© 2008 IBM Corporation

IBM Software Group

May 2007 | John Pape, IBM WebSphere SWAT

What your JVM has been trying to tell you…

A look at available PD options within the IBM JVM

IBM Java Technology

© 2007 IBM Corporation2

JVM Overview

JVM Components

Problem Scenarios

Summary

References

Agenda

IBM Java Technology


JVM Overview

IBM Java Technology


JVM Overview

JVM = A Java Virtual Machine (JVM) is a virtual machine that interprets and executes Java bytecode. (compiled Java programs)

It is a program/process like any other.

Virtual Machine = software that creates a virtualized environment between the computer platform and its operating system, so that the end user can operate software on an abstract machine.

The JVM allows a developer to write Java code and compile it, one time, and then deploy that bytecode to any JVM on any platform and have it run basically the same way.

The JVM abstracts OS level functionality and programming interfaces from the Java developer.

IBM Java Technology

© 2007 IBM Corporation

Java App

Java App

Java App

3rd party libraries

5

JVM Overview cont…

Java language libraries (I.e java.*, javax.*)

Java Virtual Machine

Operating System APIs

Operating System Kernel

Abstraction

IBM Java Technology


JVM Components

IBM Java Technology


Components of the JVM

Native Code

Execution Management (XM)Core Interface (CI)

Diagnostics (dg)

Execution Engine(XE)

Classloader(CL)

Lock (LK)Data

Conversion (dc)Storage

(ST)

HPI (Hardware Platform Interface)

Java and JNI code

Operating System Platform

Native LibrariesExecution Management (XM)

Just In Time Compiler (JIT)

Object Request Broker (ORB)

Java Class Extensions

Java Class Libraries

IBM Java Technology


Core Interface

This subcomponent encapsulates all interaction with the user, external programs, and operating environment. It is responsible for initiation of the JVM.

Provides presentation (but not execution) of all external APIs (for example, JNI, JVMDI, JVMPI)

Processes command-line input

Provides internal APIs to enable other sub-components to interact with the console

Holds routines for interacting with the console; nominally, standard in, out, and err

Provides support for issuing formatted messages that are suitable for NLS

Holds routines for accessing the system properties

IBM Java Technology


Execution Engine

This subcomponent provides all methods of executing Java byte codes, both compiled and interpretive.

Executes the byte code (in whatever form)

Calls native method routines

Contains and defines byte code compiler (JIT) interfaces

Provides support for math functions that the byte code requires

Provides support for raising Java exceptions

IBM Java Technology


Execution Management

This subcomponent provides process control and management of multiple execution engines. Is initiated by the core interface. It provides:

Threading facilities

Runtime configuration; setting and inquiry

Support for raising internal exceptions

End JVM processing

Support for the resolution and loading of native methods

IBM Java Technology


Diagnostics

This subcomponent provides all diagnostic and debug services and facilities. It is also responsible for providing methods for raising events.

Support for issuing events

Implementation of debug APIs v Trace facilities

Reliability, availability, and serviceability (RAS) facilities

First failure data capture (FFDC) facilities

IBM Java Technology


Class Loader

This subcomponent provides all support functions to Java classes, except the execution.

Loading classes

Resolution of classes

Verification of classes

Initialization of classes

Methods for interrogation of class abilities

Implementation of reflection APIs

IBM Java Technology


Data Conversion

This subcomponent provides support for converting data between various formats.

UTF Translation

String conversion

Support for primitive types

IBM Java Technology


Lock

This subcomponent provides locking and synchronization services.

Maintains Java lock monitors

Manages thread locking in the JVM

Provides report on deadlocks in Java thread dumps

Provides report on monitor pool in Java thread dumps

IBM Java Technology


Storage

This subcomponent encapsulates all support for storage services.

Facilities to create, manage, and destroy discrete units of storage

Specific allocation strategies

The Java object store (garbage collectable heap)

IBM Java Technology


Hardware Platform Interface

This subcomponent consists of a set of well-defined functions that provide low-level facilities and services in a platform-neutral way.

The HPI is an external interface that is defined by Sun.

IBM Java Technology


Problem Scenarios

IBM Java Technology


Problem Scenarios

My application is running slow when running a certain code path, why?

My application has very erratic response times; sometimes it’s great, others it’s terrible, why?

I am getting out of memory errors on my application due to heap fragmentation, how can I tell what objects are pinned and dosed?

What does my memory footprint look like for my application?

Which parts of my application consume the most CPU time?

Which parts of my application are taking up the most memory?

Is my application performance as result of synchronization problems in my threads?

I need to know the exact activities of the JVM when a certain method is executed.

IBM Java Technology


My application is running slow when executing a certain code path, why?

This problem can be approached by setting a JVM method trace.– Example:

– Database calls are slow when called from my stateless session EJB. The database class is com.mycorp.db.DatabaseWrapper and the EJB is com.mycorp.ejb.session.LogicBean.

– The following trace could be enabled on the JVM (the –D parameters are delimited by spaces, there is no line break)

– -Dibm.dg.trc.maximal=mt –Dibm.dg.trc.methods=com/mycorp/db/DatabaseWrapper.*(),com/mycorp/ejb/session/LogicBean.*() –Dibm.dg.trc.output=/tmp/jvm.trc

– JVM tracing incurs overhead and can produce large quantities of data, very rapidly. It is best to test the code path with a single request to minimize any concerns with disk space usage.

– After taking the trace, it cannot be read until formatted. To do this you must execute the TraceFormat tool contained in the IBM JDK.

– java com.ibm.jvm.format.TraceFormat <trace file> -indent– -indent is used to provide helpful formatting of the nested levels of method tracing– Timestamps printed in the JVM trace are in UTC format. This means you must translate the

times into your time zone.

The net result of this tracing is an EXACT code path taken in the application. It may be beneficial to add some other classes or packages to the trace to obtain a clear overall picture (e.g. include some WebSphere Resource Adapter classes to see server activity around the application calls)

IBM Java Technology


My application has very erratic response times; sometimes it’s great, others it’s terrible, why?

This problem can be approached in 2 ways:

– 1. JVM method trace

– 2. Enable and analyze verbose garbage collection (GC) output

Since JVM method tracing has been demonstrated already, we’ll focus on verbose GC analysis

– To enable verbose GC:

– Add –verbosegc or –verbose:gc to JVM arguments– WebSphere has a check box to tick off, other servers/products

may have similar methods on enablement– Use –Xverbosegclog:/path/to/desired/gcfile.txt to specify a file to

output verbose GC data to. – By default, verbose GC outputs to native_stderr.log. It is also possible

to create rolling, generational verbose GC logs. Consult the IBM JVM Diagnostic Guide for your version of JDK for details.

IBM Java Technology


I am getting out of memory errors on my application due to heap fragmentation, how can I tell what objects are pinned and dosed?

Pinned Object = Objects on the heap that are permanently immobile until explicitly unpinned by JNI. Moving a pinned object would cause a null pointer in the native code referencing it.

Dosed Object = Objects on the heap that are temporarily immobile.

Pinned and Dosed objects cannot move and thus cannot be compacted thereby reducing the overall amount of contiguous free space in memory.

Useful traces

– Add -Dibm.dg.trc.print=st_verify

• Displays the # of pinned/dosed objects on the heap

– Add -Dibm.dg.trc.print=st_compact_verbose

• Displays the pinned/dosed objects on the heap during GC compaction

Knowing the layout of the objects on the heap can help you troubleshoot OutOfMemoryErrors as well.

IBM Java Technology


What does my memory footprint look like for my application?

Heapdumps are the primary means of viewing memory heap contents.

To enable signal-based heapdumps – that is heapdumps that produced on a kill -3 signal:

– Add an environment entry called IBM_HEAPDUMP with a value of true

– Heapdumps can also be called from Java code

– E.g. com.ibm.jvm.Dump.HeapDump();

Heapdumps will be produced when a JVM exhausts its Java heap and throws an OutOfMemoryException.

– To disable this behavior: add environment entry IBM_HEAPDUMP_OUTOFMEMORY with a value of false (same can be done for javacores/javadumps – IBM_JAVADUMP_OUTOFMEMORY=false)

– On older JVM’s you may need to set this value in order to produce heapdumps on an OutOfMemoryException

Heapdumps on JDK 1.4.1 SR1 and later explicitly do a GC before dumping, this assures that only live objects are in the heapdump.

Heapdumps are representations of memory, so if the JVM has a large heap size, expect a large dump.

Heapdumps can be produced in binary (.phd) format or in text (.txt) format.

The Sun HotSpot JVMs handle heapdump generation differently and have different parameters to invoke. Consult with the references section of this presentation for more information.

IBM Java Technology


Which parts of my application consume the most CPU time?

Using the HPROF profiler, included with the JDK, you can determine which methods are consuming the most CPU time.

To invoke HPROF:

– Add –Xrunhprof:<name>=<value> / where <name> and <value> are name/value pairs of HPROF parameters

– To obtain CPU calculations:

– -Xrunhprof:cpu=samples

– To obtain greater detail (with a performance trade-off)

– -Xrunhprof:cpu=timings

EPROF can be executed on SUN HotSpot JVM's

– -Xeprof

Running HPROF can result in an unstable JVM which can crash unexpectedly, use with caution (not for production environments)

IBM Java Technology


Which parts of my application are taking up the most memory?

The HPROF profiler can assist here once again

– To obtain data on memory allocations by method:

– Add –Xrunhprof:heap=sites

HPROF will provide a sorted list of sites with the most heavily allocated objects at the top.

This data will show you were the “hot spots” are in the code, that is, the places in the code path that is responsible for the generation of new objects.

Useful for determining what part of the application is contributing the most to the overall memory footprint.

Can be used to stem potential memory leaks way before they become an issue.

IBM Java Technology


Is my application performance a result of synchronization problems in my threads?

HPROF can be used here once again

–To collect thread and synchronization data from the JVM:

–Add –Xrunhprof:monitor=y,thread=y

This setup will provide data shows how much time threads are waiting to access resources that are already locked (resource contention)

It also provides a list of active monitors in the JVM, this info can be useful to determine the presence of deadlocks.

IBM Java Technology


I need to know the exact activities of the JVM when a certain method is executed.

In this case, you need to trigger a dump of some kind on a specific trigger.

The JVM can be setup to trigger several different types of dumps on many conditions:

– Example : trigger java dump on uncaught ArrayIndexOutOfBoundsException

– -Xdump:java:events=uncaught,filter=*ArrayIndexOutOfBoundsException*

Types of dumps that can be produced

– Java dump / Java core / thread dump

– Heap dump

– Core dump / System dump

– Snap trace

– Stack dump (JDK 5 SR10 and onwards)

IBM Java Technology


Thanks!