Troubleshooting the Cause for High CPU utilization with J2EE application

on UNIX server

Abstract

There is increasing concern among developers in identifying the relation and cause for high CPU usage with developed Java/J2EE application running on different workloads. This paper defines a methodology how to identify the process id for high CPU utilization on UNIX server with predefined tools with in UNIX server and also helps in identifying the root cause with in J2EE application which is consuming more CPU time with in the available CPU resources.

PardhaSaradhi DSenior Technology Architect

Infosys Limited

1. Introduction

CPU utilization (or process utilization) is the magnitude of usage for which a central processing unit (CPU) is involved in processing logical steps of a program or system. For example, processing on input/output (I/O) operations or method execution, etc. CPU usage allows to quantify the overall business of the system. When the CPU usage is above threshold, let’s say 99%, the logical request fired on the server may experience lag. In such scenario high CPU usage may indicates insufficient processing capability. To meet the requirement, it is necessary to either the CPU capacity needs to be increased or to analyse the process which is CPU intensive and fine tune the program/application associated with the process. Otherwise the user experience may degrade, who is a service requestor of the server.

In the events of multiple processes, the CPU shares the available resources among the processes. If anyone/multiple of the processes demand more resources for processing a logical task/s, the CPU will allocate the resources within its bucket and resulting high CPU usage. Since the CPU resources are shared among all the processes running on the server, the other running process may experience lag in completion of the task. CPU time may increase because of waiting for input or being suspended to allow other programs to run, increased elapsed time which time includes I/O time and all other types of waits incurred by the other operation, etc.

2. J2EE application work load

J2EE application such as web applications running in standard application servers like WEB SPHERE, WEBLOCIG, JBOSS, JRUN etc. sometimes consumes more CPU resources depending on the nature of work load. As the logical or transaction flow might consume more CPU cycles base on the developed code to meet the functional requirement. The methodology suggested consider each and every logical or transaction flow for testing purpose. Inject a user work load [a repetitive invocation of a specific logical/transaction flow] on the application. A repetitive invocation of specific logical or transaction flow can be achieved by any load simulation tool (freeware or licensed) available in the market. It is advised to initiate the load on the application in parallel by different users or virtual users (when load simulation tool is being used). It is assumed that typical application server would process the request and render the response to the end user or requested entity.

3. Identifying the CPU usage

Once the user work load is injected on the application server, the CPU usage on the UNIX server, hosting the application server, can be monitored by using the predefined tools available with the UNIX server.

IBM AIX, HP, Solaris, and Linux: the commands can be used to monitor the CPU usage.

‘vmstat 1’ – “vmstat” reports information about processes, memory, paging, block IO, traps, and cpu activity. Additional reports give information on a sampling period of length delay. As below.

With the above command, the percentage of CPU usage by user and system process can be easily identified. In order to understand the CPU usage by application server (hosting the J2EE application), it is necessary to ‘grep’ (ps –ef | grep java) for the java process and identify the process id (PID) of the application server.

In case of IBM AIX, Solaris, HP and Linux servers ‘top’ command will list out all the PIDs running on the UNIX server and their resource utilizations. Depending on the version of ‘top’ command, there are different formats as shown below. However the purpose of the command in the current methodology remains same, is to identify the CPU usage.

For Solaris:

For HP:

For AIX:

4. Analysing the high CPU utilization by a J2EE application4.1. Case study: Analysing a sample J2EE application on AIX UNIX servers

A sample work load has been tested on AIX Unix operating system (server with 4 core processor) having 11 threads running with in a java instance. Let us understand how the analysis goes.

4.1.1. Executing tprof Command on AIX server

The tprof command generally reports processor usage for each individual programs and also system as a whole. This command is a useful tool which reports the CPU usage of any process ID associated with a Java, C, C++, or FORTRAN program that might be processor-bound. This is helpful in identifying which sections of the program are most heavily using the processor.

It is necessary to execute the below script/commands to find the CPU usage on AIX server and also execute ‘kill -3 <JAVA_PID>’ for java core generation. Otherwise the below script can be executed to generate the required information from the AIX server.

#!/bin/ksh

print "StatsGather>> aixperf.sh script starting..."# Set the first command line argument equal to the variable PROBLEMATIC_PID.PROBLEMATIC_PID=$1print "StatsGather>> PROBLEMATIC_PID is: $PROBLEMATIC_PID" print "StatsGather>> Collecting the first javacore..."kill -3 $PROBLEMATIC_PIDprint "StatsGather>> First javacore complete."tprof -skex sleep 10&

print "StatsGather>> tprof data complete. Output sent to: sleep.prof"print "StatsGather>> Collecting the second javacore..." kill -3 $PROBLEMATIC_PIDprint "StatsGather>> Second javacore complete."sleep 5print "StatsGather>> Collecting the third javacore..." kill -3 $PROBLEMATIC_PID

4.1.2. Analyzing the output files

The above script will generate ‘sleep.prof’ and a set of java core files, which are mandatory to analyze the issue.

In the above output of tprof command, it is evident that “/usr/java6_64/bin/java” process has a 76.70% of CPU utilization. There are 11 child process IDs associated with java process ID – 4849782.

TID – thread id of each process ID. The Thread IDs associated with each java process is seen to be utilizing 6.5% CPU approximately.

Java core files (Thread dump) generated, will contain the current status/stack trace of threads, which may be processing a piece of code or even they are idle. Each thread in java core file is associated with thread id in hexadecimal format.

Example: First thread id in above figure for java PID 4849782 is 12255519 and its equivalent hexadecimal values is BB011F.The hexadecimal value of the thread id can be used to identify the thread, which costs 6.5% CPU for a database query operation. "WebContainer : 141" daemon prio=3 tid=0x0000000BB011F waiting for monitor entry [0xffffffff4e4fd000] java.lang.Thread.State: BLOCKED (on object monitor)

at com.ibm.ws.classloader.CompoundClassLoader.loadClass(CompoundClassLoader.java:549)- waiting to lock <0x00000006c1c7faf0> (a com.ibm.ws.classloader.CompoundClassLoader)at java.lang.ClassLoader.loadClass(ClassLoader.java:247)at org.apache.xerces.impl.dv.ObjectFactory.findProviderClass(Unknown Source)at org.apache.xerces.impl.dv.ObjectFactory.newInstance(Unknown Source)at org.apache.xerces.impl.dv.SchemaDVFactory.getInstance(Unknown Source)at org.apache.xerces.impl.dv.SchemaDVFactory.getInstance(Unknown Source)at org.apache.xerces.impl.xs.traversers.XSDComplexTypeTraverser.<init>(Unknown Source)at org.apache.xerces.impl.xs.traversers.XSDHandler.createTraversers(Unknown Source)at org.apache.xerces.impl.xs.traversers.XSDHandler.prepareForTraverse(Unknown Source)at org.apache.xerces.impl.xs.traversers.XSDHandler.parseSchema(Unknown Source)at org.apache.xerces.impl.xs.XMLSchemaLoader.loadSchema(Unknown Source)at org.apache.xerces.impl.xs.XMLSchemaLoader.loadGrammar(Unknown Source)at org.apache.xerces.impl.xs.XMLSchemaLoader.loadGrammar(Unknown Source)at org.apache.xerces.jaxp.validation.XMLSchemaFactory.newSchema(Unknown Source)at javax.xml.validation.SchemaFactory.newSchema(Unknown Source)at

com.infosys.feba.framework.controller.handlers.FormatHandlerHelper.validateXMLContentAgainstSchema(FormatHandlerHelper.java:250).

The programmer should sensibly present code and be aware of the costs associated with it. In this example, the thread executing the xml parsing and locked on class loader object for a longer period of time (as compared with next java core). The reason might be a new instance creation of DTM manager and class loading, hence further tuning is anticipated. As this paper is intended to identify the cause for high CPU utilization, hence will not cover the tuning aspects.

4.2. Case study: Analysing a sample J2EE application on HP-UX UNIX servers

A sample workload has been tested on HP-UX Unix operating system (server with 4 core processor) having a single thread running with in a java instance. A sample service based

program is designed to pull – push records between two tables within a database. If no records exist, thread will sleep and notified if new records are ready for processing.

4.2.1. Glance tool:

HP Glance is powerful. It gives a complete picture, highlighting processes that are starved for resources and those using resources. It provides an expert assistance and uses a comprehensive set of rules developed by performance specialists to alert whenever possible.

Command to be executed: /opt/perf/bin/glance -j 25

4.2.2. Analysing the output files

The content of glance output file is as shown below:===========================================================================-----11:11:39gbl cpu utilization = 51%alive processes = 220active processes = 33priority queue = 0.0run queue = 1.2context switch rate = 385network packet rate = 117.0disk phys io rate = 9.1memory pageout rate = 0.0total mi lost buffers = 32448(recommend restart of perftools: midaemon -T; mwa restart)system call rate =1211385process fork rate = 1.5proc completion rate= 0.7highest syscall rate= 1622 for system call getmsgarm compl tran rate = 0.0 Process name pid ktid pri cpu% scall% dsptch cpu-swtch shell-cmd------------ --- ---- --- ---- ------ ------ --------- ---------java 7804 5805720 232 99 45 1950 174 /opt/java6/bin/IA64N/java -DLOjava 14906 545806 137 0 0 34 1 /was/IBM/WebSphere/AppServer/jjava 7804 5805721 168 0 0 554 0 /opt/java6/bin/IA64N/java -DLOjava 14906 542803 137 0 0 25 0 /was/IBM/WebSphere/AppServer/jjava 14906 542697 168 0 0 225 0 /was/IBM/WebSphere/AppServer/jjava 14906 545824 137 0 0 26 1 /was/IBM/WebSphere/AppServer/jjava 7804 5805697 137 0 0 4 0 /opt/java6/bin/IA64N/java -DLOjava 7804 5805714 168 0 0 369 0 /opt/java6/bin/IA64N/java –DLO

The gbl cpu utilization shows how much of the central processing unit's time was spent on various activities during the last interval.

The java process which are running at that instance are shown in the glance output along with process ids (pid) and thread ids (ktid). Glance also highlights the CPU percentile of utilization of each java PID with in gbl cpu utilization. It is recommended to generate java thread dumps for the java process id which has high CPU.In this case, the ‘global cpu utilization’ is 57%, out of which 99% of the CPU has been utilized by a java process id ‘7804’. A set of java thread dumps has been generated for java PID ‘7804’ at a time interval of 5seconds by issuing command ‘kill -3 7804’ in the HP UNIX server.

It is observed that ‘Thread-8’ is in runnable state in all generated thread dumps and stack trace is shown as below:

"Thread-8" prio=7 tid=00d2a800 nid=27 lwp_id=5805720 runnable [21b40000] java.lang.Thread.State: RUNNABLE

at java.lang.Thread.sleep(Native Method)at com.xxxx.xxx.batch.process.InActiveUserThreadManager.run(Unknown Source)at java.lang.Thread.run(Thread.java:662)

Observations: The code snippet for the "Thread-8” execution is as shown below, which runs for infinite loop with asleep of 5 milliseconds.

public void run(){ int sleepTime =5; Thread currentThread = Thread.currentThread(); try{ while (true){

int counter = inActiveUserThreads.size();for (int j = 0; j < counter; j++){

…...…………

}Thread.sleep(sleepTime);

} }catch (InterruptedException interuptExc){

…………………. }}

5. Conclusion

High CPU utilization on production and test environments on UNIX platforms are very common nowadays while working with j2ee applications. Troubleshooting the cause for high CPU utilization with j2ee application on UNIX can be easily carried out with this approach. This paper demonstrates the methodology for identifying the root cause for abnormal CPU utilizations using inbuilt tools from the unix platforms.

References & Appendix http://publib.boulder.ibm.com/infocenter/aix/v7r1/index.jsp?topic=

%2Fcom.ibm.aix.prftools%2Fdoc%2Fprftools%2Fidprftools_profiling_tpofex.htm http://publib.boulder.ibm.com/infocenter/javasdk/tools/index.jsp?topic=

%2Fcom.ibm.java.doc.igaa%2F_1vg0001475cb4a-1190e2e0f74-8000_1006.html https://forums.manageengine.com/topic/glance-command-for-hp-ux http://terminalinflection.com/hpux/hp-perf-monitoring/ http://it-project-work.com/doc/HP-doc/openview/glanceplus_parameter.pdf

Author Biography

THANK YOU!