HP SiteScope WebSphere Solution Template Best Practicesdocshare01.docshare.tips/files/22077/220774163.pdfSiteScope WebSphere Application Server Solution Template To address the needs

HP SiteScope

for the Windows, Solaris and Linux operating systems

Software Version: 10.00

SiteScope WebSphere Application Server Solution Template

Document Release Date: January 2009

Software Release Date: January 2009

2

Legal Notices

Warranty

The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

The information contained herein is subject to change without notice.

Restricted Rights Legend

Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license.

Third-Party Web Sites

HP provides links to external third-party Web sites to help you find supplemental information. Site content and availability may change without notice. HP makes no representations or warranties whatsoever as to site content or availability.

Copyright Notices

© Copyright 2005 - 2009 Mercury Interactive (Israel) Ltd.

Trademark Notices

Adobe® and Acrobat® are trademarks of Adobe Systems Incorporated.

Intel®, Pentium®, and Intel® XeonTM are trademarks of Intel Corporation in the U.S. and other countries.

JavaTM is a US trademark of Sun Microsystems, Inc.

Microsoft®, Windows®, Windows NT®, and Windows® XP are U.S registered trademarks of Microsoft Corporation.

Oracle® is a registered US trademark of Oracle Corporation, Redwood City, California.

Unix® is a registered trademark of The Open Group.

3

Documentation Updates

This guide’s title page contains the following identifying information:

• Software Version number, which indicates the software version.

• Document Release Date, which changes each time the document is updated.

• Software Release Date, which indicates the release date of this version of the software.

To check for recent updates, or to verify that you are using the most recent edition of a document, go to:

http://h20230.www2.hp.com/selfsolve/manuals

This site requires that you register for an HP Passport and sign-in. To register for an HP Passport ID, go to:

http://h20229.www2.hp.com/passport-registration.html

Or click the New users - please register link on the HP Passport login page.

You will also receive updated or new editions if you subscribe to the appropriate product support service. Contact your HP sales representative for details.

4

Support

You can visit the HP Software Support web site at:

http://www.hp.com/go/hpsoftwaresupport

This web site provides contact information and details about the products, services, and support that HP Software offers.

HP Software Support Online provides customer self-solve capabilities. It provides a fast and efficient way to access interactive technical support tools needed to manage your business. As a valued support customer, you can benefit by using the HP Software Support web site to:

• Search for knowledge documents of interest

• Submit and track support cases and enhancement requests

• Download software patches

• Manage support contracts

• Look up HP support contacts

• Review information about available services

• Enter into discussions with other software customers

• Research and register for software training

Most of the support areas require that you register as an HP Passport user and sign in. Many also require a support contract.

To find more information about access levels, go to:

http://h20230.www2.hp.com/new_access_levels.jsp

To register for an HP Passport ID, go to:

http://h20229.www2.hp.com/passport-registration.html

5

HP SiteScope WebSphere Application Server Solution Template

SiteScope’s WebSphere Application Server Solution Template allows you to monitor and troubleshoot WebSphere Application Servers 5.x and 6.x with SiteScope.

This chapter includes:

➤ Introduction on page 5

➤ WebSphere Application Server Performance Overview on page 7

➤ SiteScope WebSphere Application Server Solution Template on page 8

➤ Default Metrics on page 9

➤ Key Performance Indicators and Troubleshooting on page 18

➤ Appendix - Sources on page 26

Introduction

The proliferation of Web-based applications is well-known. Today, many companies have Web-enabled significant numbers of business processes and the trend continues. This proliferation has driven the need for solid Web infrastructure based on open standards. One standard that has proven itself over again is the Java 2 Platform Enterprise Edition or J2EE.


6

6

J2EE has provided a standard for developing component-based, multi-tier Web applications that many companies have embraced and adopted. A key work horse of many J2EE implementations is the application server. Application servers have stepped forward as the workhorse of any Web infrastructure, primarily because they make it fairly simple and easy to implement complex business logic using open interfaces and standardized components.

Because application servers house business logic, which is really a company’s business processes translated into Web-based technologies, it is a critical system in the infrastructure. In an application server is not functioning, it generally means that critical business processes, which can often be related directly to revenue generation, are also not functioning. In additions, because application servers assume the "middleman" role in communication with the presentation layer (Web servers) and the back-end layer (databases, mainframes, ERP, CRM, etc.), ensuring application server performance is paramount.

One application server that has proven its worth is IBM’s WebSphere Application Server. This document describes how HP’s SiteScope product helps organizations ensure the performance of IBM WebSphere Application Servers through the SiteScope WebSphere Application Server Solution Template. This document contains the following information:

➤ An introduction to general WebSphere server performance concepts.

➤ An introduction to the HP SiteScope WebSphere Application Server Solution Template.

➤ A detailed list of the default metrics included in the SiteScope solution template, including why each metric is important.

➤ A "key indicators" section detailing which metrics are most important for uncovering WebSphere performance problems and basic troubleshooting steps that can be taken.


7

7

WebSphere Application Server Performance Overview

The WebSphere Application Server, like other J2EE-compliant application servers, provides all of the standard J2EE services. Therefore, while WebSphere performance depends on many factors, those factors related to J2EE services are the most critical. Below are important WebSphere performance categories:

➤ Enterprise Java Bean (EJB) performance

EJBs are the main workhorses of any J2EE application server. They are the components that implement business logic and represent persistent application entities. EJB response time is the critical performance metric.

➤ Servlets

Servlets act as server-side "applets" and perform many standard Web application tasks. Response time is also important for servlets as are aspects such as number of requests and servlet session manager performance.

➤ Java Virtual Machine (JVM) Runtime

The JVM is abstract computing entity. Similar to a hardware computing machine, it has instruction sets and controls memory. Memory usage and general memory statistics are key performance indicators for the JVM.

➤ ORB container

An object request broker enables communication between disparate objects by taking care of all interoperability issues. Thread metrics and pool sizes are key aspects or ORB performance.

➤ Transaction manager

Transaction managers are responsible for supporting transparent, safe and reliable transactions between J2EE components. Key performance aspects for the transaction manager include transaction duration, concurrent active transactions and transaction timeouts.

➤ Database connections

Database connections are perhaps the hardest working portion of an application server. Access to all back-end data takes place via these connections. Important performance metrics include connection pool sizes and time executing and waiting for transactions.


8

8

SiteScope WebSphere Application Server Solution Template

To address the needs of WebSphere Application Server performance monitoring, HP has built a WebSphere Application Server solution template. A solution template is a pre-built collection of monitors that consists of default metrics, default thresholds (where applicable) and proactive tests (where applicable) all of which have been researched using best practice data and expertise from various sources including HP’s administrators, HP’s professional services organization, technical documentation, books from industry experts and IBM’s own collateral and documentation.

The purpose of a solution template is to provide easy-to-use, comprehensive monitoring without requiring the SiteScope user of the iT organization to be experts on the application. Benefits of the WebSphere solution template include:

➤ Reduces the need for WebSphere server performance domain expertise

➤ Reduces the time to configure and deploy WebSphere Application Server monitors

➤ Helps identify both real-time performance bottlenecks and longer term trends

➤ Adds no overhead to production systems

➤ Supports WebSphere Application Servers 5.x and 6.x (there is a separate solution template for each version)


9

9

Default Metrics

Perhaps the greatest value in the SiteScope WebSphere Application Server solution template is the built-in-list of default metrics and thresholds. These default metrics are heavily based on IBM’s monitoring recommendations which can be found in various locations including the IBM corporate Web site. Below are tables listing the most critical WebSphere metrics that are enabled as defaults in the solution template.

Note: In addition to the metrics listed below, WebSphere utilizes several log files that should be monitored. These log files can be monitored via a standard SiteScope log file monitor. Important log files include:

➤ Systemount.log (located in installation_directory/logs/applicationServerName)

➤ Systemerr.log (located in installation_directory/logs/applicationServerName)

➤ Activity.log (located in installation_directory/logs/applicationServerName)

WebSphere Application Server MetricsNum

Counter/Metric Description

EJB General

Num Instantiates The number of times that bean objects were instantiated.

Num Destroys The number of times that bean objects were freed.

Concurrent Lives The number of concurrently live beans.

Total Method Calls The total number of method calls.

Avg. Method Rt (ms) The average time in milliseconds on the bean methods (home, remote, local).


10

10

Active Methods The number of concurrently active methods - the number of methods called at the same time.

Entity EJB Performance


Drains from Pool The number of times that the daemon found the pool was idle and attempted to clean it (entity and stateless).

Num Removes The number of times that beans were removed.

Returns Discarded The number of times that the returning object was discarded because the pool was full (entity and stateless).

Gets Found The number of times that a retrieve found an object available in the pool (entity and stateless).

Avg Method Rt The average response time in milliseconds on the bean methods (home, remote, local).

Num Stores The number of times that bean data was stored in persistent storage (entity).


Gets From Pool The number of calls retrieving an object from the pool (entity and stateless).

Num Creates The number of time that beans were created.

Num Passivates The number of times that beans were passivated (entity and stateful).

Returns To Pool The number of calls returning an object to the pool (entity and stateless).

Num Activates The number of times that beans were activated (entity and stateful).


Num Loads The number of times that bean data was loaded from persistent storage (entity).



11

11



Avg Drain Size The average number of objects discarded in each drain (entity and stateless).

Pool Size The number of objects in the pool (entity and stateless).

Stateful Session EJB Performance


Num Removes The number of times that beans were removed.



Num Creates The number of times that beans were created.

Num Passivates The number of times that beans were passivated (entity and stateful).

Num Activates The number of times that beans were activated (entity and stateful).




Stateless Session EJB Performance




12

12

Drains From Pool The number of times that the daemon found the pool was idle and attempted to clean it (entity and stateless).












Message Driven EJB Performance


Drains From Pool The number of times that the daemon found the pool was idle and attempted to clean it (entity and stateless).



13

13












Database Connections

Num Creates Total number of connections created.

Num Destroys Total number of connections destroyed.

Num Returns Total number of connections returned.

Num Allocates Total number of connections allocated.

Pool Size The size of the connection pool.

Concurrent Waiters The number of threads that are currently waiting for connections.



14

14

Avg. Wait Time (ms) The avg. waiting time in milliseconds until a connection is granted.

Faults The total number of faults, such as timeouts, in the connection pool.

Percent Used The average percent of the pool that is in use.

Percent Maxed. The average percent of the time that all connections are in use.

PrepStmt Cache Discards

The total number of statements discarded by the least recently used (LRU) algorithm of the statement cache.

JVM Runtime

Total Memory (bytes) The total memory in the JVM run time.

Free Memory (bytes) The free memory in the JVM run time.

Used Memory (bytes) The used memory in the JVM run time.

Servlet Session Manager

Created Sessions The number of sessions that were created.

Invalidated Sessions The number of sessions that were invalidated.

Session Lifetime The average session life time in milliseconds (time invalidated - time created).

Active Session The number of concurrently active sessions. A session is active if the WebSphere Application Server is currently processing a request that uses that session.

Live Sessions The number of sessions that are currently cached in memory.

ORB Container Thread Pool

Thread Creates The total number of threads created.

Thread Destroys The total number of threads destroyed.



15

15

Active Threads The number of concurrently active threads.

Pool Size The average number of threads in pool.

Percent Maxed The average percent of the time that all threads are in use.

Web Container Thread Pool

Thread Creates The total number of threads created.

Thread Destroys The total number of threads destroyed.

Active Threads The number of concurrently active threads.

Pool Size The average number of threads in pool.

Percent Maxed The average percent of the time that all threads are in use.

Transaction Manager

Global Trans Begun The total number of global transactions started on the server.

Global Trans Involved The total number of global transactions involved on the server (for example, begun and imported).

Local Trans Begun The total number of local transactions started on the server.

Active Global Trans The number of concurrently active global transactions.

Active Local Trans The number of concurrently active local transactions.

Global Trans Duration The average duration of global transactions.

Local Trans Duration The average duration of local transactions.

Global Before Completion Duration

The average duration of before_completion for global transactions.

Global Prepare Duration

The average duration of prepare for global transactions.



16

16

Global Commit Duration

The average duration of commit for global transactions.

Local Before Completion Duration

The average duration of before_completion for local transactions.

Local Commit Duration The average duration of commit for local transactions.

Num Optimizations The number of global transactions converted to single phase for optimization.

Global Trans Committed

The total number of global transactions committed.

Local Trans Committed The number of local transactions committed.

Global Trans RolledBack

The total number of global transactions rolled back.

Local Trans RolledBack The number of local transactions rolled back.

Global Trans Timeout The number of global transactions timed out.

Local Trans Timeout The number of local transactions timed out.

Web Applications

Num Loaded Servlets The number of loaded servlets.

Number Reloads The number of reloaded servlets.

Total Requests Total number of requests that a servlet processed.

Concurrent Requests The number of requests that are concurrently processed.

Response Time (ms) The response time, in milliseconds, of a servlet request.

Num Errors Total number of error in a servlet or JavaServer Page (JSP).

Servlets

Total Requests Total number of requests that a servlet processed.



17

17

Note: Depending on the version and configuration of WebSphere being monitored, some of the metrics may differ or be unavailable.

Num Errors Total number of errors in a servlet or JavaServer Page (JSP).

Concurrent Requests The number of requests that are concurrently processed.

Response Time The response time, in milliseconds, of a servlet request.



18

18

Key Performance Indicators and Troubleshooting

The goal of any monitoring solution is to help administrators avoid or at the very least identify performance problems before they affect large numbers of users. SiteScope solution templates include key metrics that can act as early indicators of performance problems. This section provides a list of key performance indicators of WebSphere Performance and describes a basic process for troubleshooting performance issues. Apart from checking standard system performance metrics such as CPU and memory utilization and Web server threads, the process of troubleshooting WebSphere performance problems relates to four key areas:

➤ EJBs

➤ Servlets

➤ Thread pools

➤ Database connection pools

➤ JVM heap

Note: This section does not include all of the detail necessary to uncover the root cause of WebSphere performance problems. Information is provided in these sections to help you take the first steps in troubleshooting. For more information on troubleshooting or tuning situations, it is recommended to visit the IBM WebSphere information center at: http://publib.boulder.ibm.com/infocenter/ws51help/index.jsp.

http://publib.boulder.ibm.com/infocenter/ws51help/index.jsp


19

19

EJBsSince EJBs are a critical workhorse component of any Web application, their performance is a direct indicator of overall Web application performance. The SiteScope WebSphere Application Server solution template contains many metrics related to EJB performance. The following table specifies the most important and useful metrics.

It should be noted that most Web applications contain hundreds if not thousands of EJBs. It is therefore important to have some knowledge of the Web application to be monitored in order to identify which EJBs are most important to be monitored. Once the critical EJBs have been identified, use the following metrics for ongoing performance monitoring.

Metric What to Look For Troubleshooting

Entity and Stateful EJBs

➤ Num Passivates

➤ Num Activates

If these metrics are exhibiting high values, then the EJB container is changing between a beans’s active (in use) statue and uts passive (stored in the disk) state.

Consider increasing the EJB cache size.

All EJBs

➤ Avg. Method RT

This metric should be baselined for critical methods within EJBs and compared over time. Start with methods within EJBs that access back-end systems such as databases and mainframes.

High avg. method response times usually indicates inefficient code. Contact the Web application developers.


20

20

Timeout Ratio

➤ Lock Manager Timeout Total Count

➤ Lock Manager Access Count

(applies to entity and stateful\session beans)

By dividing the Lock Manager Timeout Total Count by the Lock Manager Access Count, a ratio is provided that gives an indication of the number of times a thread timed out while waiting for a lock on an EJB.

A high ratio indicated a high occurrence of timeouts that can cause exceptions and transaction rollbacks, Check the "transaction timeout" value and consider increasing it. Increasing this metric too high can cause server threads to wait too long and may increase request times.

Transaction Timeout Ratio

➤ Transaction Timed Out Total Count

➤ Transactions Committed Total Count

By dividing the Transactions Timed Out Total Count by the Transactions Committed Total Count, a ratio is provided that indicates how often threads are timing out waiting for a transaction to complete.

A high timeout ratio generally means the "transaction timeout" setting is too low. Be careful to adjust this in increments because too high of a setting can cause other bottlenecks.



21

21

ServletsAnother important component of Web-based applications is servlets. Servlet performance often affects presentation layer responsiveness and can also affect overall application response time. The following table specifies key servlet metrics.


Num Errors Servlet errors can indicate anything from a problem serving graphics or text to problems with plug-ins and security. Ideally, this metric should be 0.

1 View the error code that is generated in the browser.

2 Search on "servlet errors" at the following URL: http://publib.boulder.ibm.com/infocenter/ws51help/index.jsp

Total Requests This metric indicates how many requests a servlet processes. By baselining its value and comparing it to current performance, it is possible to track whether or not a servlet capacity is staying high. A drop in total requests indicates that the servlet is struggling to process requests.

Small drops in total requests are good indicators of potential future performance problems. Large drops indicate current problems. Evaluating load on the application server in terms of numbers of users is a good first troubleshooting step. Investigating dynamic caching is also a good next step.

Response Time This metric also needs to be baselined but a good rule of thumb is that response time shouldn’t be greater than 8 seconds.

Many times, servlet response time degrades because of problems accessing back-end systems. Check database connection pool performance as a first step. Also, check to make sure that EJB performance is ok since many times, servlets rely on EJBs.


22

22

Thread PoolsKey containers such as the Object Request Broker and Web Application containers contain pools or resources such as threads and connections for processing requests. Creating and destroying threads is an expensive process and therefore can greatly affect performance. Monitoring thread pool activity in order to ensure that thread settings are optimal is therefore an important aspect to ensuring optimal performance. The following table outlines important metrics for both the Web and ORB container thread pools.


➤ Thread Creates

➤ Thread Destroys

A significant number of threads creates and destroys indicates that the thread pool does not have enough allocated threads. Creation and destruction of threads can include a significant amount of overhead and negatively affect performance.

In conjunction with evaluation the Percent Maxed metric below, consider increasing the thread pool size by increasing the "maximum number if threads to allow".

Percent Maxed Since this metric specifies the percentage of time that all threads are in use, it is a good indicator of how much head room exists in thread pools. If this metric is consistently in double digits, the application server may be approaching a performance slowdown scenario.

In conjunction with evaluating the Thread Creates and Destroys, consider increasing the thread pool size by increasing the "maximum number of threads to allow".


23

23

Database Connection PoolsDatabase connections are perhaps the single most likely contributor to applications server performance problems. Application server performance can and does slow down due to time waiting on database processing or inefficient connections to data stores. This section provides metrics that help identify if these types of situations are occurring and what to do about them.

Pool


Concurrent Waiters

Ideally, this counter should never be greater than 0. If it is, it means that Web applications are waiting for a connection to a back-end data store.

Consider increasing the connection pool "maximum connections" value if CPU load is not close to 100%. If CPU load is high, this is most likely the problem.

Percent Used If this percentage is low under normal application usage, this indicates that the connection pool may be too big.

Consider decreasing the connection pool "maximum connections" value.


24

24

Pool Size This metric can help administrators determine if their back-end database is configured for an appropriate number of connections. A situation can occur where the pool size has been increased past what the database was configured to handle.

Use this value to ensure that number of connections that the back-end database is configured to support is at least equal to the WebSphere database connection pool size. Of these are not equal, applications will fail and issue SQL exception errors.

Avg. Wait Time

It may not always be possible to obtain a balance between "maximum connections" and "used connections" such that "concurrent waiters" equals 0. If this is the case, there will be some wait time for a connection. Tracking/baselining "avg. wait time" can help identify if wait time is getting abnormal.

Baseline this metric and then determine if current values are out of ordinary. If wait times are growing, it is a good indication that the pool size should be increased. If wait times are within norms but significant "Connection Wait Timeout Exceptions" are being thrown, then an opportunity exists to tune the "connection timeout" value to wait a little longer.



25

25

JVM HeapRuntime memory usage is a critical aspect of WebSphere Application Server health because object creation and destruction are memory intensive processes and require sufficient memory allocation. By monitoring a couple of critical JVM heap (memory) counters, it is possible to keep a close eye on this critical resource. The following table specifies important JVM heap metrics.


➤ Free Memory (bytes)

➤ Used Memory (bytes)

There are three main aspects to check regarding these to metrics:

➤ If memory is increasing at a much higher rate than free memory, this is a likely indication of a memory leak.

➤ Check to see if free memory is 85% or more.

➤ Compare memory consumption patterns with the performance of critical EJBs and Servlets over time.

Memory leak situations require an in-depth analysis of the Web application code. It may be necessary to utilize tools such as HP’s Deep Diagnostics for J2EE to troubleshoot at the method and call levels.

85% free memory indicates a maximum heap size value that is too large which usually means the application server is under utilizing the memory allocated for the heap.

Longer-term memory usage patterns that coincide with slow EJB or Servlet response times can indicate inefficient code ot component communication issues.


26

26

Appendix - Sources

The following sources were used in developing this best practice guide. It is highly recommended that these sources be consulted in the process of troubleshooting and tuning WebSphere application servers.

IBM Web Site

http://www-306.ibm.com/software/webservers/appserv/was/performance.html


http://www.redbooks.ibm.com/abstracts/sg246198.html?Open

http://www-306.ibm.com/software/webservers/appserv/was/performance.html

http://www.redbooks.ibm.com/abstracts/sg246198.html?Open


Documents

HP SiteScope WebSphere Solution Template Best Practicesdocshare01.docshare.tips/files/22077/220774163.pdfSiteScope WebSphere Application Server Solution Template To address the needs