Performance Tuning for Business Service Management€¦ · Performance Tuning for Business Service Management Problem statement Initially, any performance tu ning effort requires

www.bmc.com

White Paper

Performance Tuning for Business Service Management

October 2007

If you have comments or suggestions about this documentation, contact Information Design and Development by email at [email protected].

Contacting BMC Software

You can access the BMC Software website at http://www.bmc.com. From this website, you can obtain information about the company, its products, corporate offices, special events, and career opportunities.

United States and Canada

Address BMC SOFTWARE INC2101 CITYWEST BLVDHOUSTON TX 77042-2827 USA

Telephone 713 918 8800 or800 841 2031

Fax 713 918 8000

Outside United States and Canada

Telephone (01) 713 918 8800 Fax (01) 713 918 8000

© Copyright 1991–2007 BMC Software, Inc.

BMC, BMC Software, and the BMC Software logo are the exclusive properties of BMC Software, Inc., are registered with the U.S. Patent and Trademark Office, and may be registered or pending registration in other countries. All other BMC trademarks, service marks, and logos may be registered or pending registration in the U.S. or in other countries. All other trademarks or registered trademarks are the property of their respective owners.

DB2 is a registered trademark of International Business Machines Corporation.

IBM is a registered trademark of International Business Machines Corporation in the United States, other countries, or both.

ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.

Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.

Oracle is a registered trademark of Oracle Corporation.

UNIX is a registered trademark of The Open Group.

Java is a trademark or registered trademark of Sun Microsystems, Inc., in the U.S. or other countries.

BMC Software considers information included in this documentation to be proprietary and confidential. Your use of this information is subject to the terms and conditions of the applicable End User License Agreement for the product and the proprietary and restricted rights notices included in this documentation.

Restricted rights legendU.S. Government Restricted Rights to Computer Software. UNPUBLISHED -- RIGHTS RESERVED UNDER THE COPYRIGHT LAWS OF THE UNITED STATES. Use, duplication, or disclosure of any data and computer software by the U.S. Government is subject to restrictions, as applicable, set forth in FAR Section 52.227-14, DFARS 252.227-7013, DFARS 252.227-7014, DFARS 252.227-7015, and DFARS 252.227-7025, as amended from time to time. Contractor/Manufacturer is BMC Software, Inc., 2101 CityWest Blvd., Houston, TX 77042-2827, USA. Any contract notices should be sent to this address.

http://www.bmc.com

mailto:[email protected]

Customer Support

You can obtain technical support by using the Support page on the BMC Software website or by contacting Customer Support by telephone or email. To expedite your inquiry, please see “Before Contacting BMC Software.”

Support website

You can obtain technical support from BMC Software 24 hours a day, 7 days a week at http://www.bmc.com/support_home. From this website, you can:

■ Read overviews about support services and programs that BMC Software offers.■ Find the most current information about BMC Software products.■ Search a database for problems similar to yours and possible solutions.■ Order or download product documentation.■ Report a problem or ask a question.■ Subscribe to receive email notices when new product versions are released.■ Find worldwide BMC Software support center locations and contact information, including email addresses, fax

numbers, and telephone numbers.

Support by telephone or email

In the United States and Canada, if you need technical support and do not have access to the Web, call 800 537 1813 or send an email message to [email protected]. (In the Subject line, enter SupID:<yourSupportContractID>, such as SupID:12345.) Outside the United States and Canada, contact your local support center for assistance.

Before contacting BMC Software

Have the following information available so that Customer Support can begin working on your issue immediately:

■ Product information

— Product name— Product version (release number)— License number and password (trial or permanent)

■ Operating system and environment information

— Machine type— Operating system type, version, and service pack— System hardware configuration— Serial numbers— Related software (database, application, and communication) including type, version, and service pack or

maintenance level

■ Sequence of events leading to the problem

■ Commands and options that you used

■ Messages received (and the time and date that you received them)

— Product error messages— Messages from the operating system, such as file system full— Messages from related software

http://www.bmc.com/support_home


License key and password information

If you have a question about your license key or password, contact Customer Support through one of the following methods:

■ E-mail [email protected]. (In the Subject line, enter SupID:<yourSupportContractID>, such as SupID:12345.)

■ In the United States and Canada, call 800 537 1813. Outside the United States and Canada, contact your local support center for assistance.

■ Submit a new issue at http://www.bmc.com/support_home.


http://www.bmc.com/support_home

White Paper


This paper outlines best practices for Business Service Management (BSM) performance tuning. The following topics are provided:

“Introduction” on page 6“Problem statement” on page 7“Problem triage” on page 9“Tuning the mid tier” on page 12“Tuning the database server” on page 14“Tuning the AR System server” on page 19“Overall tuning strategy” on page 21“Performance tuning checklist” on page 22

Performance Tuning for Business Service Management 5

White Paper

IntroductionBusiness Service Management (BSM) can dramatically simplify a business's IT environment by providing process automation and service management solutions. The critical role the BSM product suite plays, however, makes performance and scalability of vital concern. Though BMC Software developers substantially benchmark and improve BSM products before they are released, understanding how best to tune a given environment is still useful.

This paper outlines best practices for BSM performance tuning. Though it does not cover all environments and configurations, it provides guidance for several cases typical of existing BMC Software enterprise customers. Specifically, it explains how to evaluate an entire configuration running BSM applications and how to identify a potential bottleneck that might cause a performance issue. It also suggests methods of collecting and interpreting detailed diagnostics that can help identify the root cause of the performance issue.

Product and configuration focusTo reduce the scope of this subject, this white paper focuses primarily on the Oracle database. Other databases are reasonable choices for an enterprise environment, but Oracle provides robust diagnostic utilities that this document takes advantage of.

With respect to products, any BSM environment should roughly align with an ITIL-compliant product set, which includes a configuration management database (CMDB), automated discovery sources, service impact management and service level management tools, and functionality such as incident, problem, and change management. Therefore, this paper considers several products based on the BMC Remedy Action Request System (AR System) server, such as BMC Atrium CMDB and BMC Remedy IT Service Management Suite, as well as products such as BMC Service Impact Manager and BMC Configuration Discovery. The majority of the techniques discussed in this paper, however, are generic best practices that can be applied to any product set.

6 Performance Tuning for Business Service Management


Problem statementInitially, any performance tuning effort requires a clear problem statement. System performance has two common metrics that should be included in the statement:

Throughput—Frequency of an operation over time, for example, the number of tickets created per hour or configuration items (CIs) processed per second.

Response time—Seconds between a significant operation (for example, clicking Save to create an incident ticket) and the observed time that the operation is completed and control of the system is returned to the user. Response time can be described subjectively, such as “the system seems slow,” but to identify the problem effectively, you should evaluate it more objectively. This is commonly done by having multiple users with different client configurations manually time the operations in question. See the following section, “Collecting response-time data.”

In addition, the problem statement should include a list of recent changes that might have led to the current state. If the problem has always existed, this is not relevant. But in production-related issues, change control or other sources can often provide a log of the system changes that took place right before the problem began. Such change history might not point to the root cause of the problem, but it provides essential background nonetheless.

Collecting response-time dataTo measure response time, it is often useful to collect times from various users who are experiencing the problem. Be sure to record each user’s client configuration and location. It is important to take multiple timings from each user because caching effects might cause the first test run to be longer than subsequent test runs.

Figure 1-1 shows an example of a spreadsheet for capturing such data. Note that each test should be specified with enough detail to tell testers exactly what to do and what to time. Timing can be done with a tool or a stopwatch. All testers should use a similar timing device.

Problem statement 7

White Paper

Figure 1-1: Sample spreadsheet for collecting response times

Best practices for problem statementsWhen creating a problem statement, use this checklist:

For a checklist that includes all performance tuning issues, see “Performance tuning checklist” on page 22.

Characterize throughput or response time Before problem was observed

After problem was observed

List any system changes that took place soon before the problem was observed

Patch changes

Configuration changes

Workload changes

User count changes



Problem triageInitial triage of performance problems involves observing the resource consumption across the configuration. Such resource profiles can point to the root cause of the problem, or they can simply be another useful piece of diagnostic information that helps focus the search for a cause.

For each tier of the system, consumption of these resources should be analyzed:

Central processing units (CPUs)—Include system time, user time, and wait I/O time. For benchmarks simulating online users, get CPU usage data for the machine or machines driving the load. See “Monitoring CPU consumption” on page 9.

Memory—See “Monitoring memory consumption” on page 11.

Monitoring CPU consumptionCPU usage substantially affects the response time of a system, usually in a nonlinear way. Figure 1-2 shows an example of a typical response-time curve: response time starts at a baseline and grows very slowly while CPU use is low; as CPU use increases, the response-time slope becomes steeper and steeper until it is almost vertical.

Figure 1-2: Typical response-time curve

Well-behaved response-time growth is expected until the knee of the curve is reached. The point at which the knee is reached varies, but it is generally when CPU use for at least one machine in the system is at least 60% and sometimes as high as 90%. Generally, running just one server at high load in a multiserver configuration can substantially affect response time, so having some servers in the configuration largely idle does not guarantee good response time.

If the knee of a configuration's response-time curve is similar to the knee shown in Figure 1-2 but occurs while CPU consumption on all systems is still low, the consumption of some resource other than a CPU is becoming the bottleneck of the system.

CPU Use

Resp

onse

Tim

e

Problem triage 9

White Paper

CPU time has three components:

User time—Time consumed by applications performing useful activity on the system. Generally, user time should be the vast majority of the time observed.

System time—Time consumed by the operating system (OS) kernel for core processing. If system time is higher than 5%, that indicates that an OS-level resource or locking problem exists, which is typical in memory management issues.

Wait I/O time—Time consumed waiting for a request to the I/O subsystem to return. If wait I/O time becomes substantial, the I/O subsystem of the configuration cannot keep up with the CPU processing of the application. The system will likely have poor response time unless the application requirements for data are tuned or the I/O subsystem bandwidth is improved. Generally, I/O concerns are only relevant on the database system of a multitiered configuration.

Typical tools used to track CPU consumption are perfmon for Windows and sar for UNIX and Linux. In both cases, the data collected should be stored for consideration and interpretation later. For UNIX systems, top is useful for identifying high-load processes, but its data is harder to store and review later, so ps might be a useful alternative. Although overall CPU use is the metric of primary interest, knowing the relative CPU consumption among the various running processes is also helpful, and top can be particularly useful for that.

Best practices for monitoring CPU consumptionWhen monitoring CPU consumption, use this checklist:


Collect CPU usage data for each machine in the configuration.

Make sure less than 70% of the total CPU capacity is used on every system in the configuration.

Note which processes are consuming the majority of the CPU resources.

Note whether wait I/O time is a substantial component of CPU use. (This indicates that theI/O subsystem is not keeping up with demand.)



Monitoring memory consumptionTracking memory consumption is not always a straightforward task because some systems assign unused memory to swap space or to other system resources and because memory allocation algorithms vary between operating systems. Understanding roughly how much of a configuration's physical memory is in use during application execution is important, however. Modern operating systems always provide a virtual memory space that far exceeds the system's physical memory. Thus, if an application consumes nearly all of the system's physical memory, processes in shared memory are written to disk to free up space for new memory requirements. This practice is informally called “swapping.” Although swapping is useful, from a performance perspective it is never ideal because it degrades both CPU consumption and response time. (Do not confuse process swapping with simple memory paging, which can be normal system behavior and does not necessarily consume extra resources.)

Therefore, the primary goal of evaluating memory consumption is to make sure that the system is not swapping (or nearly swapping) and that the application can still allocate memory when needed. Given that swapping causes CPU consumption, the first symptom of running out of memory might be that CPU resources are fully used. Observing how memory consumption grows over time is often useful. A process that continues to consume memory quickly over time might have a memory leak, which is likely to lead eventually to swapping or failure.

When considering the impact of memory consumption, take into account whether the OS and applications are 32 bit or 64 bit. For a given process, a 32-bit OS is generally limited to 2 gigabytes (2 GB) or 4 GB of shared memory, though some tricks in Linux and Windows might allow slightly more memory for a single process. A 64-bit OS can have essentially unlimited physical memory, but the process footprint of 32-bit applications (such as many of the BMC Remedy products discussed in this paper) is limited to 2 GB or 4 GB. Therefore, if memory consumption for a process on the application tier (called the AR System server tier in BMC Remedy products) is approaching 2 GB, its growth cannot continue much beyond the current level, even if the overall memory consumption on such a 64-bit OS is low.

Best practices for monitoring memory consumptionWhen monitoring memory consumption, use this checklist:


Make sure no system in the configuration is running out of physical memory and swapping.

Be aware of 64-bit and 32-bit limitations on both the OS and applications.

Track memory growth over time to identify any memory leaks.

Problem triage 11

White Paper

Tuning the mid tierThe web client for the AR System server is BMC Remedy Mid Tier (the mid tier). This section provides basic directions for getting the best use out of the mid tier.

Comparing default web serversThe mid tier can be implemented with a wide range of web servers, as specified in the AR System server compatibility matrix. By default, however, AR System 7.0 uses the Internet Information Services (IIS) web server and the New Atlanta ServletExec servlet engine. In the recently released AR System 7.0.01, Apache Tomcat replaced ServletExec as the default that ships with the product. One advantage of this change is that a licensed third-party product such as ServletExec does not need to be introduced into a customer's production environment. In terms of performance, the Tomcat servlet engine running with the Apache web server seems to be a stable and efficient solution. Figure 1-3 compares response times of these combinations:

IIS and ServletExec

IIS and Tomcat

Apache and Tomcat

As the figure shows, by moving to Apache Tomcat, modest gains in response time with roughly the same or slightly lower CPU consumption are achieved.

Figure 1-3: Comparison of web server response time

The benchmark in Figure 1-3 consists of a variety of BMC Remedy IT Service Management (ITSM) 6.0 self-service use cases run with moderate load on the 7.0.01 mid tier.

0.6

0.5

0.4

0.3

0.2

0.1

0

OverallResponse

Time

CreateCHG

Requester

OpenSupportConsole

CreateHPD

Requester

ModifyHPD

Requester

OpenCHG

Support

OpenHPD

Support

OpenRequesterConsole

OpenHPD

Support

IIS & ServletExec

Apache & Tomcat

IIS & Tomcat

Seco

nds



Mid-tier tuningFirst, be aware that the mid tier depends heavily on caching to achieve reasonable response times. Whenever the web server is shut down and restarted, system response time can be up to a minute for the first access to an uncached form. Caching is based on role and access control group, so different user types might need to recache form views in the mid tier separately. In other words, if an incident support user accesses a form and then a change support user accesses the form a bit later, the correct form view might not be cached for the change support user. This can occur because workflow is based on groups, and workflow is associated with a form view, so it is not possible to cache a generic view of a form that works for all groups. To avoid large cache start-up latencies, use the precache.xml utility to precache forms on the web server.

Only a few settings have impact:

Mid-tier caching—By default, the client frequently checks with the mid tier to see whether there are updates for any of the forms being used. Such updates are rare and ideally are done only in batch windows or during system downtime, so you can reduce the frequency of checking. In particular, in the Cache Settings page of the Mid Tier Configuration Tool, do the following:

Clear the Perform Check check box.

Set the Resource Check Interval to 86400 seconds (one day).

Mid-tier logging—In the Log Settings page of the Mid Tier Configuration Tool, set Log Level to Severe to reduce unnecessary logging.

Java Virtual Machine (JVM) heap size—BMC Software recommends these heap settings:

Minimum heap size should be 512 megabytes (MB).

Maximum heap size should be 1 GB.

This provides enough memory for substantial user counts (typically 200–300 for a 2 CPU, 4 GB mid tier) but will not risk swapping in your system. Systems with less than 2 GB of RAM, however, might have issues with such settings.

ServletExec idle time-out—When IIS is used with ServletExec, the ServletExec idle time-out can cause recaching on the mid tier (this does not occur when IIS is used with Tomcat). As previously discussed, such recaching is very expensive. To avoid this, clear the Idle Timeout option in the IIS administrative form as follows:

1 Choose Control Panel > Administrative Tools > Internet Information Services (IIS) Manager.

2 In the Console Tree, right-click Application Pools.3 Choose Properties.4 Go to the Performance tab.5 Clear the “Shutdown worker processes after being idle for (time in minutes)” check

box.

Tuning the mid tier 13

White Paper

Mid-tier configuration file—In the config.properties file, these parameters can limit connections in some cases:

arsystem.pooling_max_total_connections—Set to 2000 to ensure that the maximum connections limit is not reached.

arsystem.pooling_max_connections_per_server—For high-load systems (over 300 users), set to 500. This connections-per-server value is related to session pooling on the mid tier, so the default value 80 should be sufficient for fewer than 300 users because those users share a smaller number of mid-tier connections.

Best practices for tuning the mid tierWhen tuning the mid tier, use this checklist:


Tuning the database serverThis section provides the following information for the most common database in use today, Oracle:

Tips to configure your AR System database server for optimal performance

Diagnostic techniques for understanding and resolving issues

A similar methodology should also be effective for other databases, though the commands and syntax differ.

Use precaching to avoid high latency for first-time access to a form.

Set resource_check_interval to one day.

Set minimum heap size to 512 MB and maximum heap size to 1 GB.

Set pooling_max_total_connections to 2000.



Initial database configurationThe machine in your BSM configuration that requires the largest amount of I/O bandwidth is the AR System database server. External storage is strongly recommended for this system because local disks might fail to keep up with the demands of a substantial implementation, and any I/O bottleneck limits overall throughput and increases the system’s response time.

The System Global Area (SGA) of an Oracle database determines the amount of cache available for both SQL statements and data. Recommended values for small, medium, and large databases are shown in the following table. In systems running a 32-bit operating system (such as Linux or 32-bit Windows), only the small configuration fits within the limits of the operating system.Table 1-1: Recommended values for selected Oracle 10g parameters and sizing

* This parameter is necessary only in Oracle 9i. It is obsoleted by the sga_target setting in Oracle 10g. In Oracle 10g, the components of the SGA are automatically sized according to demand, so only sga_target and pga_aggregate_target are needed to size the Oracle SGA and PGAs (Program Global Areas) fully. For pga_aggregate_target to work, the parameter work_area_policy should be set to AUTO, the default.

Perhaps the most important parameter setting in Oracle for BMC Remedy products is cursor_sharing. Cursor sharing enables SQL code that does not use bind variables—such as BMC Remedy product code—to be reused as if it does use bind variables. Typically, cursor_sharing is set to force for 9i databases and to similar for 10g databases. The setting is most important for 10g databases, where the default is exact, which essentially disables the feature and means nonbind variable SQL is generally not reused.

System components Database size

Small Medium Large

Suggestions

CPUs (typical configuration) 2 4 8Memory (typical configuration) 4 GB 8 GB 16 GBConcurrent users 1–400 401–800 800 or more

Requirements

sga_target 1–2 GB 3–4 GB 6–8 GBdb_cache_size * 1.5 GB 3.2 GB 6 GBshared_pool_size * 500 MB 800 MB 2 GBshared_pool_reserved_size * 50 MB 100 MB 100 MBpga_aggregate_target 1 GB 2 GB 4 GBSessions 300 500 1000Processes 300 500 1000Temporary tablespace 1 GB 1–2 GB 2–4 GBLog files 3

(500 MB each)3

(500 MB each)3

(500 MB each)

Tuning the database server 15

White Paper

Oracle optimizersOracle provides two optimizers to generate SQL execution plans:

Cost-based optimizer (CBO)

Rule-based optimizer (RBO)

Oracle 10g and the CBOIn Oracle 10g, the RBO has essentially gone away, and only the CBO can be used. The CBO requires database statistics to be gathered because it uses them to determine the right SQL plan. In 10g, by default, statistics are automatically gathered during the maintenance window (between midnight and 6:00 a.m.).

When used with 10g, BMC Remedy products execute with the CBO, so if you do not change the default settings, statistics are automatically gathered every night. To see when statistics were last gathered, query the database. If it was not recently, make sure the automatic gathering of statistics is enabled. If it is, you might have poor SQL execution plans and hence inefficient database processing.

The only time you might need to gather statistics manually in a 10g database is during batch jobs that begin with an empty destination table. Statistics on such a table reflect that it is empty. If the table subsequently becomes very large, operations against the table might begin to perform poorly because the statistics no longer reflect its true size. If this occurs, you can gather statistics during the execution of the batch job. This needs to be considered only for batch operations that take hours rather than minutes.

Oracle 9i and the RBOIf you use BMC Remedy products with Oracle 9i, BMC Software recommends that you do not gather statistics. Therefore, continue to use the RBO.

NOTE The majority of QA and performance testing using the 9i database was done with RBO, though some customers have successfully migrated to CBO with 9i. Overall, use of SQL in BMC Remedy products has a low-to-modest level of complexity, so substantial execution plan changes are not expected when you move between RBO and CBO.



Oracle database diagnosticsTypically, Oracle diagnostics come in the form of a report called statspack in 9i and AWR in 10g. The reports are essentially analogous. To create a report, a snapshot is taken before and after the period of interest; the report is then generated to show how the system counters (V$ tables) changed between the two snapshots. In 10g, AWR snapshots are automatically taken every hour. In general, reports are most valuable when they focus solely on a period of interesting activity. When other less interesting activity is included in a report, it dilutes the essence of the interesting activity.

The reports show a high-level summary of system usage followed by specific observed wait events and then by a list of high-load SQL statements. How to interpret such reports is well documented in Oracle literature, but here are a few guidelines:

First, make sure that both buffer cache and shared pool are being used well.

If the buffer cache hit ratio is below 98%, the buffer cache might be too small.

A section near the top of the report addresses shared pool usage. If the amount of consumed shared pool memory is above 80% and the use of statements is low (for example, less than 40% of memory is used by statements executed more than once), you are not reusing SQL statements well. This could be due to cursor_sharing being set to exact.

Best practice is to set cursor_sharing to force for 9i applications and to similar for 10g applications. For AR System server–based applications, the cursor_sharing setting in the ar.cfg file must match the setting in the init.ora file (see the white paper Oracle's Cursor Sharing for BMC Remedy Products).

Overall sizing for buffer cache and shared pool in Oracle depends on whether the OS is 32 bit or 64 bit. If you encounter a problem, you might want to add memory to the cache or the pool, but you should not make them larger than the size shown in Table 1-1.

If buffer cache and shared pool use are good, check for blocking wait events.

In the wait event summary at the top of the report, the CPU time event should be near or at the top of the list and ideally have a percentage of 70 or better. Typically, CPU time might be low if you are I/O bound. If the top wait events are I/O-related, you might be getting poor SQL execution plans. Look further down the report to see high-load SQL statements ordered by buffer gets. If you see statements with very high buffer gets per execute, you might be lacking an index on a table accessed by the statement. db_file_scattered_read wait events are associated with full table scans and also indicate the potential need for additional indexes (or up-to-date statistics for 10g).

Tuning the database server 17

White Paper

For more information about the SQL statements being executed and their execution plans, use the Oracle SQL trace utility.

When SQL tracing is on, raw trace files are generated in an Oracle dump directory. These trace files can be consumed by the tkprof utility to create reports on executed SQL statements. Multithreaded applications might produce multiple trace files at the same time, and the trace files might get large quickly, so managing the generated trace files can be challenging. Other ways to evaluate SQL execution plans include using the SQL plan tables—such as V$SQLAREA, V$SQL_TEXT, and V$SQL_PLAN—directly. If the EXECUTION PLAN has been aged out of V$SQL_PLAN, EXPLAIN PLAN and AUTOTRACE can be good starting places.

If full table scans are under control, the join clause of the execution plan might still be nonoptimal.

In general, hash join is one of the more problematic join clauses. BMC Software recommends keeping the parameter HASH_JOIN_ENABLED set to false (the default) to avoid hash joins.

Recommended init.ora settingsIf you use an Oracle database with your AR System–based products, BMC Software recommends using these init.ora settings:

Parameter Recommended value

_b_tree_bitmap_plans falsecursor_sharing similar (10g), force (9i)cursor_space_for_time falsedb_block_checking falsedb_block_checksum truedb_block_size 8kdb_multiblock_readcount 16log_buffer 10485760open_cursors 500optimizer_dynamic_sampling 2session_cached_cursors 100statistic_level typicaltimed_statistics trueundo_retention 14400work_area_policy auto



Best practices for tuning database (Oracle) serversWhen tuning the AR System database server, use this checklist:


Tuning the AR System serverAfter the database has been tuned and SQL execution appears to be efficient, the AR System server tier should be considered.

The most important aspect of tuning the AR System server is to remember this:

It is a multithreaded application.

Its default thread count settings are very low.

If the thread count is set too low, the AR System server will have low CPU use, poor throughput, and potentially poor response time under high load. Suggested thread counts are three times the number of CPUs for fast and five times the number of CPUs for list. So a two-CPU box might have six threads for fast and ten threads for list. BMC Software recommends using the same value for the minimum and maximum setting.

AR System server–based applications should work well on a variety of hardware platforms, including symmetric multiprocessor (SMP) systems that have many CPUs. The OS for such systems, however, might begin to have thread contention with very high thread counts for fast and list threads. Therefore, such systems with more than 8 CPUs might perform better with somewhat smaller thread counts, where the maximum for any thread pool is never greater than 30.

Thread counts can also be used to allocate resources to different workload components. In particular, reporting can be associated with a private thread with a modest thread count to limit the machine resources allocated to reporting.

A new setting for AR System server 7.0 and above is the NEXT_ID_BLOCK_SIZE. This specifies the number of IDs that can be allocated to create operations at one time. In previous version of the server, workloads with frequent create operations could encounter contention for such IDs. Setting NEXT_ID_BLOCK_SIZE to 10 is recommended for typical workloads—this usually eliminates any ID contention.

Use external storage to increase I/O bandwidth on the database server.

Use an appropriately sized System Global Area (SGA).

Set cursor_sharing to force in 9i and similar in 10g.

Use statspack (9i) or AWR (10g) reports to collect diagnostics.

If system is I/O bound or db_file_scattered_read events exist, consider whether high-load SQL statements need tuning or additional indexes should be added.

Keep HASH_JOIN_ENABLED set to false (the default).

Tuning the AR System server 19

White Paper

The AR System server has various logging capabilities that can provide diagnostics during a tuning exercise. For example, you might log API, SQL, and filters into the same file to get a comprehensive picture of the work going on during a critical operation. Such log files quickly become large, however, so use such logging conservatively in production environments. To reduce the impact of such logging operations, turn on buffered logging.

One risk with AR System server–based applications is that users will make requests that return very large sets of data. Typically, such large sets of data are not useful to users because they are unmanageable. In addition, making such large requests is an expensive operation—both on the database and on the client. The following AR System server settings can reduce the occurrence of expensive, open-ended queries in your system:

Max Entries Returned by GetList—Limits the size of all return sets. Any requested return set that is larger than the return list maximum causes an error message to be displayed to the user. Possible values might be 5000 or 1000 entries.

Server Table Field Chunk Size—Restricts the number of entries that are returned at one time, but the user can request the next “chunk.” For example, if you set this value to 50 but a query returns 200 entries, only the first 50 entries are initially returned. To see all 200 entries, users can request the next 50 entries three times. Typical values are 25 or 50 (users generally have trouble making use of more than 50 on a form). This option can be set for the entire system or for individual tables.

Best practices for tuning the AR System serverWhen tuning the AR System server, use this checklist:


Set thread count for fast and list to appropriate values.

Set NEXT_ID_BLOCK_SIZE to 10.

Use AR System server logging to obtain additional diagnostic information.

Set Max Entries Returned by GetList and Server Table Field Chunk Size to reduce the impact of queries with large return sets.



Overall tuning strategyGenerally, you should follow these steps to improve system performance:

1 Create a clear statement of your performance problem (see page 7).

2 Using the guidelines described in this paper, evaluate the following components to find out whether they might be contributing to the problem and, if so, how they might need to be changed:

a CPU and memory resources (see page 9)

b Mid tier (see page 12)

c Database server (see page 14)

d AR System server (see page 19)

IMPORTANT You should defer making changes until after you survey all the components.

3 If multiple changes are identified, prioritize them according to which are most likely to solve the issue identified in the problem statement.

4 Implement the top-priority change.

5 Evaluate the effect of the change.

IMPORTANT Changes in a production environment should be made one at a time. Each time a substantial change is made, quantitatively evaluate the effect of that change before making another change. Whenever multiple changes are made at the same time, there is a risk that new problems will occur and that it will be unclear which change caused the problem and which should be reversed first.

6 Repeat steps 4 and 5 for each change in order of priority.

Best practices for tuning the overall systemWhen tuning your overall system, use this checklist:


Survey each area of your system for possible tuning issues before making any changes.

Prioritize all changes by their likely impact on the identified problem.

Implement changes one at a time in a prioritized manner.

Quantitatively evaluate the impact of every change before considering whether another change should be made.

Overall tuning strategy 21

White Paper

Performance tuning checklistThis checklist includes the checklists that appear at the end of each preceding section of this paper.

Problem statement Characterize throughput or response time Before problem was observed

After problem was observed

List any system changes that took place soon before the problem was observed

Patches

Configuration changes

Workload changes

User count changes

CPU consumption monitoring Collect CPU usage data for each machine in the configuration.

Make sure less than 70% of the total CPU capacity is used on every system in the configuration.

Note which processes are consuming the majority of the CPU resources.

Note whether wait I/O time is a substantial component of CPU use. (This indicates that the I/O subsystem is not keeping up with demand.)

Memory consumption monitoring Make sure no system in the configuration is running out of physical memory and swapping.

Be aware of 64-bit and 32-bit limitations on both the OS and applications.

Track memory growth over time to identify any memory leaks.

Mid-tier tuning Use precaching to avoid high latency for first-time access to a form.

Set resource_check_interval to one day.

Set minimum heap size to 512 MB and maximum heap size to 1 GB.

Set pooling_max_total_connections to 2000.

Database (Oracle) server tuning Use external storage to increase I/O bandwidth on the database server.

Use an appropriately sized System Global Area (SGA).

Set cursor_sharing to force in 9i and similar in 10g.

Use statspack (9i) or AWR (10g) reports to collect diagnostics.

If system is I/O bound or db_file_scattered_read events exist, consider whether high-load SQL statements need tuning or additional indexes should be added.

Keep HASH_JOIN_ENABLED set to false (the default).

AR System server tuning Set thread count for fast and list to appropriate values.

Set NEXT_ID_BLOCK_SIZE to 10.

Use AR System server logging to obtain additional diagnostic information.

Set Max Entries Returned by GetList and Server Table Field Chunk Size to reduce the impact of queries with large return sets.

Overall system tuning Survey each area of your system for possible tuning issues before making any changes.

Prioritize all changes by their likely impact on the identified problem.

Implement changes one at a time in a prioritized manner.

Quantitatively evaluate the impact of every change before considering whether another change should be made.


*85503**85503**85503**85503*

*85503*

Documents

Performance Tuning for Business Service Management€¦ · Performance Tuning for Business Service Management Problem statement Initially, any performance tu ning effort requires