Volume Test APO 4.1

mySAP SCM Volume Testing Best Practice for Solution Management

Version Date: February 2002 The newest version of this Best Practice can always be obtained through

the SAP Solution Manager or the SAP Service Marketplace.

Contents Applicability, Goals, Requirements ...........................................................................................................1 Best Practice Procedure and Verification .................................................................................................4

Preliminary Tasks ...............................................................................................................................4 Check the Technical SCM Environment.......................................................................................4 Apply Recommendations for Critical Success Factors ................................................................4 Set Up the Test Environment .......................................................................................................6 Review and Sign-Off the Test Setup ..........................................................................................11 Applying this Best Practice to LiveCache ..................................................................................12

Test Procedure .................................................................................................................................13 Coordinating the Test .................................................................................................................13 Technical Preparation for Assistance from SAP.........................................................................13 Monitoring...................................................................................................................................14

Results Evaluation............................................................................................................................17 Documenting the Test Results ...................................................................................................17 Influence of Untested Processes and Process Steps ................................................................18 Judging the Results....................................................................................................................18

Further Information .................................................................................................................................20 Templates .........................................................................................................................................20 Feedback and Questions .................................................................................................................20

Applicability, Goals, Requirements To ensure that this Best Practice is the one you need, consider the following goals and requirements.

Goals of Using this Service The goal of this volume test is to run a realistic simulation of expected workload during the implementation period of a mySAP Supply Chain Management (SCM) solution. This simulation tests the solution's performance, sizing and scalability. Specifically, it focuses on runtimes and data throughput for specific business processes and interacting components.

Best Practice: mySAP SCM Volume Testing 2

Volume tests are performed to answer the following kinds of questions:

• Does the implementation of business processes meet your company's specific performance requirements for those processes?

• Will the implemented solution meet the company's specific performance requirements with respect to response time and throughput?

• Will the implemented solution be able to handle the maximum expected volume within the available time frame, thereby achieving the required throughput?

• Does interactive processing comply with the maximum allowed response times?

• Was the hardware sizing accurate, so that the planned hardware of the production system will be sufficient?

• Is the implemented solution scalable?

• Will there be resource-contention problems during production operation and what will be their performance impact?

It is often difficult to determine in advance whether the planned hardware will be sufficient to handle the planned workload. This is especially true for highly resource-consuming (or "expensive") customizing settings, user exits, or customer enhancements. Such elements usually cannot be taken into consideration in advance by sizing tools such as the SAP Quick Sizer.

Scalability can be endangered by the contention for central resources due to concurrent processing activities, such as when there are exclusive lock waits on number ranges (database table NRIV). During the volume test, activities that will be running concurrently in production operation are scheduled to run concurrently in order to identify negative interactions between them.

You can apply the methodology described here not just to the complete solution, but also to specific components. For example, you can use it to test the performance of a particular interface such as the CIF or to determine whether the size of the LiveCache sufficient (see below).

Staff and Skills Requirements To run the volume test, create it as a sub-project within the SAP implementation project by defining a test team that is responsible for:

• Determining the test plan

• Planning and preparing the execution of the test

• Evaluating the test result

• Initiating corrective actions as required by the test result

A team leader should be appointed to:

• Ensure sound planning of the test with all necessary success factors

• Perform a review of the test plan

• Manage the workflow and monitoring during the tests

System Requirements The test environment must have a comparable hardware to the productive hardware solution. Ideally the test should either be performed directly with the planned productive hardware configuration or with a suitable copy.

If you are unable to use the hardware configuration that will be used in your production landscape, ensure that the application parameters are set similarly and that the hardware size has at least been confirmed by a sizing estimation.

Note: An SCM landscape for testing purposes can be obtained by creation of a database copy. See the Best Practice "mySAP SCM System Landscape Copy".


Duration and Timing The duration of the volume test strongly depends on the complexity of the tested business scenarios. It should be performed several months prior to the start of production operations and prior to introducing majors changes to a productive system environment such as:

o Implementing a new module or new functionality o Adding new users o Performing a major hardware or software upgrade

It is essential that the volume and preliminary tests are performed under live conditions, at least 4 weeks before the live date. "Live conditions" means using the same data volume as in the production system. Enough time should be left in case sizing changes are necessary and the processes have to be restarted.

When planning the timetable for testing, remember to plan the preliminary testing (that is required to ensure a proper execution of the volume test) sufficiently ahead of the volume test. This preliminary testing (see below) consists of:

• An integration test that tests the functional consistency of the customized business processes. For this, no representative data is needed and the test can normally be performed by a single user.

• Optimizing the performance of individual business steps, for example, using the GoingLive optimization session with representative data and a single user.

After these preliminary tests, the volume test can be performed as a multi user test with representative data.

How to Use this Best Practice Study the whole document prior to the project start, then use it to guide you through the different steps of the project. This Best Practice document is divided into three parts:

• Preliminary activities This section includes guidelines on how to select suitable business scenarios and create a test plan. This preparatory phase concludes with a collective review and sign-off of all participating business and IT responsibles.

• Test Procedure and Monitoring This section describes how to execute and monitor the volume test.

• Results Evaluation This section explains how to evaluate the observed values and decide whether your performance requirements have been fulfilled.

Important steps in this document are illustrated with a hypothetical company performing a volume test according to SAP recommendations. The example is presented in boxes such as this one.


Best Practice Procedure and Verification

Preliminary Tasks

Introduction Performing these volume tests follows a bottom-up approach that goes proceeds through the following stages:

• Performing analysis and measurement in single user mode

• Using a forecast to delimit the scope of volume testing

• Performing volume testing

Following this approach saves you time and money because it is less expensive to optimize procedures at an early stage, for example on the basis of an analysis of a business process step in single user mode, rather than continuously repeating unsuccessful volume tests.

Based on the expected workload and the response times measured in single user mode, a forecast is made for each business process step as to whether it will become time-critical, that is, whether the expected workload will become unable to be processed within available time frames. This forecast is then used to delimit the scope of the volume test so that it only focuses on time-critical processing steps.

Tasks The preliminary tasks to be performed are described under the following headings.

Check the Technical SCM Environment Verify that the parameter recommendations of the last EarlyWatch service session or GoingLive Analysis session have been implemented. If the last report from such a service is older than 2 months or you have changed your technical IT environment, ask SAP for a new service session to get the latest parameter and system recommendations. It is extremely important to apply these parameter recommendations before a volume test.

Apply Recommendations for Critical Success Factors Success Factor: Test team Recommendation: The test team should have skills in the areas of technology and application. At least one person of the test team should work full-time on the preparation of the volume test, focusing on planning and implementing the technical framework of the volume test.

Success Factor: Test plan Recommendation: The test plan describes the test scenarios and the test activities to be performed. It includes the processing volumes and a detailed description how to attain these volumes. If a tool is used for load generation, the test plan describes the necessary parameterization, settings and conditions to enable you to for simulate a representative load. For dialog users who participate in the


test, the test plan describes what the users need to do. The test plan also contains a time schedule of all test activities.

Success Factor: Documentation of test results Recommendation: For each test run the results should be documented. This includes, for example, the actual volume processed. Relevant performance statistics are either stored in the SAP system and can be obtained from special transactions or have to be collected manually during the test run. In addition to collecting performance statistics, document any other conspicuous events or general observations. For each success criterion defined, document the extent to which the criterion was fulfilled by the test run.

Success Factor: Functional Correctness Recommendation: Perform an integration test to verify the functional correctness of business process implementation. The integration test should precede the volume test to ensure that the volume test truly reflects the processing in subsequent production operation and prevent the volume test failing due to functional errors. The integration test requires that development and customizing activities are already completed and that the versions of all software components are frozen until the start of production.

Success Factor: Performance Analysis and Optimization in Single User Mode Recommendation: Analyze the business process steps in single user mode for expensive database accesses or RFC calls, unnecessarily long database lock times or R/3 enqueue times, as well as for the consumption of database time, CPU time and RAM. Remember that you need a representative data basis in order for the measured response times and resource consumption to be representative of subsequent production operation.

Success Factor: Hardware Capacity and System Configuration Recommendation: The hardware and software components involved in the volume test have to be set up for optimal performance and stability. The hardware must be powerful enough to handle the planned load both during the volume test as well as during subsequent production operation. To minimize the risk of general bottleneck situations due to insufficient hardware or non-optimal system configuration, take advantage of SAP's GoingLive Service or EarlyWatch Service. For already productive systems, use the EarlyWatch Service.

Success Factor: Timing Recommendation: Perform volume tests during the go-live preparation phase of an SAP implementation project. If you are using the SAP GoingLive Service, the volume test should take place after the GoingLive Optimization session since the focus of this session is the analysis and optimization of business process steps in single user mode. The GoingLive Analysis session, which precedes the Optimization session, ensures a reasonable initial system configuration.

Success Factor: Representative Data Basis Recommendation: As database tables (especially transaction data tables) start to grow continuously after start of production, database accesses that are not fully specified tend to get slower and more resource-expensive. If the volume test is performed with almost empty database tables, the response times in subsequent production operation will be higher than measured in the volume test. Therefore, for the volume test to be meaningful, you need to populate the database tables with representative data. In particular, use representative master and transactional data.

Success Factor: Representative Load Profile Recommendation: Ideally, the simulated load and the hardware used for the volume test should be identical to the load and hardware of subsequent production operation. However, for various technical, organizational or economic reasons, it is not always feasible to simulate the full load expected in subsequent production operation. In addition, is it never really possible to execute the test on hardware exactly identical to the hardware planned for production.


The simulated load must be adequate for the hardware on which the test is performed. If the test is performed on a hardware that is less powerful than the hardware planned for production operation, the simulated load might need to be scaled down correspondingly in order not to create bottleneck situations. However, if you do perform the test on the production hardware with only part of the planned load, the simulated load must still be high enough compared to the available system resources to serve as a basis to project the system behavior for the full load. If the load generated in the volume test is too low, some bottlenecks might not become visible. Load can be generated either by a tool or by real users. If possible, use real users to generate the load by simulating a certain amount of their typical daily work based on real business data. If instead you use a simulation tool, check whether the tool is certified by SAP. Plan enough time for getting familiar with the tool so as to ensure that the load generated by the tool is representative for subsequent production operation.

Success Factor: Criteria for Successful Volume Test Recommendation: Define clear, verifiable and quantified criteria whose fulfillment will indicate a successful volume test. These criteria must be defined before performing the volume test, and be in accordance with the actual business requirements.

Set Up the Test Environment For setting up the testing scenario, identify and document the core business processes. You can do this within the framework of a Solution Management Assessment (SMA) Service offered by SAP. The complete set of business processes should then be integration-tested to ensure data consistency. Next, all business steps should be tuned individually and a first forecast can be made as to whether the system will be able to handle the proposed load if no interference with other business steps occurs.

For each of the core business processes, identify the relevant steps and arrange them in several test scenarios. Dependencies between the steps need to be identified and the critical success factors for the steps must be defined.

The steps you need to perform in order to set up the text environment are described in more detail under the following subheadings.

Introducing our hypothetical example: HopN'Malt GmbH is a German company that creates its revenue by producing and selling beer.

During the implementation phase, the company had a GoingLive Analysis session performed by SAP where many parameter recommendations were given, and these are now implemented into the systems. An integration test was performed to ensure that all customized business steps are consistent. As part of the GoingLive Optimization session, the business steps were tuned to show an optimum response time in single user mode.

Before the planned production start date, the responsible IT managers at HopN'Malt want to ensure that the system will work successfully within the constraints of the planned load and time schedule. Therefore, they set up a HopN'Malt Volume Test project.

Select Business Processes for the Test The first step in setting up the test scenario is choosing the business processes to be investigated. Typical candidates are the company's core business processes as well as other processes known to be time-critical. Time-critical paths within a business process can be identified, based on the planned processing volume, the available processing time window and the response times measured in single user mode.

HopN'Malt GmbH orders a Solution Management Assessment (SMA) service as part of SAP's Safeguarding Program. The essential part of the service is the determination and documentation of the main business processes. For HopN'Malt, the two main processes are Production Planning and Sales Order Management.


Production Planning:

AP O

SN P

R /3

E xternal S ystem

B eer P roduction

SN P p lann ing run

C reate p roduction orders

D P

R elease to SN P

PP/D S

In teractive p lann ing

C reate p lanned orders Forecast run

R elease to PP /D S

P P p lann ing run

R elease production orders

R eceive p roduced goods

Settle p roduction o rder Convert to p roduction order

Sales Order management:

APOR/3

Create sales orderGlobal

availability check

Create delivery

Generate picking list

Post goods issue

Create invoice


Create a Test Plan Next, document the critical parameters of the individual business steps. This information can mostly be found in the questionnaire for the GoingLive Analysis and Optimization session as well as in an SMA report. It is absolutely essential to collect all information about critical time windows as well as a realistic estimation of the volume of documents or line items processed. At this stage, you can already exclude from testing those business steps that are definitely non critical, that is, have low resource consumption and small run times.

The DP Forecast run and the SNP Planning run do not need to be taken into account, as during integration testing the runtimes have been in the order of 15-30 minutes. The volume test will only focus on the business steps highlighted below in gray.

Production Planning:

Business step

Number of line items (orders) per day

Processing time window

Peak time window

Number of line items for peak

Frequency Online / Background

Single user processing time per line item

DP-Forecast run

22:00 – 24:00

Monthly Background

SNP planning run

22:00 – 24:00

Monthly Background

PP-Planning run

14:00 – 14:30

Daily Background

Create planned orders-APO

300 14:30 – 15:00

Daily Background

Create planned orders – R/3

300 15:00 – 15:30

Daily Background

Create production orders

300 15:30 – 16:00

Daily Background

Goods receipt

2000 16:00 – 18:00

Daily Background 800 ms

Sales Order Management

Business step

Number of line items (orders) per day

Processing time window

Peak time window

Number of line items for peak

Frequency Online / Background

Single user processing time per line item

Create Sales order

220.000 07:00 – 19:00

09:00 – 12:00

60.000 Daily Online 400 ms

Create 220.000 12:00 – Daily Background 200 ms
delivery 14:00


Work out a 24-hours time-grid showing the planned execution time-windows and associated processing volumes for each business step involved in the test. Identify all load scenarios that may differ from this model. The test scenarios that are uncritical or already included in another scenario can be neglected. In a last step, arrange the different test scenarios in a way that is most suitable for the dialog users and the system administrators who have to schedule or start the background jobs.

For the selected business steps, a timetable is created and the different load scenarios are determined.

Select different load situations (scenarios)

240 Time

CDPSB1

VL31N

VL35

VL23

VA01

VL04

VF04

(1) (2) 3 4 5 (6) 7

Easy situations are now separated from the complex ones. For HopN'Malt GmbH, scenario 1 does not interfere with other processes. It will already have been determined whether the system can use parallel processing for invoice creation. Scenarios 2 and 6 are contained in other scenarios where a similar or even higher load of the same process step is generated.

Include Interfaces Interfaces between the different SAP and non-SAP systems can be critical both in terms of throughput and generating errors. Therefore, these interfaces should be part of the volume test. For the core interface between R/3 and APO that is known as CIF, you will find monitoring recommendations in the corresponding chapters of this document. Other interfaces, SAP or non-SAP, may often be monitored similarly. In order to include interfaces in the test scenarios, they should be treated as individual business steps, since they have their own processing times, throughput, and critical success factors.

Identify Dependencies Between Different Process Steps For each test scenario, the test team should try to identify possible accesses to common objects from different process steps. These objects can include materials, document numbers or number range objects where accesses may cause lock situations during the volume test and subsequent production activities. Before the test, check whether the SAP Notes 70865, 151672, and 91580 (for material locks), as well as 37884 and 179224 (for number range objects) can be implemented in your system.

Describe Success Criteria for the Volume Test In order to determine if a volume test has been successful, it is essential that every business step owner defines the success criteria for their step. Success criteria may include the average and maximum response times for certain dialog transactions, as well as a minimum throughput per time unit. For background jobs, the criteria may include the runtime or the minimum number of parallel-running jobs.


Prepare the Data Load Database tables, especially transaction data tables, start to grow continuously after the start of production. Consequently database accesses that are not fully specified tend to become more and more expensive. If the volume test is performed with almost empty database tables, the response times in subsequent production operation will be higher than measured in the volume test. Therefore, the database tables must be populated with representative data for the volume test to be meaningful.

If you are planning to go live with both a new SAP R/3 and SAP APO system, the APO system will not contain any historical data, so forecasting and planning will be difficult. SAP recommends splitting the volume test in two parts: a volume test for the R/3 system alone, where some transactional data will be generated; and a second testing phase for the APO integration with R/3.

For the SAP APO system with LiveCache, you require an initial data load from R/3. This occurs automatically as soon as the integration models defined are activated. Now the master data and also the relevant transactional data is transferred to the APO system. Note that for Demand Planning you also require historical transactional data that are is sent to the APO system automatically but must be transferred using special programs.

Load Simulation Tools Tools are the alternative to manual testing. Manual test cases are excellent for integration tests. They are descriptions of business-process tests that a tester must perform manually on the system. For executing a volume test, these test cases can be performed by a higher number of users to generate the desired load level. This test scenario is very close to the production scenario, but may be involved and costly.

Alternatively, you can run the volume test using an automated test tool that simulates user activities. The SAP Test Workbench contains the Computer Aided Test Tool (CATT) to create and execute automatic test cases. Automatic test cases are performed by the R/3 System without user dialog, and are most useful for volume tests. The use of automatic tests can considerably reduce the test effort. Test cases test individual transactions or whole business transactions. Using the SAP CATT test tool, the transaction test case is recorded with a transaction recorder for the test module. You run the application as in normal dialog operation. The CATT tool is limited to testing only SAP R/3 modules. The SAP Test Workbench is integrated into the SAP Solution Manager, SAP’s new evaluation and implementation portal for mySAP.com. Within the SAP Test Workbench is the new test tool eCATT, which is specially developed for mySAP.com components. eCATT will be the follow-up product for the CATT tool within the Test Workbench. For more information about SAP eCATT, see the SAP Service Marketplace: go to http://www.service.sap.com and use the Advanced Search to search for eCATT. For testing cross-system integrated functionality, there are many external SAP R/3 testing tools on the market that are certified by SAP, for example:

• • • • • • • • •

AutoTester from AutoTester Inc. AutoControler from AutoTester Inc. WinRunner from Mercury Interactive LoadRunner from Mercury Interactive TestDirector from Mercury Interactive TestSuite from Mercury Interactive Rational TeamTest from Rational Software Rational Robot from Rational Software QACenter from Compuware

For more information on our Software Partners and their respective product in this area, select the Software Partner list from the SAPNet. (Note that the mySAP.com logo in the list indicates that the Software Partner achieved a certification for a recent release of this interface that is valid for the mySAP.com Edition.)

If you want to run the testing tool on the SAP APO side, you should be sure that the tool has been certified for that mySAP.com component.

http://www.service.sap.com/


Ensure Test Data is Realistic The volume test should be a realistic simulation of subsequent production operation and therefore the provision of suitable data is extremely important and often the key factor for the success of a volume test and for a smooth production start.

Transactional business steps should be performed with the line item distribution that is identical to that used later. It is important for the critical success factor definition and the evaluation of the results to know the average number of line items per document. But you should also test the maximum numbers since non-linearities can occur. For programs and reports, the processed data volume should be within the order of magnitude expected during production operation.

Another way in which test data must be realistic is that the customer numbers used, the materials and so on, must all be different and reflect the subsequent operations for the business scenario. This results in a detailed analysis of the business scenario by the business process responsibles and an explicit documentation in the test scenario.

Review and Sign-Off the Test Setup At the end of the planning phase for the volume test, it is essential that the managers and business process owners review the test scenarios and critical success factors. Each business process owner should evaluate whether all significant business steps were considered and sign-off the test results as having been successful. The Appendix contains a link to a template that you can use as a guideline for the review and sign-off.


Applying this Best Practice to LiveCache In principle, all descriptions and methodologies for preparation, monitoring and evaluation of volume testing contained in this Best Practice can also be applied to verify the LiveCache size. Some more specific details are included under the following subheadings.

Setting up the Volume Test The load on the LiveCache is normally generated by background programs, alert monitors, COM-routines, online users and data transferred via CIF from other systems. All these factors can be taken into account in the load test and should be considered if the load created is significant. SAP recommends that you create an overview of all jobs as like described above, including the respective time windows and required throughputs and response times. If you are not sure which processes are the most performance intensive and therefore cannot split the test into separate test scenarios, we recommend simulating a complete process cycle (day, week, month) within the volume test.

Providing Representative Test Data It is very important to use representative data for the volume test, that is, the master and transactional data that must be in the system before GoingLive. Otherwise, it is not possible to draw conclusions

a

At the end of the preparation phase, the “HopN'Malt Volume Test” project team has collected all information about the relevant business steps, the critical success factors and set up a test plan with four test scenarios. Before the test execution starts, the business owners for Production Planning and Sales Order Management as well as the responsible managers met to approve the test plan. For each process step, the approved values were as follows:

Sales Order Management, Create Sales Order

Value Business step Create Sales Order Number of orders/line items per day 220.000 line items Processing time window 07:00 – 19:00 Peak time window 09:00 – 12:00 Number of orders/line items during peak 60.000 line items Average number of line items per document

10

Maximum number of line items per document

100

Frequency of execution Daily (Monday – Friday) Online/Background Online Number of users/parallel jobs 200 users (call centre) Number of users/parallel jobs for test 100 users Critical success factors Max. response time for average order: 5 sec

Max. response time for 100 line item order: 70 sec Max. throughput per hour: 4000 orders (10 line items)

Interference with other business steps Material locks might occur during availability check

Preparation of realistic test data 1980 orders with 10 line items, 20 orders with 100
line items 1800 customer masters, 300 materials
bout the proper size of the LiveCache.


Test Procedure

Coordinating the Test The test coordinator is a key resource in the test team. He is responsible for executing and monitoring the test. The main aspects of his responsibility are:

• Executing the test according to the test plan • Ensuring that all actions within the test scenario are started and ended properly • Deciding when a test scenario is finished • Communicating with the dialog users • Acting as the first contact for complaints and problems • Communicating with the monitoring team • Ensuring that the right measurements are performed at the right time • Deciding whether the current situation requires the test plan to be changed

Technical Preparation for Assistance from SAP If you need assistance from SAP during the volume test, ensure that the following connections are available through SAPNet Frontend and that suitable user IDs and passwords have been created:

• R/3 Support connection to the R/3 and APO system

• SAP DB connection to the liveCache (see SAP Note 202344)

• UNIX: Telnet-connection to the liveCache

• Win/NT: PcAnywhere- or Microsoft Netmeeting-connection to the liveCache

Additionally, the remote SAPOSCOL must be installed on each server without SAP basis (for example, the stand-alone database server or the liveCache server), so as to allow monitoring of the CPU and RAM utilization from an SAP application server (see SAP Note 20624).

For the volume test, activate the writing of statistical sub-records for database procedure calls (COM routine calls) in SAP APO. To do this, proceed as follows:

• Call transaction ST03N, select ‘Expert mode’

• Choose Collector and Performance DB >> Statistics Records and File >> Online Parameters >> Dialog step statistics

• Set parameter STAT/DBPROCREC to 10

Before you start the tests, ensure that R/3 and APO have been restarted to reset some of the monitoring parameters. The counters of the liveCache should also be reset (without loosing the information in the LiveCache). Follow SAP Notes 359584 and 483776.

Execute report /SAPAPO/OM_PERFORMANCE with default settings (5 sec) on an idle APO/liveCache system and store the results. (/SAPAPO/OM_PERFORMANCE is a standard runtime check and provides basis data for APO/liveCache performance comparison.)

If APO and liveCache reside on different servers, check network communication time between APO and liveCache using transaction /SAPAPO/OM13 and then choosing Network (see SAP Note 458221). Store the results.


Monitoring Monitoring of a volume test scenario in an SCM solution landscape involves the OLTP-R/3 System, the CIF interface between the R/3 System and the APO System, the APO liveCache, and the APO Optimizer.

For most of the key figures relevant to monitoring a volume test, the SAP application server maintains statistics that can be evaluated after the end of the volume test. Continuous monitoring during the execution of the volume test is required to detect temporary bottlenecks that are not logged explicitly and that would, on average, level out. The monitoring frequency must be high enough to monitor temporary peak or bottleneck situations or temporary contention problems such as exclusive lock waits.

A monitoring log with the relevant key figures (see the template in the Appendix) should be maintained for each test scenario in order to document the measured values.

Monitoring the SAP Basis System (R/3 and APO) The CPU and RAM utilization of each server involved in the volume test needs to be monitored. For servers without SAP Basis, such as the stand-alone database server or the liveCache server, remote SAPOSCOL needs to be installed on that server to allow the monitoring from within an SAP application server.

CPU: The hourly average CPU utilization can be obtained from the Operating System Monitor (ST06 or OS07) after the test. Additionally, the CPU utilization should be monitored periodically during the volume test to be able to identify temporary CPU bottlenecks that level out within the hourly average.

RAM: The hourly average RAM utilization (paging rates) can be obtained from the Operating System Monitor (ST06 or OS07) after the test. Additionally, the RAM utilization (paging rates) should be monitored periodically during the volume test to be able to identify temporary RAM bottlenecks that level out within the hourly average.

For each SAP application server of the OLTP-R/3 System and the APO System, the following key figures need to be monitored:

• Response times and memory utilization of the programs and transactions that implement relevant business process steps

• Utilization of SAP shared buffers and SAP memory

• Number of active users.

Continuous monitoring is required to detect lock wait situations and to keep an overview of the current system activity while the volume test is in progress.

Response times: The response times, CPU times and database times of the programs and transactions that implement the business process steps involved in a test scenario can be obtained from the Single Statistical Records (STAD). In STAD, select the display mode 'Show business transaction sums' and restrict the evaluation of the single statistical records to the time interval of the test scenario. To restrict the display to a particular transaction or program, set a filter in the screen 'Workload: Business Transaction Analysis.' To drill down to task type level, click button 'Open/close all' and select 'Both aggregations'. To download the displayed list into a spreadsheet, use the 'Download' button.

Use the spreadsheet to calculate the following key figures:

• Number of executions

• Average response time per execution

• Average CPU time per execution

• Average database time per execution

• Maximum response time of all executions.

Note that you should consider the time for the complete execution of a program or transaction, rather than look at the average response times per dialog step.


SAP shared buffers: Snapshot data about the utilization of SAP shared buffers can be obtained from the SAP Memory Configuration Monitor (ST02) after the test. Immediately after each test scenario, make a note of all buffers that have swapped.

SAP memory: Snapshot data and high water marks of SAP memory utilization (extended/ roll/ paging/ heap memory) can be obtained from the SAP Memory Configuration Monitor (ST02). Make a note of the high water marks after each test scenario.

Number of active users: The number of active users can be obtained from the Global User Monitor (AL08). Make a note of the number of interactive users and RFC users during each test scenario.

Database locks: Snapshot data about exclusive wait situations for database locks can be obtained from the Database Lock Monitor (DB01). If exclusive database lock waits occur, document the locked object, lock holder (program/transaction) and lock waiter (program/transaction).

Current system activity: The main tool for monitoring the current system activity is the Global Work Process Overview (SM66). Continuously monitor the status of the work processes. Make a note of aspects such as the following:

• Are there always free work processes available (that is, work processes with status 'waiting')?

• Are there work processes in status 'stopped'? If so, see column 'Reason'.

• Are there work processes in status 'terminated'?

• For work processes in status 'running': See column 'Action/Reason for waiting' for the current activity.

• Are there work processes waiting for database locks?

• Are there work processes waiting for semaphores?

To deepen the analysis, use the appropriate detail monitor, for example, the Database Monitor (ST04), Database Lock Monitor (DB01), Operating System Monitor (ST06), SAP Memory Configuration Monitor (ST02).

Errors: After the volume test, investigate the System Log (SM21) for error messages and warnings that indicate potential functional or stability problems. Additionally, check for ABAP dumps (ST22) and update errors (SM13) that occurred during the volume test.

Load: For each test scenario, measure the actual system load (for example, the number of active users, number of sales orders processed). Compare this data with the load planned for the volume test.

Monitoring APO – R/3 Integration (CIF) For the CIF interface between the APO System and connected R/3 systems, monitor the key figures below. Continuous monitoring of the CIF interface is required while the volume test is in progress to detect hanging queues and throughput problems on sender side or on receiver side. After the volume test, evaluate the messages in the CIF application log.

RFC wait time: Monitor the average RFC wait time after each test scenario. The average RFC wait time can be obtained from the Workload Monitor (ST03 or ST03N) of the APO System and the connected R/3 System.

CIF queues: During the volume test, continuously monitor the CIF interface on APO side and on R/3 side. For hanging queues due to errors that might have occurred, use SMQ1 (outbound queues) and SMQ2 (if inbound queues are used). Also monitor the fill level of the active queues to see if qRFC requests pile up on either the sender side or the receiver side.

CIF application log: After the end of the volume test check the CIF application log for errors, warnings or other messages.

Monitoring the liveCache For the liveCache, monitor the following key figures:

• CPU and RAM utilization of the liveCache server


• Data cache usage and hit rates

• Usage of heap memory.

Continuous monitoring of the liveCache activity is required while the volume test is in progress to detect wait situations for locks, CPU, and I/O.

Data cache usage and hit rates: Information about usage and hit rates of the liveCache data cache can be obtained from the liveCache Monitor (transaction LC10, then choose liveCache Performance >> Data Cache). The data cache usage should be well below 100%. The data cache hit rate for OMS data and OMS history should be nearly 100%. Ideally, each page of the data cache is loaded once at its first access and never gets displaced from the cache so that subsequent accesses will find the page already in the cache and do not have to read it from the disk again. Take a note of the total number of liveCache accesses and of the number of failed liveCache accesses immediately before and after each test scenario.

Only for liveCache on Windows platforms using PSE36 or AWE: PSE36 for Windows NT and AWE for Windows 2000 denote techniques that allow an application to address more than 4GB of RAM. If PSE36 or AWE are used and the data cache hit rate displayed in LC10 is less than 100%, run the command x_cons <liveCacheName> to show pse_stat. For a sufficiently large data cache, the values displayed for 'reads out of range' and 'writes out of range' should be zero (see also SAP Note 398665).

Heap memory: The usage of liveCache heap memory can be monitored with the report /SAPAPO/OM_LC_MEM_MONITOR. The first line of the report displays:

• The amount of heap memory which is currently used (Total Heap)

• The amount of heap memory allocated by the operating system (Reserved Heap), reflecting reflects the maximum usage of heap memory since liveCache start

Periodically monitor the heap memory usage during the volume test. When the allocated heap memory (Reserved Heap) reaches the limit defined by the liveCache configuration parameter OMS_HEAP_LIMIT (transaction LC10, then choose liveCache performance >> Active parameters) or by the operating system limit for heap memory, the affected COM routine or the entire liveCache may terminate with an error. After the volume test, use SM49 to execute the command dbmcli -d <LCName> -n <Node> -u control,control -uSQL sapr3,sap sql_execute select sum("OutOfMemoryExceptions") from monitor_oms . This command gives the number of 'OutOfMemoryExceptions' due to Heap memory shortages since liveCache startup.

liveCache activity: During the volume test, continuously monitor the liveCache activity by using the liveCache Monitor (transaction LC10, then choose liveCache console >> Active tasks). The liveCache runs as expected if the number of active tasks is generally lower than the number of CPUs available to the liveCache and if the active tasks are mainly in status 'Running' or 'DcomObjCalled'. If the number of active tasks is greater than the number of CPUs available to the liveCache, user tasks have to wait for being processed by any of the CPUs. Status 'Vbegexcl', 'Vsuspend' and 'Vwait' indicate exclusive lock wait situations. Status 'IO Wait (R)' and 'IO Wait (W)' indicate wait situations for I/O operations.

The active and inactive times of liveCache threads can be obtained from the liveCache Monitor via transaction LC10 and choosing liveCache console >> Sleeping tasks. The idle time of liveCache threads is displayed. A liveCache thread is idle if it waits for requests from APO, for I/O completion or for internal liveCache locks. Low idle times of all threads used for COM routine execution may indicate a CPU bottleneck for liveCache. To identify the threads performing COM routine calls, call transaction LC10, then choose liveCache console >> Runtime envirnmt. Look for the entry ‘US’ in column ‘Task Cluster’. Note that all times refer to the activation time of detailed time measurement in liveCache.

CPU and RAM utilization: The CPU and RAM utilization of the liveCache server needs to be monitored. To allow monitoring via the Operating System Monitor (OS07) from within APO (or another system with SAP Basis), remote SAPOSCOL has to be installed on the liveCache server.

The hourly average CPU utilization can be obtained from the Operating System Monitor (OS07) after the test. Additionally, the CPU utilization should be monitored periodically during the volume test to be able to identify temporary CPU bottlenecks that level out within the hourly average.

The hourly average RAM utilization (paging rates) can be obtained from the Operating System Monitor (OS07) after the test. Additionally, the RAM utilization (paging rates) should be monitored periodically during the volume test to be able to identify temporary RAM bottlenecks that level out within the hourly average.


Response times of COM routines: After the volume test monitor the response times of the COM routines using the liveCache Monitor (transaction LC10, then choose liveCache monitor). The average response times should be less than 1 second. Typical values range between 10-200ms. Exceptions include:

• COM routines for mass processing (SAPAPO_GET_ALL_*)

• COM routines for Demand Planning (SAPTS*)

• COM routines for administrative purposes)

Save the name, number of calls and average response time of the top 10 COM routines by number of calls and of the top 10 COM routines by average response time. Additionally, save the data of the top 10 COM routines with highest accumulated processing time (see SAP Note 483996).

Error log files: After the volume test, the following log files should be investigated for error messages and warnings that indicate potential functional and stability problems:

• KNLDIAG

• KNLDIAG.OLD

• KNLDIAG.ERR.

To display the log files, use the liveCache Monitor (transaction LC10, then choose liveCache Configuration >> System messages).

Monitoring the APO Optimizer

For the APO Optimizer, monitor the following key figures:

• CPU and RAM utilization of the Optimizer server

• Memory for SNP Optimizer

CPU and RAM utilization: To enable you to monitor the CPU and RAM utilization of the Optimizer server via the Operating System Monitor (OS07) from within the APO System (or another system with SAP Basis), remote SAPOSCOL has to be installed on the Optimizer server.

The hourly average CPU utilization can be obtained from the Operating System Monitor (OS07) after the test. Additionally, the CPU utilization should be monitored periodically during the volume test to be able to identify temporary CPU bottlenecks that level out within the hourly average.

The hourly average RAM utilization (paging rates) can be obtained from the Operating System Monitor (OS07) after the test. Additionally, the RAM utilization (paging rates) should be monitored periodically during the volume test to be able to identify temporary RAM bottlenecks that level out within the hourly average.

Memory for SNP Optimizer: If the SNP Optimizer is used in the volume test, check the log file of the SNP Optimizer to see whether sufficient memory for the SNP Optimizer was available. The name of the log file can be found in transaction /SAPAPO/COPT01. Search for error message "Not enough memory".

Results Evaluation

Documenting the Test Results For reporting purposes as well as for later follow-up on issues detected during the volume test, a detailed written report is essential, as a documentation of the test results. The documentation should contain the hardware setup, the business processes and process steps, the test scenarios and also the monitoring results. Furthermore, an evaluation of the results should be performed as described below. The Appendix contains a template for this report.


Influence of Untested Processes and Process Steps

Business processes, process steps and programs that do not interfere with any other business step or program do not necessarily need to be part of the volume test. However, the resource consumption of these components has to be taken into account if the hardware size is evaluated. For this purpose, determine the resource consumption of these programs and steps, either during the integration testing or the single-user performance optimization.

Judging the Results The result of the volume test is the determination whether the hardware is sufficient to handle the expected system load and – at the same time – the comparison of measured and required performance according to the critical success factors. The following section describes some easy rules you can use to evaluate the results of the volume test from the monitored values.

R/3 and APO Application Servers Maximum CPU utilization: For each measurement of the hourly CPU utilization sum up the CPU utilization by user and system and enter the maximum value as 'Max. CPU utilization [%]'.

CPU rating: If the average CPU load increases to more than 70%, some CPU wait situations may occur. An average CPU load of more than 90% clearly indicates a CPU bottleneck situation.

Memory used:

1. Calculate an upper limit for the maximum memory usage MaxMemUsage1 as the sum of memory used by SAP shared buffers, SAP work processes and users contexts. An upper limited for the memory used by user contexts is calculated by adding up the maximum usage of main memory for SAP Roll Memory, SAP Paging Memory, SAP Extended Memory and SAP Heap Memory.

2. Calculate an upper limit for the maximum memory usage as the sum of RAM [MB] and the maximum paging rate measured within the hourly average. Note that you have to take the page-out rates for Unix servers and the page-in rates for Windows servers.

3. Enter the minimum of MaxMemUsage1 and MaxMemUsage2 as 'Memory used [MB]'.

Maximum paging: Based on the measured hourly paging rates, enter the value for 'Max. Paging [% of RAM]'.

Memory rating: The memory usage should be less than 125% of the RAM size. The hourly paging rates should be less than 25% of the RAM size. A memory usage of more than 150% of the RAM size or paging rates of more than 50% of the RAM size indicate a memory bottleneck.

SAP Memory usage:

Roll area: The maximum usage ('Max. used [KB]) of the roll area, configured as shared memory ('In memory [KB]'), should be less than 80%.

Paging area: The maximum usage ('Max. used [KB])of the paging area ('In memory [KB]' + 'On disk [KB]') should be less than 80%

Extended Memory: The maximum usage ('Max. used [KB]) of the Extended Memory ('In memory [KB]') should be less than 80%.

Lock situations: Lock situations may occur in every system. The duration of the lock, the frequency of occurrence and the locked object type are of importance to judge whether and what actions should be taken.

APO Optimizer CPU and Memory rating: The same rules apply as described above for the application servers.


LiveCache Memory used:

1. Calculate an upper limit for the maximum memory usage MaxMemUsage1 as the sum of memory used by the liveCache Data Cache and liveCache Heap (Reserved Heap).

2. Calculate an upper limit for the maximum memory usage as the sum of RAM [MB] and the maximum paging rate measured within the hourly average. Note that you have to take the page-out rates for Unix servers and the page-in rates for Windows servers.

3. Enter the minimum of MaxMemUsage1 and MaxMemUsage2 as 'Memory used [MB]' in the table above.'

CPU and Memory rating: The same rules apply as described above for the application servers.

Configuration:

Data cache usage: The data cache usage should be below 80%.

OMS data - Hit rate: The data cache hit rate for OMS data should be nearly 100%.

OMS history - Hit rate: The data cache hit rate for OMS history should be nearly 100%.

Heap Memory usage: When the allocated heap memory (Reserved Heap) reaches the limit defined by the liveCache configuration parameter OMS_HEAP_LIMIT (transaction LC10, then choose liveCache performance >> Active parameters) or the operating system limit for heap memory, the affected COM routine or the entire liveCache might terminate with an error. Reserved Heap should not exceed 80% of the minimum of these two limits.

Monitoring of the COM routines: The average response times should be less than 1 second. Typical values range between 10-200ms (exceptions: COM routines for mass processing (SAPAPO_GET_ALL_*), COM routines for Demand Planning (SAPTS*), COM routines for administrative purposes).


F

T

FSh

©NoSo

HopN'Malt GmbH has performed all tests and monitored the parameters as defined. In a last step, the final report has to be written and the hardware and business process steps must be evaluated.

Maximum CPU and Memory Utilization

Server Max. CPU utilization [%]

CPU Rating RAM [MB] Memory used [MB]

Memory used [% of RAM]

Max. Paging [% of RAM]

Memory Rating

Sapaix01 87 2048 2850 139 45

Sapaix02 66 2048 1856 91 12

Sapdb1 77 4096 3200 78 10

From the monitored values, the hardware capacity was evaluated and showed a possible bottleneck on server SAPAIX01, which could maybe avoided by redistributing load onto the other application server. The database server also shows a somewhat too high CPU utilization. Especially if the untested processes are taken into account, the hardware might not be sufficient.

Sales Order Management, Create Sales Order

Transaction/ Program

Critical success factor Value required

Value observed

Rating

VA01 Max. response time for average order 5 sec 4 sec VA01 Max. response time for 100 line item order 70 sec 90 sec VA01 Max. throughput per hour 4000 orders 4210 orders

For the business process step ‘Create Sales Order’, three critical success factors have been defined: the response time for a average order with 10 line items, the maximum response time for a 100 line item orders and the throughput for the peak hour. The response time for average orders can be found directly in the monitoring protocol. For the orders with 100 line items, a special monitoring process has to be defined, since the value can hardly be obtained by the general statistics. Here it would be more suitable to monitor certain users who perform only these big orders and get their average response times from the statistics. To obtain the throughput figure, the easiest approach would be via the SIS or a direct access to the database via SE16.

urther Information

emplates • Template for Review and SignOff

• Templates for Monitoring

• Template For Report

eedback and Questions end any feedback by formulating an SAP customer message to component SV-GST-SMC at ttp://service.sap.com/message in the SAP Service Marketplace.

Copyright 2001 SAP AG. All rights reserved. o part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission f SAP AG. The information contained herein may be changed without prior notice. ome software products marketed by SAP AG and its distributors contain proprietary software components of ther software vendors.

http://service.sap.com/~sapidb/011000358700000809122002E



http://service.sap.com/message


Microsoft®, WINDOWS®, NT®, EXCEL®, Word®, PowerPoint® and SQL Server® are registered trademarks of Microsoft Corporation.

IBM®, DB2®, OS/2®, DB2/6000®, Parallel Sysplex®, MVS/ESA®, RS/6000®, AIX®, S/390®, AS/400®, OS/390®, and OS/400® are registered trademarks of IBM Corporation.

ORACLE® is a registered trademark of ORACLE Corporation.

INFORMIX®-OnLine for SAP and Informix® Dynamic ServerTM

are registered trademarks of Informix Software Incorporated.

UNIX®, X/Open®, OSF/1®, and Motif® are registered trademarks of the Open Group.

HTML, DHTML, XML, XHTML are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology.

JAVA® is a registered trademark of Sun Microsystems, Inc. JAVASCRIPT® is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape.

SAP, SAP Logo, R/2, RIVA, R/3, ABAP, SAP ArchiveLink, SAP Business Workflow, WebFlow, SAP EarlyWatch, BAPI, SAPPHIRE, Management Cockpit, mySAP.com Logo and mySAP.com are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other products mentioned are trademarks or registered trademarks of their respective companies.

Disclaimer: SAP AG assumes no responsibility for errors or omissions in these materials. These materials are provided “as is” without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement.

SAP shall not be liable for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. SAP does not warrant the accuracy or completeness of the information, text, graphics, links or other items contained within these materials. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third party Web pages nor provide any warranty whatsoever relating to third party Web pages.

Documents

Volume Test APO 4.1