BEST PRACTICES FOR SAP R3 PERFORMANCE TESTING

SAP Performance Testing Best Practices Guide

Wipro Performance Engineering Practice

BEST PRACTICES FOR

SAP R3

PERFORMANCE TESTING

By

SURYAKANTH KELMANI

Performance Engineering Practice Wipro Technologies

Version No. Author Revision Description

0.1 Suryakanth Kelmani Initial draft issued for internal review

0.2 Suryakanth Kelmani Added the SAP Performance Test Lab setup

0.3 Suryakanth Kelmani Updated with the Monitoring T codes and Batch jobs execution results

0.4 Suryakanth Kelmani Updated with SAP performance testing methodology.

0.5 Suryakanth Kelmani Added SAP WEB correlations



TABLE OF CONTENT

1. INTRODUCTION ......................................................................................... 3 1.1 OBJECTIVE ................................ ................................ ................................ ........................... 3 1.2 PERFORMANCE TESTING FRAMEWORK ................................ ................................ ......... 3

2. PLANNING .................................................................................................. 4 2.1 BUSINESS STREAMS: ................................ ................................ ................................ .......... 5 2.2 TEST ENVIRONMENT ................................ ................................ ................................ .......... 8 2.3 TEST DATA MANAGEMENT ................................ ................................ ............................... 9 2.4 TEST TOOL & SETUP ................................ ................................ ................................ ..........10

3. BUILD / TEST SCRIPTS DESIGN............................................................. 12 3.1 SERVER SIDE SETTINGS ................................ ................................ ................................ ....12 3.2 CLIENT SIDE SETTINGS ................................ ................................ ................................ .....12

4. EXECUTION.............................................................................................. 15 4.1 EXECUTION OF ONLINE USERS – In SILO ................................ ................................ .......15 4.2 EXECUTION OF BATCH JOBS & REPORTS – In SILO ................................ ......................15 4.3 APPROACH FOR BATCH JOBS PERFORMANCE TESTING ................................ .............15 4.4 EXECUTION OF ONLINE USERS AND BATCH JOBS & REPORTS (Combined) ..............17 4.5 RUN TIME SETTINGS FOR EXECUTION: ................................ ................................ .........17 4.6 SCHEDULE BUILDER................................ ................................ ................................ .........18

5. MONITORS & ANALYSIS......................................................................... 19 5.1 SAPGUI MONITOR ................................ ................................ ................................ ..............19

6. SAP PERFORMANCE MEASUREMENT & METRICS ............................ 22 6.1 WORKLOAD ANALYSIS ................................ ................................ ................................ .....22 6.2 MEMORY/BUFFER MANAGEMENT ................................ ................................ ..................22 6.3 DATABASE MANAGEMENT ................................ ................................ .............................. 23 6.4 UNIX SERVERS (OS Parameters, CPUs, Memory) ................................ ................................ 23

7. SAMPLE GRAPHS..................................................................................... 24 7.1 UNIX RESOURCE GRAPH ................................ ................................ ................................ ...24 7.2 EXECUTIVE SUMMARY GRAPH ................................ ................................ .......................25 7.3 AVERAGE RESPONSE TIME GRAPH FOR T CODES ................................ ........................25 7.4 BATCH JOB/INTERFACE SAMPLE REPORT ................................ ................................ .....26

8. APPENDIX ................................................................................................. 26 A. LOADRUNNER HARDWARE & SOFTWARE ................................ ................................ .26 B. SAP T CODES & ITS FUNCTIONS ................................ ................................ ...................26 C. SAP WEB CORRELATIONS ................................ ................................ ............................. 27 D. REFERENCES ................................ ................................ ................................ ...................28



1. INTRODUCTION

1.1 OBJECTIVE

The objective of this document is to detail the best practice, approach & plan

to be followed for the Performance testing of SAP application and to make sure that the team member who is going to carry out performance test activities can quickly refer and implement the best practice followed across all phases of performance testing.

This document is intended to guide further Volume Test Planning, tool setup, Script Preparation, Execution & Result analysis activities of Performance test phase.

1.2 PERFORMANCE TESTING FRAMEWORK The below diagram depicts the robust & proven performance testing

framework for SAP performance testing, each phase has the detailed activities that needs to be carried out and the output or deliverables at the end of each phase as shown in the fig below.

Activities of each phase are explained in the below sections...

Performance Testing Framework



2. PLANNING

The objective of planning phase is to identify tasks, pre-requisite and dependencies that is required to carry out performance testing. Planning phase should explain what are the activities you are going to do and how we are going to plan in order to get these task done, what ever activities we do identify in this phase should be noted down and put these tasks on priority basis. The key tasks involved in this phase are:

• Prepare the performance test approach • Identify the critical transactions, volumes, user base and define their SLAs. • Identify the environment and tools required • Identify test data requirements • Prepare a detailed Project plan with miles stones and the deliverables.

Performance test approach document will specify the approach taken to carry out performance test, scope of testing, environments and tools used, inputs, dependencies and deliverables of the Performance testing phase.

Information regarding critical transactions for each work stream, the associated volumes and the concurrent user information is expected to be made available by the business team in the form of NFR (Non Functional Requirements) document. This NFR should contain the data like Concurrent Users, Critical T Codes, batch Jobs, Volume of transactions, transaction response time, batch execution window etc.

There are two different categories of business process types we need to

identify in order to have smooth execution of the project, they are divided into Online users and Batch Jobs/Interfaces/Reports. You can’t just neglect Batch jobs which plays an important factor in successful of SAP performance testing. Like online users, critical & data intensive batch jobs needs to be identified along with Online users, the batch jobs time window, the SLAs for batch jobs how many number of records each batch jobs should be processed, the important success factor for batch jobs execution is that, are the batch jobs able to execute n number of records within the specified time window, for example Print batch jobs able to print thousands of records within 1 hour.

During this planning phase please note that there will be other parallel

activities that can be carried out, like identification of environments as well as the tools for performance testing, test data analysis, work load profiling etc. Performance testing is normally carried out on an environment that is representative of the Production environment in terms of hardware, software configuration and database size. Performance test tools will be used to generate scripts and simulate user load on the system under test. Please note that HP Load Runner will be the best tool that can be used for SAP performance testing as it’s also been widely used & recommended by SAP.

It is necessary to have Production size data in order to get accurate test results. There is a dependency on the other team like Data Migration to provide the required volumes of data for carrying out performance testing. During planning phase, test data



requirements like material number, customer account, GL account, Sold to Party, WBS Cost centre, etc will be identified and communicated to the Functional Team or Data Creation Team.

SAP system has several Business Streams like Sales & Distribution, Financial Accounting, Supply Chain Management, SRM, Record To Report RTR, HR, Real Estate, Material Management etc and these are called business streams in SAP language and given below are the example for 4 business streams identified for one of the Energy & Utilities client for performance testing.

2.1 BUSINESS STREAMS:

The following 4 streams ( for example, it could be more ) were identified for performance testing of SAP application, each stream’s transaction codes ( T codes) were identified with critical and most frequently used on daily basis also data intensive in nature as shown in the below tables for each streams. Again these should be put them into Online and batch jobs category.

SALES & DISTRIBUTION (SD):

• Create an order with five line items. (VA01) • Create a delivery for this order. (VL01N) • Display the customer order. (VA03) • Change the delivery (VL02N) and post goods issue. • List 40 orders for one sold-to party. (VA05) • Create an invoice. (VF01)

Note: Each benchmark user has his or her own master data, such as material, vendor, or customer master data to avoid data-locking situations.

Dialog Steps

0. Logon 10. Choose Enter

1. Main screen 11. Call /nVL02n (Change delivery)

2. Call /nVA01 (Create customer order) 12. Choose [F20] (Posts goods issue)

3. First screen 13. Call /nVA05 (List orders)

4. Second screen (with five items) 14. Choose Enter

5. Choose Save 15. Call /nVF01 (Create invoice)

6. Call /nVL01N (Create a delivery) 16. Choose Save

7. First screen 17. Call /nend

8. Choose Save 18. Confirm log off

9. Call /nVA03 (Display customer order)



User interaction steps 2 - 16 are repeated n times (15 user interaction steps --> min. 150 sec. duration).

FINANCIAL ACCOUNTING (FI):

Dialog Steps

0. Logon 11. Select first line

1. Main screen 12. Call Post incoming payments

2. Call Post Document 13. Enter header data

3. Create customer item 14. Choose Process open items

4. Create general ledger account item 15. Select item 5 of the list

5. Choose Post 16. Scroll down to the end

6. Call Display document 17. Select last item

7. Enter previous posted document 18. Deactivate all selected items

8. Double-click first line 19. Choose Post

9. Call Customer line item display 20. Call /n end

10. Enter data and choose Execute 21. Confirm log off


EMPLOYEE SELF-SERVICE PORTAL (EP-ESS):

User Interaction Steps of the EP-ESS 1. Log on to the portal – Welcome page (includes three iViews)

6. Choose Address

2. Choose Working Time —> Record Working Time

7. Choose Bank Information

3. Choose Leave Request 8. Choose Benefits and Payment —> Paycheck Inquiry

4. Choose Leave Request Overview 9. Log off from portal

5. Choose Personal Information —> Personal Data



Seven different business transactions are launched per loop (steps 2 - 8). Every loop is repeated n times for each user. The user think time is 10 seconds; thus, one loop takes at least 70 seconds.

MATERIALS MANAGEMENT (MM)

The Materials Management (MM) Benchmark takes you through a series of steps to create a purchase requisition for five materials (transaction ME51N), a purchase order for the five materials (ME21N), a goods receipt (MIGO), and an invoice (MIRO) for the purchase order

Dialog Steps

0. Logon 10. Choose Execute

1. Main screen 11. Choose Post

2. Call /nME51N (Create purchase requisition)

12. Call /nMIRO (Create invoice), enter company code

3. Enter data 13. Enter basic data

4. Choose Post 14. Choose Payment

5. Call /nME21N (Create purchase order) 15. Enter data

6. Enter data 16. Choose Post

7. Choose Post 17. Call /nend

8. Call /nMIGO (Goods receipt purchase order)

18. Confirm log off

9. Enter data


Like online critical transactions as depicted above, I have also mentioned some of the batch jobs names which we identified as critical for the performance test execution during one of the SAP performance testing project.



My suggestion to the members of the performance testing team to go through the business process involved in testing as many times as possible and get familiar with the business process/flow and note down the required data that is going to be used in test scripts, this will help them to proceed with much confidant and create more robust scripts with the accurate data.

2.2 TEST ENVIRONMENT

Ideally, the simulated load and the hardware used for the volume test should

be identical to the load and hardware of subsequent production operation. However, for various technical, organizational or economic reasons, it is not always feasible to simulate the full load expected in subsequent production operation. In addition, it is never really possible to execute the test on hardware exactly identical to the hardware planned for production.

But, as a best practice performance testing should be carried out on an environment which is mimicking close to the production environment hardware and software configuration, the next better option is to execute performance test on production environment before it’s Go Live, i.e. this should happen at least 6 weeks before the Go Live so that you can make use of the best. Please note that don’t try to do performance testing on any other environment which is not at all close or equivalent to production like.

Important Note: Make sure that the hardware used for the volume test is

exclusively reserved for this purpose while the volume test is run. In-parallel usage of the same hardware or system(s) for other purposes (as, for example, training system, ) can have disastrous consequences for the volume test and leads to misinterpretation of its results.

Note that the SAP system will exchange data with a number of internal and

external non-SAP systems, i.e. Interfaces and Non SAP systems like BACS, PAY Matrix, Billing System, Cyber Source etc.. In order to carry out Performance testing, it is essential to connect to the test environments of all the systems identified during testing.



In an SAP Landscape a typical test environment consists of the following components as shown in the below mentioned lines.

• ECC 6.0(ERP Central Component) • SRM 5.5 (Supplier Relationship Management) • SCM • FICO/HR/SD/MM/MDM/PM etc • PI 3.0(Exchange Infrastructure earlier Known as XI) • BI Business Intelligence • EP (Enterprise Portal) • And other Non SAP system integrated with SAP

2.3 TEST DATA MANAGEMENT

The data required for Performance testing should be identified during the

Performance test planning phase. The data to be set up within SAP as well as the data required connecting and test with external systems will be defined.

Since database tables (especially transaction data tables) start to grow after

start of production, database accesses that are not fully specified tend to get slower and more resource-expensive. If the volume test is performed with almost empty database tables, the response times in subsequent production operation will be higher than measured in the volume test. Therefore, for the volume test to be meaningful, you need to populate the database tables with representative data that is both master and transactional data.

We must understand & set the expectation to the stakeholders that which team is going to provide all valid transactional data and who is going to create various user ids and passwords with proper roles & authorizations for the realistic simulation of user load. For example all HR user ids can’t have Sales & Distribution Roles, and all SCM user ids may not have HR roles and authorizations. Some of the example below shows what data is required during performance testing..

• Master Data, o Customer Account Number o Sold To Party o WBS Cost Element o Material Number o Employee Number etc.

• Transactional Data o Bank A/c number o User Name o GL Account Number etc.

• User Data o Delivery Date o Material Price o Customer Name etc.



Please note that the user ids and password for emulation of the load should be created by the Tech arch team or BASIS team (SAP back end administrators are called as Basis), make sure that the team creates the user ids with appropriate naming conventions so that each users can be identified in the back end of the system during performance testing as shown in the examples below. Sales & Distribution SD

Supply Resource Management SRM

Supply Chain Management SCM

Enterprise Portal EP

Finance & Accounting FI

SD0001 SRM0001 SCM0001 EP0001 FI0001 SD0002 SRM0002 SCM0001 EP0002 FI0002 ….. ….. ….. …… ….. SD0200 SRM0400 SCM0150 EP0800 FI0300 Any data (Transaction data, user data etc) that is required during performance testing will be provided by each business streams.

2.4 TEST TOOL & SETUP

The diagram below shows that the performance test lab setup which make use

of 4 Load injectors (Load generators) with one Loadrunner Controller, all the m/c are connected to each other on a business LAN which has access to SAP system, each system has the 2GB RAM with Intel CPU speed 2.8GHz.

Please note that the machine running a scenario with SAPGUI Vusers must

have a Loadrunner typical installation. Running Vusers from machine with only the controller installed is NOT SUPPORTED, and each Loadrunner systems (Injectors and Controller m/c) should have the SAPGUI client installed so that they can talk to back end SAP ECC system. LoadRunner is HP (formerly Mercury Interactive) load/performance testing tool. It operates using a record/replay method in which the software allows a user to record a business process and then replay it. LoadRunner can simulate many users on the system at once. It accomplishes this by capturing all data that passes from the client machine to the SAP server and back while recording. When replaying the script, LoadRunner sends the same data that it captured back to the server, thus performing the exact same business processes that were recorded thus by emulating real users.



Fig.1 Performance Test Lab Setup with the Environment

Terminal Services: Machine running SAPGUI Vusers may be limited in the number of Vusers that can run, due to the graphic resource available on that machine and due to the limitations imposed by Microsoft windows on GDI objects and user objects.

To rectify the above mentioned issue and allow more SAP users to be run, it is recommended that multiple Terminal Server sessions be run on the load generator machine (Injector) and relate each terminal server session to the load generator. This means a Terminal Server must be running on the load generator.

Please note that you can run maximum of 30 to 50 SAPGUI users on each terminal session, MS Windows 2003 server Standard edition should be installed on each Load Generators machine in order to run terminal sessions, by default you can have three terminal session run on each machine, so with this 3X50=150 total users can be run each m/c, if you want to run more sessions then you need to buy the license from MS vendor in order to increase the session which helps you to run more number of SAPGUI users. Let us recap the following points in order to complete the Planning phase, The main activities in test preparation phase include:

• Identification of Test Scenarios for Batch job window • Identification of Test Scenarios for On-line users (SAPGUI and Portal users) • Set up and configure the Production environment with test data.



• Test Plan & approach document • NFR (Non Functional Requirements)

The success factor: • NFR document Sign-off • Performance test plan & approach document sign-off by the stake holders. • Tools & Environment Setup completion. • Identification of Test Data. • Test Data Management

3. BUILD / TEST SCRIPTS DESIGN Once the planning phase is in place and signed off from the business, with the input from NFR and from the functional team, its time to design work load module & develop automated scripts in order to simulate the real users. Before we start scripting, we need to change some settings on the SAP server which enables the automated tool to record user actions on SAPGUI protocol.

3.1 SERVER SIDE SETTINGS

Before you start recording in the tool, you need to set certain parameters in SAP server in order to generate the script

• Enabling scripting on Server Side: By default, the scripting API is disabled in SAP R/3 system,. To enable scripting API,

• For all users of a given system, the system administrator needs to set the “sapgui/user_scripting” profile parameter to “TRUE” on all the application servers and make sure that the above value TRUE should be in upper case,

• The administrator needs to use the T code RZ11 in order to change the above profile parameter.

3.2 CLIENT SIDE SETTINGS

Enabling scripting on Client Side: You need to enable Scripting as circled in the red colour in the screen shot

below and disable all check boxes as shown in the screen shot shown below. In order to do this, log on to SAP server, select the “Rainbow” Icon and select “Options”. And you must disable the remaining two check boxes

1. Notify When a Script attaches to a Running GUI 2. Notify When a Script Opens a Connection



Configuring the Model Dialogue Boxes for F1 and F4-Client Side: a) For F1 In order to configure this option, Log on to the SAP serverà HelpàSettings, select F1Help Tab and select the in modal dialog box (R/3) option in the display section as shown in the screen shot below.



b) For F4 To open F4 help in model dialog boxes, the following procedure must be completed by a SAP administrator. Choose Help>Settings. Click the F4 Help tab. In the Display section (Bottom Left), choose system defaults. In the display portion of the system defaults sections (bottom right). Select the Dialog. Click on the pencil icon first before you can change the sections. Click on the icon again to save lock the selection as shown in the screen shot below

Once the above settings are configured on the SAP App Server, please select SAPGUI protocol for creating SAPGUI scripts and SAPWEB protocol for creating Enterprise Portal scripts. The scripting for SAPGUI users and SAPWEB users are quite simple and not to worry much about scripting part as its 80% direct record & Replay with very few code enhancements in SAPGUI, and few complex correlations in SAP web EP applications. There is not at all correlation concept in SAPGUI except capturing the o/p of one process and input the same value for next process, for this you can use Loadrunner sapgui_get_Status_bar () functions.

Once the test scripts are developed and enhanced to meet the business

requirements, parameterise various data, correlating the dynamic values etc, these scripts should be run for various iterations with different data inputs and cross check with the records created in the back end of the system. Once you have confirmed that all scripts are doing what its intended to do, then get these scripts run with the business team to confirm that they are in line with the NFR and satisfies their requirements.

At this point of time, ask the Functional team to identify the Batch & Schedule the batch jobs for the batch execution window, usually these batch jobs are scheduled



and executed by Functional Team either manually or by using some tool like Controller M ( is a batch job work load scheduler tool )

The success factor: • Successful development & Replay of Automated Test Scripts. • Make sure that you use coding standards implemented in all the scripts. • Automated Test Scripts Sign-off by Business Team. • Production like setup batch jobs scheduled on the test environment. • Batch jobs are successfully smoke tested.

4. EXECUTION The approach for performance test execution will be carried out in the following manner, and it should look like this as shown below.

• Performance Testing of Online Users (Individually for Online users) • Performance testing of Batch Jobs (Individually for Batch Jobs) • Combine Online Users and Batch Jobs together and execute together

4.1 EXECUTION OF ONLINE USERS – In SILO

Once the test scripts development is completed then all the above identified streams like S&D, FI, EP(ESS) and MM should be executed in Loadrunner controller for their respective concurrent loads and analyse the result of each execution to find any bottlenecks in the stream wise. Please note that the execution of these streams should be done in isolation of each stream (i.e. stream S&D T codes should be executed individually in isolation of other streams). This helps the project team to find any performance bottleneck in the early phase of the performance test execution.

After successful execution of individual streams, its now time to execute all streams together for concurrent number of users defined in the NFR and analyse the performance of the system overall

4.2 EXECUTION OF BATCH JOBS & REPORTS – In SILO Batch performance models will be built to represent the average and peak load of

batch processing in SAP systems. Batch scheduling (Functional) team will define the production batch schedule and estimate average and peak load. This helps the team to know if any batch jobs & Reports are taking much time and not fitting the batch window (For example 10PM to 2:00AM ).

Batch performance model will include the daily batch jobs running in ECC, BI, SCM, SRM, XI, and JCAPS etc

4.3 APPROACH FOR BATCH JOBS PERFORMANCE TESTING

Below is the detailed approach for Batch jobs performance testing..

Objective v Check batch job performance in ECC XX environment. To ensure current

level of performance. Pre-requisite v Deployment of Control M* and related application on the pre production

environment for the identified critical interfaces.



v End to End connectivity setup for flow. v Charged day identification for data recording/ input file for the batch jobs.

v Configure monitoring tools on ECC XX to capture the performance metrics for the batch executions.

Approach v Capture batch job baseline data of selected interface from the current ECC

system. v All the critical batch jobs which are to/from the SAP system should be

considered ( including third party systems) v Trigger Job on SAP ECC system using Control M for reference measurements v Use the input file and process in ECC XX Environment v Trigger Job in ECC XX system using Control M for validation v Tuning if required v generate performance report

*Please note that Control M is batch job scheduler tool like AppWorkx A typical 24 hrs batch jobs approach is depicted in the below diagram, which illustrates the batch jobs executions in SAP and Non SAP integrated environment, please note that the batch jobs should be given much importance as process large amount of data to/from the SAP & third party applications and they need to meet the performance requirements/KPIs as per the NFR document.

Fig: Typical Batch Jobs executions in SAP performance testing



4.4 EXECUTION OF ONLINE USERS AND BATCH JOBS & REPORTS (Combined) After completion of performance test execution for online users and Batch Jobs, we

need to combine both the scenarios Online Users & Batch jobs to emulate the real scenario and measure the system performance of SAP R/3.

The below screen shot shows how the scenarios with the load generator look like, please note that all the load generators sessions are running on MS terminal server sessions and each session is mapped with their respective names like as shown in the screen shot below.

4.5 RUN TIME SETTINGS FOR EXECUTION:

The below screen shot shows the various RunTime Settings for the scenarios

executions, its important to apply these runtime settings to in order to execute the scenario successfully.

General: à Run Logic à No Change à Pacing à With fixed delay of 40 seconds. (Depends) . à Log à Standard Log

àThink Timeà Replay Think Time, use random percentage with Min 100% Max 150% with Limit Think Time to 10Sec



à Additional attributes à No Change à Miscellaneous à Run Vuser as Thread

SAPGUI:à General à Disable Show SAP client during Replay Network:à Speed Simulation à Use Max Band Width.

4.6 SCHEDULE BUILDER

Go to Edit Schedule and Set the following settings in the schedule builder as shown below.

• RampUp: Start 5 Vusers for every 30 Seconds. • Duration: Run for 5 Hours (depends up which test you are doing, for example

duration should be 12 to 24 hours). • RampDown: Stop 5 Vusers for every 60 Seconds.



Checklist before execution: HP LoadRunner script testing will occur after creation of each automated script. Scripts can be added to performance test bed when:

1. No scripting errors occur. 2. Looping successfully; replaces parameterization variables with various values. 3. Activity is noted on each system tier. 4. A minimum load of ten users is simulated. 5. Scripts are run on a subsequent day (dates correlated correctly).

5. MONITORS & ANALYSIS You need to configure the following monitors in order to analyse the loadrunner results, if you don’t have SAPGUI monitor license then you can ask the Basis team to monitor the application through CCMS tools (using STO3n Tcode) which will give same data as of SAPGUI and Database Monitors.

• SAP Monitor (SAPGUI Monitor) • UNIX Resource Monitor. • Oracle Database Monitor • Windows Monitor for WAS server

5.1 SAPGUI MONITOR

The following measurements will be taken for each script as it is executed in LoadRunner:

Response time: This starts when a user request enters the dispatcher queue; ends when the next screen is returned to the user, this will measure the end to end response from the client to the system



Wait time: This is the time a user request sits in the dispatcher queue. Roll-in time: This is the amount of time needed to roll user context into the

work process. Load time: This is the time needed to load from the database and generate

objects like ABAP code, CUA, and screen information. Processing time: This is the time spent in the work process executing ABAP code. Database time: Starts when a database request is put through to the database

interface; ends when the database interface has delivered the result.

CPU time: This is the total CPU time used by the SAP R/3 work process.

The SAP monitor displays statistics about the resource usage of a SAP R/3 system during the scenario or session step run. This monitor is appropriate for SAP R/3 server and SAPGUI (version 6.20 or later) environments.

SAP GUI transactions help the tech arch team and performance team to quickly isolate the performance issues within the SAP R/3 infrastructure, and to authoritatively delegate problem resolution to the right SAP application teams. The solution provides a layered breakdown of performance of SAP R/3 transactions into database, system, processing, and interface layers. Database layer shows the amount of time spent by an end-user business process or an individual dialog step in the database tier. System layer shows a breakdown of time the dispatcher is waiting for a free work process, time taken to load or generate ABAP programs by the work process, time taken to associate the user context with a work process, and time taken to save user context data on the outbound path. Processing layer shows the amount of actual time taken by the work process to process the ABAP programs and SAP application server. Interface time refers to the performance characteristics of accessing external systems - either SAP or non-SAP — such as Remote Function Call time. Below diagram shows the break down of each layer and this should help the team to find out the root cause of the bottleneck.



5.2 UNIX RESOURCE MONITOR

UNIX monitor gives the system resource parameters of Unix server box, in order to run UNIX monitor successfully on Loadrunner controller, you need to run rstatd daemon service on Unix server box, you can ask database admin team to set this service running, or else you can ask him to carry out the following steps as shown

To configure the rstatd daemon: 1 Run the command: su root 2 Go to /etc/inetd.conf and look for the rstatd row (it begins with the word rstatd). If it is commented out (with a #), remove the comment directive, and save the file. 3 From the command line, run: Kill -1 inet_pid Where inet_pid is the pid of the inetd process. This instructs the inetd to rescan the /etc/inetd.conf file and register all daemons which are uncommented, including the rstatd daemon. 4 Run rup again. If the command still does not indicate that the rstatd daemon is configured, contact your system administrator. PS: You can configure the remaining monitors like Oracle Database and Windows resource monitor just by adding their respective IP addresses.



6. SAP PERFORMANCE MEASUREMENT & METRICS The following activities should be planned for the performance test: • Use the SAP system monitoring procedure to check the peak hour’s activity. • Identify long running jobs and run it to test server performance. • Run critical business transactions and measure runtime. • Define peak hours and the system to be monitored during execution in order to

identify performance problems. The monitoring should be done using SAP’s Comput ing Center Management System

(CCMS) and the UNIX utility “vmstat”. CCMS provides performance monitors that track various aspects of the SAP systems while the “vmstat” provides detailed operating system performance information.

SAP system performance actually depends on many factors such as modules

implemented, database size, and number of users. Because of this, there are no standard values that constitute good performance. However, based on experience and SAP recommendations, the Technical Architecture Team has determined the following stress test performance metrics are indicative of satisfactory performance:

6.1 WORKLOAD ANALYSIS

Transaction ST03n records the response time of the system and of individual

transactions. It can be used to see how fast the system is responding. ST03n also divides response times into CPU time, wait time, load time, and database request time. These times will be evaluated as a percentage of Average Response Time (ART) for each server during each test so that it can be determined where increases occur.

Parameter Requirement

Measuring Tool

Description

Average Response Time (ART)

<1 second (<1000 ms)

ST03n Response time of a dialog step. Excludes network transmission time from application server to presentation layer (SAPGUI)

Average CPU Time <50% of ART

ST03n CPU time per dialog step

Average Wait time <1% of ART

ST03n dispatcher queue time per dialog step

Average Load time <10% of ART

ST03n load time per dialog step

Average DB request time

<50% of ART

ST03n database request time per dialog step

Direct Reads <10ms ST03n average time for direct reads per dialog step

Sequential Reads <40ms ST03n average time for sequential reads per dialog step

6.2 MEMORY/BUFFER MANAGEMENT

SAP stores frequently used information in buffers in memory. Transaction ST02

monitors the usage of these buffers. The most critical factors it reports are hit ratio, free space, and swaps. The hit ratio is the percentage of requests that can be fulfilled by the information in each buffer (preventing expensive database requests), and perhaps is the most



important statistic on the screen. Free space measures how full the buffer is, and swaps show how many times data had to be removed from the buffer to make room for other data.

The Hit Ratio of the Single Record Buffer increases very slowly from system startup:

therefore, a hit ratio less than 90% is a concern only if there is no free space left in the buffer. Low hit ratios are not always due to a performance problem, as an example, a transport into a system can reduce hit ratios.

Parameter Requireme

nt Measuring Tool

Description

Hit Ratio*** >95% ST02 Percentage of requests filled by SAP buffer (memory) and not disk.

Free Space >5% ST02 the unused space percentage in each buffer

Free Directory >5% ST02 free directory percentage in each buffer

Swaps <3 ST02 number of times an object was swapped from a buffer to make room of another object

6.3 DATABASE MANAGEMENT

Parameter Requireme

nt Measuring Tool

Description

Data buffer busy waits

<5% of reads

ST04 ratio of data buffer busy waits to reads

Recursive Call Ratio

<10% ST04 ratio of recursive calls to user calls

Parsed Call Ratio <25% ST04 ratio of parsed calls to user calls Sorting Ratio <5% ST04 ratio of disk sorts to memory sorts

Transaction ST04 reports detailed database usage statistics. The Technical team will

watch the buffer, recursive calls, parsed calls, and sorting on this screen. Recursive calls are a result of low DD (Data Dictionary) cache and parsed calls are a result of low library cache, both indicate deficiencies in the Shared Pool size.

6.4 UNIX SERVERS (OS Parameters, CPUs, Memory) For UNIX Servers, it is recommended to check memory usage during peak hours and do the following:

• Check CPU utilization during peak hours • Check SWAP space usage during peak hours • Check shared memory

The monitoring should be done using UNIX utility “vmstat”.



CPU Management While monitoring the CPU, the following guideline needs to be followed:

Parameter Requirement

Measuring Tool

Description

Utilization <90% vmstat None of the CPUs should have utilization of more than 90%

Load Average <3 vmstat No more than 3 processes should wait for the CPU

To monitor the operating system performance during stress testing, the UNIX utility

“vmstat” needs to be used. This utility should be set to record data at a set interval (30 seconds) and send it to a binary file that can be analyzed after the test. .

Work Process CPU Time: Transaction SM50 should be used to gather information about CPU times used by

each work process in the application and database servers during the test. To calculate CPU times during each test, the cumulative CPU time of each work process before the test was subtracted from the cumulative time after the test (shown in ss:hh = seconds, hundredths of seconds).

The CPU usage of work processes gives an idea of the proper number of work

processes a server needs. If all the work processes of a particular type (like all the dialog work processes) use large amounts of CPU time, more work processes of that type are possibly needed. Conversely, if some work processes have a very small change in CPU time, they can possibly be removed from the configuration.

7. SAMPLE GRAPHS

7.1 UNIX RESOURCE GRAPH

Below are the sample UNIX performance graphs for your reference.



7.2 EXECUTIVE SUMMARY GRAPH

7.3 AVERAGE RESPONSE TIME GRAPH FOR T CODES



7.4 BATCH JOB/INTERFACE SAMPLE REPORT

Below is the sample report for the batch jobs execution from SAP system

8. APPENDIX

A. LOADRUNNER HARDWARE & SOFTWARE

Following are the hardware configuration and software recommended for setting up performance test lab. The same configuration is applied for LR Controller, Load Generators and VuGen

Hardware Requirements ( depicted only and can be make)

• IBM P5 m/c Intel CPU 2.8GHz, 2GB RAM, 80GB HDD with MS Widows 2003 Release 2 Standard Edition, Service Pack 1

Software Requirements

• Windows 2003 (Standard and Enterprise editions), this is required if you need

to run more SAP GUI users on a single load injector • To install LoadRunner on a Windows platform, you must log in as a Local

Administrator. • Requires Internet Explorer 6.0 SP1 or later. • If you are running McAfee or Aladdin's eSafe anti-virus applications, close

them before installing LoadRunner.

B. SAP T CODES & ITS FUNCTIONS

Tcodes Function /n Exits the current R/3 screen and displaes initial screen

/o Sends the current user session to the background and creates a new user session to display the initial screen

/h Turns the debug mode ON AL08 Displays the list of all users logged on AL11 Displays SAP directories SM51 List of servers



SM59 Details about RFC connection and load balancing properties. SM21 System log SM37 Simple job selection ST06 CPU, memory and swap memory usage of the server ST07 Details on application monitors ST02 SAP buffers and tune summary monitor SM66 Global work process overview

SM50 Process overview; Shows the currently executing work processes on a dialog instance

ST03

Workload distribution monitor. Can be used to analyze performance bottlenecks in database, R/3 memory management and buffers.

ST04 Database performance monitor AL12 Display table buffers - buffer monitoring tool ST10 Table call statistics SM31 SAP Table maintenance SE13 Specifies whether a table should be buffered or not

SE16

We can provide the table names - LIKP, VBAP/VBAK and look for details Inbound/Outbound deliveries. VBRK - for retrieving the invoice numbers; select billing type as ZF22 FAGFLEXA - provides the number of entries in GL_LINE_ITEM_FACT

ST05 View SQL traces ST12 Error log files

WE02 idoc Order status, check for status03 which indicate the successful

VL10A

Sales order status and also we can run a background job (variant program) for creating delivery notes for the orders received in ECC.

C. SAP WEB CORRELATIONS

Please note that following consideration should be given for the quick correlation in SAP web which will save much time for the team to identify the correlations.. Ø JessionID Ø Sap_ext_sid Ø Sap_wd_tstamp Ø uwlSessionID

//(J2EE413382100)ID0658168150DB5168491f44203a1890a55a7862908a86b43764dbEnd is replaced with parameter {C_jsessionid}. web_reg_save_param( "C_jsessionid", "LB=jsessionid=", "RB= \"", "Ord=1", "Search=Body", LAST );



// fK4qxjVJPG3OBy_pCnIKKQ--fdvZw3_mBW8ve_DxKg9Mkg-- is replaced with Parameter {C_sap_ext_sid} web_reg_save_param( "C_sap_ext_sid", "LB= value= \"", "RB=\"", "Ord=1","Search=Body", LAST ); //1322068237859 is replaced with Parameter {C_sap_wd_tstamp} web_reg_save_param( "C_sap_wd_tstamp", "LB = value=\"", "RB=\"", "Ord=3","Search=Body", LAST ); // 1322068237858@uwl@2599 is replaced with Parameter {C_uwlSessionId} web_reg_save_param( "C_uwlSessionId", "LB= value= \"", "RB=\"", "Ord=12","Search=Body", LAST );

D. REFERENCES

• For all the screen shots Loadrunner tool & Loadrunner online references • sap website for business process steps

Documents

BEST PRACTICES FOR SAP R3 PERFORMANCE TESTING