How to Conduct Load Test

8/6/2019 How to Conduct Load Test

1/16

1

H ow t o Se t U p a Cu st o m er L oa d T es t

Performance, Data Management & Scalability

Learn ing Object ives

As a result of this session, you will:

Estimate the required resources for a performance test

Create a project plan for a custom performance test


2/16

2

1. Introductory Remarks

2. Planning a Custom Performance Test

3. Walking Through the Process Steps

4. Conclusion

Agenda

Di f ferent Per formanc e Test s (1 /2)

Performance AnalysisDetermine how fast some aspect of a system performs, such as

Single user tests

A single transaction test

Understand what parts of the system or workload cause the system to performbadly (diagnosis case)

Sca labi l i ty Test ingMeasure system behavior with varying workloads

Determine effects of adding additional hardware to distribute work among systemcomponents

Testing for linear resource consumption to obtain scalability with

Number of objects, size of objects, number of concurrent users,

Can be a series of single user performance tests as well as of multi-user stresstests


3/16

3

Dif ferent Per formanc e Test s (2 /2)

Volume Test ing or Load Test ingBlanket term that is used in many different ways across the professional software

testing community

Often used as a synonym for multi-user (load) testing

Practice of modeling the expected usage of a software program by simulating theusage of specific program services, thus representing important business scenarios

Concurrent users, that is, online activity

Background jobs

Various workloads (expected daily load)

Stress Test ingSubset of load testingDetermines the breaking point or unacceptable performance point of a system

Demonstration that the system meets performance criteria

Reflects peaks beyond the expected daily load

Per fo rmance Tes ts Approaches

Single user testSmall test system (QA, development), oneuser

Analyze & measureScalable behavior

Quality and implications of accessesto persistence layer

Linear resource consumption

Parallel processing mechanisms,load balancing

Memory usage and memory leaks

Disk requirements

Front-end network load

Performance predictions/sizingfor high volume environment

Volume tes t

Equivalent to multi-user test,stress test, load test Verification of assumptions +


4/16

4

Some Goals of High Volume Test s

Verification of scalable software and system infrastructure

Load balancing

Parallel processing

Locking behavior

System tuning and parameterization

Bottleneck analysis for infrastructure

Fail-over, back-up and disaster recovery strategies

Monitoring and system administration, calibration of thresholds

Robustness

No deadlocks

Data consistency

Possible performance degradation in overload situations

Verification of sizing models

Identification of memory leaks (especially with Java applications)

Cri t ic a l Succ ess Fact ors of High Volume Tests

Top-bottom: Success of the project depends on

Project management

Multiple stakeholders

Human and financial resources

Time restrictions

Handling infrastructure / hardware expenses

Dedicated test systems, the more complex, the more servers Complexity of test cases and their setup

Availability of test systems

Discipline during the testing: Avoid changes to test case / process

Expertise to analyze test results, especially to identify critical components

Simulation of proper system behavior

Note

Single analysis including tracing and profiling still has to be done


5/16

5

Side St ep: The Right People on the Bus

High as compared to s ing le user tes tsPeople from different departments and with different levels of expertiseare needed for the execution:

Development analysis of test results, setup of tests

Application Experts definition of test case including test data, definition of KeyPerformance Indicators

IT test environment, system administration

Test tool specialist

Performance measurement and monitoring specialist

In the single-user tests you may require

One application specialist One developer

(in most cases the developer is the application specialist)

One performance specialist

Three or all-in-one

Project

complexity

Info

rmatio

nal

valu

eofresults

Hardw

are

costs

Single user test

Volume load test

Effortof

executio

n

Analysisofcustom-

specificbehavior

Rat ing of Test Opportun i t ies


6/16

6




4. Conclusion

Agenda

A few Words on the Tes t Sys tems

System for s ing le user test Possibly QA-System, Development System

Goals: test functionality, remove performance bugs, testchanges in settings

2-CPUs (to be somewhat independent from other load)

Less than 10% CPU utilization

Load tes t sys tem Volume test

Any configuration (close to planned production)

Dedicated, possibly QA?

Separate sys tem s para l le l ac t iv i t ies


7/16

7

A few Words on the Team Members &

Assumpt ions

Teams Application team (AT)

Includes people responsible for implementation and application experts

Test system admin and technical team (IT)

Ski l ls Know the application and business requirements

Understand complete system architecture

Perform system analysis

I/O, OS, RDBMS, WebAS,

Where are the Team s involved

xxEvaluate test results

xxPerform volume tests

xxPerform single-user tests

(x)xCreate actual test data

x(x)Decide on methods of load and user simulation

xDetermine infrastructure of load test system

xSet up test scenarios for test systems

(x)xDo initial capacity planning for load test system

xEstablish test scenarios for business processes

xDetermine data distribution

(x)xDetermine needed functionality

xSet throughput and response time expectations

xDetermine critical business processes

ITApplicationMain Tasks


8/16

8

Most Tasks Can Run in Para l le l (Example)

Evaluate test results

Perform volume tests

Perform single-user tests

Create actual test data

Decide on methods of load anduser simulation

Determine infrastructure of loadtest system

Set up test scenarios for testsystems

Do initial capacity planning forload test system

Establish test scenarios forbusiness processes

Determine data distribution

Determine needed functionality

Set throughput and responsetime expectations

Determine critical businessprocesses

Mile stones

Actual hands-on

Planning steps

Elapse time: 3 months+




4. Conclusion

Agenda


9/16

9

Steps of a Load Tes t (1/8)

0. A l loca te p rojec t m embers, make sure they can ded ica tethemselves to the task1. Determine the bread-and-butter processes

Determine what users require for their daily work

Define top 5 (or 10) business processes for performance and scalability testing

Examples for important business processes

Daily high volume upload at night

Strong reporting at year-end closing

2. Document the expec ted th roughput f i gu res and /o r userand sys tem response t imes Include end users or implementation team

For example:

Upload of 75,000 sales order line items in 15 minutes

Upload orders and run MRP for 300,000 products in 2,5 hours

Processing of nn http requests per sec from mm parallel users

Expected response times 0.5 s in an Interaction Center

Entire scenarios can be included as well

Usually these figures are part of a requirement or specification document

Milestone: Target Def in i t ion is Set

This target definition will show whether your entire project is a successor not

Why? These expectations will become your internal benchmarks against which

all new releases have to be compared to

Also, he lpfu l for assessing the rea l costs of t he test Complexity

Batch tests are "cheaper" than online usage simulation


10/16

10


3a. Determine the complet e set o f func t ional i t ies neededfor the load tes t and the resul t ing sof tw arecomponents For example

ATP (Available-to-Promise) Check in APO system

Pricing in IPC

Workflow or interfaces to other systems

3b. Def ine t he scope Discuss if the chosen business processes can actually be implemented

In close cooperation, but with a view from the IT perspective settle for the possible

For example

The AT might want to test the year-end closing but the IT team finds out thatimporting the data from another system may take too long

So the decision may be taken to use only 50% of the resources and test only

50% of the volume and then scale out


4. Determine t he data d is t r ibut ion i .e . the t ype of mas terdata you need Avoid artificial bottlenecks, for example all orders use the same customer and

products

It is not enough to create a vast amount of data by simply copying it

5. Create t es t procedures Starting conditions must allow reproducible measurements It might be necessary to ensure this, using a backup and restore strategy, in other

cases a clean-up program may be sufficient

Even if you test in a small system you ought to have regular back-ups in order to starta test from scratch again or to repeat a test phase

Specify in detail and document well

Individual and detailed process steps

Input / output for the steps

Customizing environment


11/16

11


6. Size and lay out o f the t es t sys tem Do sizing based on planned data volume (Quick Sizer, )

7. IT team sets t es t scenar ios Create detailed test cases for infrastructure

Example: WAN infrastructure, hardware load balancer

8. Prov ide and set up load tes t sys tem and sys teminf ras t ruc ture Prepare test environment

Servers, front ends, disk layout, instances and clients

Ensure that the system landscape is well configured

Backup and restore strategy in place

In some cases it makes sense to start testing with a small system which yougradually extend


9. Dec ide on met hods of load and user s imulat ion Decide what data is inserted by real users or by tools

Advantage of real users

Training for end users

Feedback loop

Which tool is going to be used, is determined by the IT Team

Responsible for implementation Tool experience

10. Create tes t data For example: master data, customizing,


12/16

12


11. Run the s ing le-user tes t on a smal l tes t sys tem For optimal results: have the system all to yourself

Stable

No background processes, CPU utilization below 10%

Measure performance and analyze the application

Check scalability of application,

e.g. linear resource consumption

Adherence to performance guidelines

Eliminate performance bugs

Side Step : Tes t ing fo r L inear Resource

Consumpt ion

To test whether the solution is scalable, perform "linearity tests"

These tests are basically a repeatable pattern for (example):

Scalability with an increasing number of objects

1 object, five line items

10 objects, five line items

100 objects, five line items

Scalability with an increasing of line items 1 object, 1 line item

1 object, 10 line items

1 object, 100 line items

The results of these controlled tests can be analyzedto determine if any performance bottlenecks exist.


13/16

13


12. Run the vo lume load t es t Start with a small load and increase load step-by-step towards the final throughput

target

Measure the throughput

Measure the resource consumption using appropriate monitoring tools

Carefully monitor the expected performance bottlenecks

Optimize system parameter settings and repeat the test executionsif necessary


13. Evaluate the t es t resul ts Document the achieved throughput numbers and the corresponding resource

consumption

Document the limiting components (bottlenecks)

Document the test system environment (hardware and software releases) carefully

The numbers may be needed for sizing of further systems


14/16

14




4. Conclusion

Agenda

Concluding Remark s

Del iverables f rom vo lume t es ts Sizing verification

Proposals for optimal system configuration and parametrization

"Operations handbook", lessons learned

Tuning

Monitoring

Backup & recovery

High availability

Data archiving

Background job scheduling

Escalation procedures

Settings for thresholds (alerts)


15/16

15

Too ls & Exper ience

SAP Tools eCATTs

For single user tests

Particularly handy for multiple testing of scenarios

Together with Global Performance Analysis

Benchmark tool kit

For volume load tests

Keep i t s imple Focus on most important scenario/component/process otherwise youll lose focus

Part and parc el Project management across teams, organizations, companies

Bear in mind milestones, deadlines

Drive team members

Communicate

Documentation

THANK YOU FOR YOUR

ATTENTION !

Q & A

QU EST ION S SU GGEST ION S DI SCU SSI ON


16/16

16

Copyright 2008 SAP AG

Al l r ights reserved

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changedwithout prior notice.

Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.

SAP, R/3, xApps, xApp, SAP NetWeaver, Duet, SAP Business ByDesign, ByDesign, PartnerEdgeand other SAP products and services mentioned herein as well as their respective logos aretrademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned and associated logos displayedare the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.

The information in this document is proprietary to SAP. This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This documentcontains only intended strategies, developments, and functionalities of the SAPproduct and is not intended to be binding upon SAP to any particular course of business, product strategy,and/or development. SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, orother items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties ofmerchantability, fitness for a particular purpose, or non-infringement.

SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitationshall not apply in cases of intent or gross negligence.

The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in thesematerials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages

Weitergabe und Vervielfltigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrckliche schriftliche Genehmigung durchSAP AG nicht gestattet. In dieser Publikation enthaltene Informationen knnen ohne vorherige Ankndigung gendert werden.

Einige von der SAP AG und deren Vertriebspartnern vertriebene Softwareprodukte knnen Softwarekomponenten umfassen, die Eigentum anderer Softwarehersteller sind.

SAP, R/3, xApps, xApp, SAP NetWeaver, Duet, SAP Business ByDesign, ByDesign, PartnerEdgeund andere in diesem Dokument erwhnte SAP-Produkte und Services sowie diedazugehrigen Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und in mehreren anderen Lndern weltweit. Alle anderen in diesem Dokument erwhnten Namenvon Produkten und Services sowie die damit verbundenen Firmenlogos sind Marken der jeweiligen Unternehmen. Die Angaben im Text sind unverbindlich und dienen lediglich zuInformationszwecken. Produkte knnen lnderspezifische Unterschiede aufweisen.

Die in diesem Dokument enthaltenen Informationen sind Eigentum von SAP. Dieses Dokument ist eine Vorabversion und unterliegt nicht Ihrer Lizenzvereinbarung oder einer anderenVereinbarung mit SAP. Dieses Dokument enthlt nur vorgesehene Strategien, Entwicklungen und Funktionen des SAP-Produkts und ist fr SAP nicht bindend, einen bestimmtenGeschftsweg, eine Produktstrategie bzw. -entwicklung einzuschlagen. SAP bernimmt keine Verantwortung fr Fehler oder Auslassungen in diesen Materialien. SAP garantiert nicht dieRichtigkeit oder Vollstndigkeit der Informationen, Texte, Grafiken, Links oder anderer in diesen Materialien enthaltenen Elemente. Diese Publikation wird ohne jegliche Gewhr, wederausdrcklich noch stillschweigend, bereitgestellt. Dies gilt u. a., aber nicht ausschlielich, hinsichtlich der Gewhrleistung der Marktgngigkeit und d er Eignung fr einen bestimmten Zwecksowie fr die Gewhrleistung der Nichtverletzung geltenden Rechts.

SAP bernimmt keine Haftung fr Schden jeglicher Art, einschlielich und ohne Einschrnkung fr direkte, spezielle, indirekte oder Folgeschden im Zusammenhang mit der Verwendungdieser Unterlagen. Diese Einschrnkung gilt nicht bei Vorsatz oder grober Fahrlssigkeit.

Die gesetzliche Haftung bei Personenschden oder die Produkthaftung bleibt unberhrt. Die Informationen, auf die Sie mglicherweise ber die in diesem Material enthaltenen Hotlinkszugreifen, unterliegen nicht dem Einfluss von SAP, und SAP untersttzt nicht die Nutzung von Internetseiten Dritter durch Sie und gibt keinerlei Gewhrleistungen oder Zusagen berInternetseiten Dritter ab.

Alle Rechte vorbehalten.

Documents

How to Conduct Load Test