Performance Testing DesignBy Omri Lapidot

Symantec Corporation

Email: olapidot@gmail.com

Mobile: 0544-497179

At SIGiST Israel Meeting

November 2007

Agenda

• Why Test Performance?• How not to test• Performance Testing Phases

– Designing a Usage Model– Tests and environments creation– Load and tune

• The Politics of Performance Tests• Use Case: Symantec I3• Summary

System under test assumptions

• User Interface which multiple users manipulate concurrently

• Core applications• Core database with data loading

mechanism• Dynamic components customers choose

and install

– Without performance testing, functionality is likely to suffer in increased loads• Most QA is done on under-loaded environments and

unrealistic configurations designed for locating specific functional problems

– Performance tuning is a repetitive, cyclic process• Performance problems can not be addressed via normal

patching process unlike functional problems– Determine realistic configuration

recommendations• Customers and field personnel need to know what are the

recommended hardware and software configurations– Let the field people know what they are up against

• If we release the product with known performance issues, field personnel should know what problems should they expect and how to work around them

Why test for performance?

– “just load the system with users”• Set up a system• Blast it with gazillion of virtual users• Check V next to “Performance Tests”

– The problems of overloading• Chasing ghosts wastes dev resources• Reduce QA accountability

– The problems of under loading• We won’t find real performance issues• Reduce QA accountability

How not to test

– Usage Model design– Tests and environments creation– Load and Tune

Performance testing phases

• Load is generated by three factors:– User activity

• User actions on user interface• Typically: select operations, configuration changes

– External automated activity• Data flows into the system• Typically: insert operations

– Internal automated activity• Data manipulation• Typically: aggregation and data purge activity, internal

processes

Performance testing phasesDesigning a Usage Model

• Mapping the loading metrics of the system we want to test

– What causes the load on the system (Loading Parameters)?

– Settle for a finite number of metrics• Obtaining the loading metrics

– Objective sources• Customer logs• Customer databases• Customer support calls

– Subjective sources• Field personnel• Support personnel• Product marketing• Selected customers

Performance testing phasesDesigning a Usage Model

For each customer size:– Users

• How many concurrent users?• What is the user activity distribution?• How long does a typical session last?

– Hardware• What is the estimated hardware?• How is it configured?

– Software• How many Alerts are defined?• How many are set off each second?• How many SLAs?• Etc’

Performance testing phasesDesigning a Usage Model

– Hardware– Monitors– Data emulation– User emulation

Performance testing phasesTests and environments creation

– (Optional) Run a baseline run– Load and Tune

Performance testing phasesLoad and Tune

Increase Load

Yes

FinishedYes No

Run load

Does the system handle the load?

Tune Code

No

Did we reached expected thresholds?

– The three phases of QA-Dev interaction• Cooperation• Shock• Retaliation

– Back yourself up• Involve dev personnel in test design• Involve field personnel in test design• Know how each metric you use is relevant to real life

The Politics Of Performance Tests

– A Symantec I3 system consists of the following components:

• Customer’s monitored applications and servers

• Data collectors

• Data Loaders

• Database (PW)

• GUI

Use case: Symantec I3

Use case: Symantec I3 Behemoth System

Ares-z1 (8cpu/16Giga mem)

-I3FP-I4Oracle FP-Insight FP-I4J2EE FP-I4NET FP-I4Web FP

Ares-z3 (4cpu/8Giga mem)

-PWFP-IS Collector

qaperfaix1 (4cpu/8Giga mem)

-PW DB-IS Collector-I4Oracle Collector

qaperfaix3 (2cpu/4Giga mem)

-J2EE agent-IS Collector

qasys53 (VM)

-Web collector-.NET collector-IS collector

3 concurrent Virtual Users

Instance Weights

Indepth/ Oracle

Indepth/ SQL

Indepth/ Web

Indepth/ J2EE

Indepth/ SAP

Tuxedo

MQ

Netw ork

OS

OA

Insight/ J2EE

Indepth/ .Net

0100020003000400050006000700080009000

100001100012000130001400015000160001700018000190002000021000220002300024000250002600027000

1 13 25 37 49 61 73 85 97 109 121 133 145 157 169 181 193 205 217 229 241

Amount of Customers

Wei

gh

ted

Inst

ance

s

– Who are the large, medium and small customers?– Support file analysis

• PW team had size estimation for each monitored technology

• Support files hold hundreds of customers each with the amount of instances in each monitored technology

• When the analysis was through, we knew what is the average configuration for small, medium and large customers

Use case: Symantec I3

– Field interaction• Product experts feedback• Subjective data source

– Performance Test Plan• Formalize our goals, tests, tools and schedule

Use case: Symantec I3

– Test preparation• Synthetic Vs Real data

• User activity patterns

• Monitoring

– No thresholds• Unable to open performance bugs

• No measurable software standards

– Testing• Set up a system

• Transmit data

• Generate virtual users

• Measure user experience, internal processes, hardware resource usage

Use case: Symantec I3

– Summary• Our reports pressured dev to improve GUI response time

by 40%

• We issued a table of hardware recommendations based on the expected load

• Two most problematic application within the system identified and will be dealt with

• We changed the way load is measured in our organization

• Dev and support teams now contact us with performance problems, performance tests are now integral part of QA tests

Use case: Symantec I3

Agenda

• Why Test Performance?• How not to test• Performance Testing Phases

– Designing a Usage Model– Tests and environments creation– Load and tune

• The Politics of Performance Tests• Use Case: Symantec I3• Summary

Summary– Releasing software without performance

testing is like releasing new running shoes without running with them.

– Always design a usage model• What are the loading parameters?• Retrieve data from objective source• Verify data with subjective source

Thank YouOmri Lapidotolapidot@gmail.com