Concepts on Software Performance Testing

testrun.co.nz@testrunnzConcepts on

Software Performance

Testing

Chris SAUNDERS 2014

testrun.co.nz@testrunnz


Presentation's Purpose


Presentation's Purpose

• To talk about software performance testing

• To explore concepts, ideas and things to consider

• Not talking about tools


What is Software Performance?


What is Software Performance?Software performance is the manner in which

software functions, operates, or behaves; based on a given set of factors.

Software performance testing is to determine how a system performs in terms of responsiveness and stability with a defined set of factors.

What factors?

Each person takes two sticky notes and writes their answer down, and places it on the wall

Note this question may be asked again later


General Measures


General Measures

Measures Description

1. Known measures Measure where the system meets a given predefined performance criteria

For example: The CPU & Memory resources will not exceed 80% for more than 5 second duration.


General Measures



2. Comparable measures Measure between the performance of two systems to determine which has better performance

For example: The new calculation engine will have a 20% increase in processing time and data volume

throughput.


General Measures



2. Comparable measures Measure between the performance of two systems to determine which has better performance

3. Unknown measures Measure where the system has a possible performance issue in a given area

(One does not exactly no where or how it will fail)

For example: Determine the system performance on the system due to teamed network card failure.


The Business Case


The Business Case

“What is the business case for performance testing?”

“How does it add business value?”

Need to define “realistic performance related goals”


Performance Goals5 Examples Types


Performance Goals:Water pipe analogy

Consider software performance testing like Christchurch’s water pipe system


Performance Goals:Response Time

Measuring the duration of time between two points

For example: The duration for a user waiting on a file to upload

A BDuration

0:00:00.29 or 290ms


Performance Goals:Concurrency

Measuring the effect on the system with concurrent processes

For example: The effect of the different number of users trying to upload files at almost the same time


Performance Goals:Throughput

Measuring the effect on the system with throughput of information (volume)

For example: The effect of different sized files being uploaded

20 l/s98 l/s


Performance Goals:Interdependence

Measuring how the system responds to different factors

For example: Consider the relationship between the following1. Response time 2. Concurrency 3. Throughput

A BDuration


Performance Goals:Interdependence

Compare these two scenarios:1. 100 users are each uploading 20MB files, start

uploading files within a 5 minutes duration.2. 10 users are each uploading 200MB files, start

uploading files within a 10 seconds duration.

Which scenario has better performance?


Performance Goals:MTBF & MTTR

Mean Time Between Failures & Mean Time To Repair

Determine the performance of when the system fails and how long does it take to repair?

ConsIder the relationship of the system to other systems or internal components


Performance Goals:MTBF & MTTR

Source : http://www.greenm3.com/gdcblog/2012/7/31/netflix-sets-chaos-monkey-free-for-all-to-use-next-comes-mor.html

The Chaos Monkey is a software tool that was

developed by Netflix engineers to test

the resiliency and recoverability of their Amazon Web Services

Open Source https://github.com/Netflix/SimianArmy

Netflix engineers develop automatic recovery mechanisms


Performance Goals:Summary

1. Response Time2. Concurrency3. Throughput4. Interdependence5. MTBF & MTTR6. …


Some Performance Testing Approaches


Performance Testing Approach:Load Testing

Measure how the system behaves under a specific expected load.

Load characteristics: Define number of concurrent users, with a determined throughput for a given duration

Could be looking for bottlenecks


Performance Testing Approach:Stress Testing

Measure how the system behaves with loads beyond normal operational capacity.

Measure modes of failure.

Determine the robustness of the system: How, when & where is it going to fail


Performance Testing Approach:Spike Testing

Measure how the system behaves with the sudden increase in load

Determine system response; performance profile or system failure


Performance Testing Approach:Endurance (Soak) Testing

Measure how the system operates under a continuous expected load

Determine performance degradation


Environmental changes to determine system performance

System controlled failure

System configuration changes


Performance Testing Approach:Summary

1. Load Testing2. Stress Testing3. Spike Testing4. Endurance Testing5. Environmental changes to

determine system performance6. …


What’s Software Performance?


What is Software Performance Testing?

Software performance testing is to determine how a system performs in terms of responsiveness and stability with a defined set of factors.

What factors?(Each person can change their two sticky notes on the wall)

Discuss software performance testing…

testrun.co.nz@testrunnzConcepts on

Software Performance

Testing

Chris SAUNDERS 2014

Documents

Concepts on Software Performance Testing