Performance testing – an insider Mar 26, 2011 Manoj Jain Manager – Software Engineering, CA Tech. when title IS NOT a question there is NO ‘WE CAN’ in

performance testing – an insider

Mar 26, 2011

Manoj JainManager – Software Engineering, CA Tech.

− Understanding performance testing

− Case study

− Effective performance testing−Activities Identification−Best Practices

− Questions

agenda

2 Performance Testing - An Insider Copyright © 2010 CA. All rights reserved.April 18, 2023

understanding performance testing

performance testing


What

Why

How

a case study

Forward Inc. is a leading firm in providing Banking & Financial solutions to medium and large scale organizations

Forward Inc. is eager to conduct performance testing as: More number issues are related to poor response time. Customers are demanding very high response time

To resolve the issues at the earliest, the company leased a leading performance testing tool

Forward Inc. – A Case Study


Forward Inc. – Queries


What other business functions should be included in the performance test?

Which platform (HW/OS) should be tested first?

Which database should be used in performance test?

What should be the record volume in the database?

How do we monitor the resource usage (CPU/Memory) at the server(s)?

Forward Inc. – Issues Faced


—Performance test suite was aborted as−Test master machine ran out of space−Rollback segment was full due to large number of operations−Test tool was not able to validate licenses from License Server

—Large number of users terminated abnormally as−Users’ password expired during suite execution - 20%−A show-stopper occurred due to a code fix - 10%−An agent machine ran out of memory – 5%

Forward Inc. – Issues Faced (contd.)


The report interpretation was very difficult as checkpoints are listed in alphabetical order instead of their chronological order

Test data needs to be prepared again as backup data cannot be restored

The response time was very poor as during suite execution, some other activity was also being performed on the database server

Required HW/SW is not available for dial-up network testing

QA team is idle when development team is using the testing environment for fixing performance issues

effective methods of performance testing

performance testing phases & activities involved


Phase Tool required Activity

Project Kick-off No NA

Test Planning/Scheduling No NA

Test Bed/Test Data Preparation No NA

Virtual User Script Recording Yes • To record & validate VU scripts• To define and load the data pool

Performance Test Suite Preparation

Yes To prepare performance test suit with Agents, User Groups, Scenarios etc.

Test Suite Execution Yes To execute the test suite with pre-decided number of user load

Publishing Performance Test Results Report

No

best practices – project kick off


Deciding test environment i.e. HW/SW/DB

Deciding number of virtual users

Deciding Expectation Matrix

Escalation Matrix to handle any delays

Deciding team size and identifying team members from QA, Dev., DBA & Infrastructure team

Identification of business functions (Workload Analysis)

best practices – test planning


Arrange separate test environments for QA and Dev. team

Time required by individual teams to conduct their assigned activities i.e. QA – Scripting/Test data/Suite Exec./Test Report Dev – Identification & fixing of performance issue, testing of tuned

sources, tuned source deployment DBA – Analysis of database usage reports, highlighting queries, which

are having high execution time

Activities should be broken down to lowest possible level with dependencies

Time to backup and restore database with verification and analysis

best practices – VU scripting


Script naming conventionsConvention:Version_Module_ScriptSequence_ScriptExample: r11_Acc_01_GenRepWhere: r11 – Version of the application under test

Acc – Abbreviation of the module e.g. Accounting01 – The sequence number out of total virtual user scriptsGenRep – Abbreviation of the Script e.g. Generate Report

Checkpoint naming conventionConvention:Version_Module_ActionSequence_ActionExample: r11_Acc_GenRep_01_LoginWhere: r11 – Version of the application under test

Acc – Abbreviation of the module e.g. AccountingGenRep – Abbreviation of the action e.g. Generate Report01 – The sequence number out of total virtual user scripts

Notes:- The action sequence number is local within a script- Actions are numbers within a script as per the chronological order

No duplicate check point names

best practices – VU scripting (contd.)


15

CloseRepCloseRepGenGenLoginLoginLogoutLogoutRepModuleRepModuleSelectRepSelectRep

r11_Acc_GenRep_01_Loginr11_Acc_GenRep_02_RepModuler11_Acc_GenRep_03_SelectRepr11_Acc_GenRep_04_Genr11_Acc_GenRep_05_CloseRepr11_Acc_GenRep_06_Logoutr11_Finan_GenRep_01_Loginr11_Finan_GenRep_02_RepModuler11_Finan_GenRep_03_SelectRepr11_Finan_GenRep_04_Genr11_Finan_GenRep_05_CloseRepr11_Finan_GenRep_06_Logout

Without Conventions With Conventions

best practices – test data preparation


Backup of database schema should be performed at each milestone.

Test data prepared using SQL Scripts, tools should be validated.

Database schema should be analyzed at regular intervals.

Backup of database schema should NOT be stored on the same database server.

No resource consuming activity should be performed on the database server during test data creation.

best practices – test suite preparation


Test suite should be configured using a pre-decided think time value

Logical group names should be assigned to user groups

It is preferred to assign users as % instead of fixed nos.

Agent machines should be logical

While selecting an agent machine, it needs to be ensured that agent hardware and OS is well supported by test tool

It needs to be ensured that agent machine is neither over-allocated nor under-allocated

best practices – test suite execution


Ensure that all user ids used in the VU script are working fine and valid for large number of days

Ensure that server(s), test master and test agent machine(s) have enough free disk spaces

Paging file size of the agent machine should be increased to maximum.

Ensure that all monitors have been configured properly to monitor the servers’ resources usage

Response time & working of the application under test should be verified manually before executing the suite

During suite execution, response time of the application under test might be checked manually

During suite execution, no other activity should be performed on the machines being used in the test

best practices – test results report


Documenting executive summary

Test methodology followed

Test results with different percentiles

Test environment configuration

Summary of Workload Distribution

List of all scripts along with checkpoint names

Comparative display of test results in graphical form

thank you

[email protected]

Documents

Performance testing – an insider Mar 26, 2011 Manoj Jain Manager – Software Engineering, CA Tech. when title IS NOT a question there is NO ‘WE CAN’ in