View
218
Download
2
Category
Preview:
Citation preview
performance testing – an insider
Mar 26, 2011
Manoj JainManager – Software Engineering, CA Tech.
− Understanding performance testing
− Case study
− Effective performance testing−Activities Identification−Best Practices
− Questions
agenda
2 Performance Testing - An Insider Copyright © 2010 CA. All rights reserved.April 18, 2023
understanding performance testing
performance testing
4 Performance Testing - An Insider Copyright © 2010 CA. All rights reserved.April 18, 2023
What
Why
How
a case study
Forward Inc. is a leading firm in providing Banking & Financial solutions to medium and large scale organizations
Forward Inc. is eager to conduct performance testing as: More number issues are related to poor response time. Customers are demanding very high response time
To resolve the issues at the earliest, the company leased a leading performance testing tool
Forward Inc. – A Case Study
6 Performance Testing - An Insider Copyright © 2010 CA. All rights reserved.April 18, 2023
Forward Inc. – Queries
7 Performance Testing - An Insider Copyright © 2010 CA. All rights reserved.April 18, 2023
What other business functions should be included in the performance test?
Which platform (HW/OS) should be tested first?
Which database should be used in performance test?
What should be the record volume in the database?
How do we monitor the resource usage (CPU/Memory) at the server(s)?
Forward Inc. – Issues Faced
8 Performance Testing - An Insider Copyright © 2010 CA. All rights reserved.April 18, 2023
—Performance test suite was aborted as−Test master machine ran out of space−Rollback segment was full due to large number of operations−Test tool was not able to validate licenses from License Server
—Large number of users terminated abnormally as−Users’ password expired during suite execution - 20%−A show-stopper occurred due to a code fix - 10%−An agent machine ran out of memory – 5%
Forward Inc. – Issues Faced (contd.)
9 Performance Testing - An Insider Copyright © 2010 CA. All rights reserved.April 18, 2023
The report interpretation was very difficult as checkpoints are listed in alphabetical order instead of their chronological order
Test data needs to be prepared again as backup data cannot be restored
The response time was very poor as during suite execution, some other activity was also being performed on the database server
Required HW/SW is not available for dial-up network testing
QA team is idle when development team is using the testing environment for fixing performance issues
effective methods of performance testing
performance testing phases & activities involved
11 Performance Testing - An Insider Copyright © 2010 CA. All rights reserved.April 18, 2023
Phase Tool required Activity
Project Kick-off No NA
Test Planning/Scheduling No NA
Test Bed/Test Data Preparation No NA
Virtual User Script Recording Yes • To record & validate VU scripts• To define and load the data pool
Performance Test Suite Preparation
Yes To prepare performance test suit with Agents, User Groups, Scenarios etc.
Test Suite Execution Yes To execute the test suite with pre-decided number of user load
Publishing Performance Test Results Report
No
best practices – project kick off
12 Performance Testing - An Insider Copyright © 2010 CA. All rights reserved.April 18, 2023
Deciding test environment i.e. HW/SW/DB
Deciding number of virtual users
Deciding Expectation Matrix
Escalation Matrix to handle any delays
Deciding team size and identifying team members from QA, Dev., DBA & Infrastructure team
Identification of business functions (Workload Analysis)
best practices – test planning
13 Performance Testing - An Insider Copyright © 2010 CA. All rights reserved.April 18, 2023
Arrange separate test environments for QA and Dev. team
Time required by individual teams to conduct their assigned activities i.e. QA – Scripting/Test data/Suite Exec./Test Report Dev – Identification & fixing of performance issue, testing of tuned
sources, tuned source deployment DBA – Analysis of database usage reports, highlighting queries, which
are having high execution time
Activities should be broken down to lowest possible level with dependencies
Time to backup and restore database with verification and analysis
best practices – VU scripting
14 Performance Testing - An Insider Copyright © 2010 CA. All rights reserved.April 18, 2023
Script naming conventionsConvention:Version_Module_ScriptSequence_ScriptExample: r11_Acc_01_GenRepWhere: r11 – Version of the application under test
Acc – Abbreviation of the module e.g. Accounting01 – The sequence number out of total virtual user scriptsGenRep – Abbreviation of the Script e.g. Generate Report
Checkpoint naming conventionConvention:Version_Module_ActionSequence_ActionExample: r11_Acc_GenRep_01_LoginWhere: r11 – Version of the application under test
Acc – Abbreviation of the module e.g. AccountingGenRep – Abbreviation of the action e.g. Generate Report01 – The sequence number out of total virtual user scripts
Notes:- The action sequence number is local within a script- Actions are numbers within a script as per the chronological order
No duplicate check point names
best practices – VU scripting (contd.)
15 Performance Testing - An Insider Copyright © 2010 CA. All rights reserved.April 18, 2023
15
CloseRepCloseRepGenGenLoginLoginLogoutLogoutRepModuleRepModuleSelectRepSelectRep
r11_Acc_GenRep_01_Loginr11_Acc_GenRep_02_RepModuler11_Acc_GenRep_03_SelectRepr11_Acc_GenRep_04_Genr11_Acc_GenRep_05_CloseRepr11_Acc_GenRep_06_Logoutr11_Finan_GenRep_01_Loginr11_Finan_GenRep_02_RepModuler11_Finan_GenRep_03_SelectRepr11_Finan_GenRep_04_Genr11_Finan_GenRep_05_CloseRepr11_Finan_GenRep_06_Logout
Without Conventions With Conventions
best practices – test data preparation
16 Performance Testing - An Insider Copyright © 2010 CA. All rights reserved.April 18, 2023
Backup of database schema should be performed at each milestone.
Test data prepared using SQL Scripts, tools should be validated.
Database schema should be analyzed at regular intervals.
Backup of database schema should NOT be stored on the same database server.
No resource consuming activity should be performed on the database server during test data creation.
best practices – test suite preparation
17 Performance Testing - An Insider Copyright © 2010 CA. All rights reserved.April 18, 2023
Test suite should be configured using a pre-decided think time value
Logical group names should be assigned to user groups
It is preferred to assign users as % instead of fixed nos.
Agent machines should be logical
While selecting an agent machine, it needs to be ensured that agent hardware and OS is well supported by test tool
It needs to be ensured that agent machine is neither over-allocated nor under-allocated
best practices – test suite execution
18 Performance Testing - An Insider Copyright © 2010 CA. All rights reserved.April 18, 2023
Ensure that all user ids used in the VU script are working fine and valid for large number of days
Ensure that server(s), test master and test agent machine(s) have enough free disk spaces
Paging file size of the agent machine should be increased to maximum.
Ensure that all monitors have been configured properly to monitor the servers’ resources usage
Response time & working of the application under test should be verified manually before executing the suite
During suite execution, response time of the application under test might be checked manually
During suite execution, no other activity should be performed on the machines being used in the test
best practices – test results report
19 Performance Testing - An Insider Copyright © 2010 CA. All rights reserved.April 18, 2023
Documenting executive summary
Test methodology followed
Test results with different percentiles
Test environment configuration
Summary of Workload Distribution
List of all scripts along with checkpoint names
Comparative display of test results in graphical form
thank you
manoj.jain@ca.com
Recommended