Computer Measurement Group, India 1Computer Measurement Group, India 1

www.cmgindia.org

CLOUD PERFORMANCE TESTING - KEY CONSIDERATIONS

Abhijeet Padwal, Persistent Systems

Computer Measurement Group, India 2

Contents

• Cloud• Case study– Performance Testing Requirements– Approach– Selection And Setup of Test Environment– Execution and Results– Challenges

• Conclusion

Computer Measurement Group, India 3Computer Measurement Group, India

Cloud

• A style of computing in which scalable and elastic IT-enabled capabilities are delivered as a service using internet technologies. [Gartner 2014]– Scalable and elastic– Accessibility over the internet– Service-Oriented

• Types– Private– Public– Hybrid

Computer Measurement Group, India 4Computer Measurement Group, India

Cloud (Contd.)

• Types of Services– Infrastructure as a Service (IaaS)– Platform as a Service (PaaS)– Software as a Service (SaaS)

• Service Providers– Amazon, Google, Microsoft Azure, Openstack

• Market Outlook– 17.7% growth is expected from 2011 through 2016

Computer Measurement Group, India 5Computer Measurement Group, India

Case Study• Customer –

– A leading software company delivering Retail Solutions – Solutions includes POS, CRM, SCM and ERP.

• Application under test (AUT)– An enterprise class retail solution to manage the front end and backend

operations within a retail store and – controlling the stores from the head office through a single application.

Computer Measurement Group, India 6Computer Measurement Group, India

Performance Testing Requirement

• Performance Benchmarking for 3000 stores

• 5 most critical business scenarios spread across all components

– Scenario 1 –Replication Replication of billing data from store to AS for 3000 stores.

– Scenario 2 – Billing counter Multi user (minimum 25 parallel counters) performing billing transactions which include the Bill, Sales

Return, Bill Cancellation, Lost Sales (in order of execution priority) with max line item not above 200 and minimum of 20 line items with Cash and Credit card as payment.

– Scenario 3 – AS Access the reports to be checked while data from Store (minimum 20+ stores) is being updated to AS.

– Scenario 4 – Operations Access stock management functions with 1000 + line items namely with 5/10 users

– Scenario 5 –Updates Download of patch for more than 100 stores simultaneously. Various patch sizes to be tested namely

50MB, 80MB, 100MB

Computer Measurement Group, India 7Computer Measurement Group, India

Scenario 1 - Replication Load Test Requirement- Replication of data from Store to Server for 3000 stores. Each store would have 100 billing counters and each counter generating bill with 200 line items.

3 Parts of the scenario –1. Collation of billing data

from all the counters and generate the xml message files.

2. Transfer the xml message files from store to server (replication client -> replication server).

3. Extract the xml files and store the extracted billing data on the head office database.

Computer Measurement Group, India 8Computer Measurement Group, India

Test Harness Setup

• To simulate this scenario a test harness was created which had 5 parts, – xml messages folders on injector machine – Vb based replication client (.exe) on injector machine – IIS and sql server based replication server – xml message folder on the head-office server and – Perfmon setup for monitoring the resource consumption on the AS as

well as load injectors.

• Batch• Response Time• Finding out the number of Load injectors

Computer Measurement Group, India 9Computer Measurement Group, India

Server Machine Configurations

Computer Measurement Group, India 10Computer Measurement Group, India

Setup Selection• Usage Pattern

• Comparison

*Other advantages of Azure – High amount of data and No cost for OS and S/W

Computer Measurement Group, India 11Computer Measurement Group, India

Azure Setup

Computer Measurement Group, India 12Computer Measurement Group, India

Initial Test Run Results on Azure

Computer Measurement Group, India 13Computer Measurement Group, India

Resource Utilization

Computer Measurement Group, India 14Computer Measurement Group, India

Moving Back to Local Setup

• Upgrade, • CPU power 4 –>8 core• and disk speed – 10k rpm -> 15 krpm

• Azure – • CPU upgrade possible however disk array has been shared so no changes

• Local Setup • Server with exact configuration available

Computer Measurement Group, India 15Computer Measurement Group, India

Results on Local Setup

Computer Measurement Group, India 16Computer Measurement Group, India

Challenges/ issues faced in Cloud during execution

• Managing multiple injectors and Switching between injectors to initiate the test

• Test data setup

• Monitoring

• Data transfer

Computer Measurement Group, India 17Computer Measurement Group, India

Conclusion

• Cloud provides an edge over the conventional on-premises test setups however…

There are few disadvantages or challenges with the cloud environment which could severely hamper the purpose.

• All pros and cons should be studied in details and that too in the context of the load testing requirement and define the holistic strategy to perform load testing on cloud.

Computer Measurement Group, India 18Computer Measurement Group, India

Q&A

Computer Measurement Group, India 19Computer Measurement Group, India

Thanks

Abhijeet Padwal

Persistent SystemsAbhijeet_padwal@persistent.co.in