Software Testing in the Cloud

Leah Riungu-Kalliosaari

Objective and Background

Objective: To understand how organizations can successfully use the cloud for testing through empirical observations

Cloud computing: A model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g. networks, servers, storage, application and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction (NIST definition)

STaaS: A model for software testing whereby testing of an application is provided as a service across the internet.

Cloud-based testing is provided on-demand and billed on pay-per-use basis, so that the user pays only for the resources they have used

Work distribution during projects

MASTO/ESPA - 3 interview rounds (Fall 2009, Spring 2010 & Fall 2010)- 4 publications (2 conference + 1 conference + 1 journal)- Researcher exchange (04-06/2011, Limerick, Ireland)- About half the dissertation work completed

STX- About half of the dissertation work to be completed - 1 interview round (Fall 2011)- 1 Journal article; 1 book chapter in progress- Data analysis formulating results of the thesis- More publications and thesis finalization

What has been done so far?

Conditions that influence software testing as an online service

Research Issues for software testing in the Cloud

The effect of testing in the cloud (most recent)

Effect of Testing in the Cloud (1)

On actual testing work More efficient performance testing -

e.g. Running 40 Amazon instances for generating about 20 000 realistic user experiences

Quicker testing – e.g. Being able to test run a developed feature in say 10 different environments and getting (almost) immediate results

More realistic test resultsRiungu-Kalliosaari, L., Taipale, O., Smolander, K., ”Testing in the Cloud: Exploring the Practice”, Special Issue on Software Engineering for Cloud Computing, IEEE Software, March/April 2012

Effect of Testing in the Cloud (2)

On delivery and support of testing services Better availability of testing tools and

options – e.g. being able to carry out technical feasibility studies when choosing the tools to use.

Improved developer-tester communication – due to equal access to the system

Enhanced service delivery for vendors – quicker time to market and meeting of customer demands.Riungu-Kalliosaari, L., Taipale, O., Smolander, K., ”Testing in the Cloud: Exploring the

Practice”, Special Issue on Software Engineering for Cloud Computing, IEEE Software, March/April 2012

Roadmap towards testing in the Cloud Develop an understanding of cloud computing

Consider the benefits and risks Carry out pilot projects

Safe way to explore potential benefits Come up with elaborate strategies

Consider what to test; select a cloud provider Enhance team interaction and prepare for

complexities Developer/tester interaction while working with cloud-based

environments for development and testing Enhance co-operation between research and industry

Aim at addressing cloud-related issues faced by the software industryRiungu-Kalliosaari, L., Taipale, O., Smolander, K., ”Testing in the Cloud: Exploring the Practice”, Special Issue on Software Engineering for Cloud Computing, IEEE Software, March/April 2012

Work in Progress (1)

18 interviews from 11 organizations (Autumn 2011)

Data analysis – an iterative process Data from previous project also in use

Evaluation of quality attributes Development of cloud-based testing

strategy that can be used while considering adoption of cloud-based testing

WiP (2)… Quality Attributes

Cloud based testing improves testing of certain quality characteristics, for example, performance and scalability Justification: Cloud testing offers , for example,

computing resources in performance testing, human resources in crowd testing, resources for testing scalability. Further cloud testing reduces testing costs e.g. pay per use pricing model.

Cloud-based testing improves certain areas of testing, but it introduces also new issues e.g. security issues, and testing associated with huge data sets.

WiP (3)…Quality Attributes

Peformance

Cloud-based Testing

Scalability

MaintainabilityElasticity

improves

controls

supportsis part of

Assess The need

Consider/EvaluateThe line of business

Ch

oo

se/

Se

lect

Critical vs. non-critical applications

Size

Critical vs. non-critical applications

Size

Acquire/Realize/ implement

M

an

ag

e/R

e-

eva

l ua

te

What would you like to achieve?

Security risks/threats

Trade-offs

What would you like to achieve?

Security risks/threats

Trade-offs

The delivery approach

The cloud provider

SLAs, TOS, etc

The delivery approach

The cloud provider

SLAs, TOS, etc

The cloud service

Pay attention to SLAs, TOS, Security

The cloud service

Pay attention to SLAs, TOS, Security

Achieved vs. non-achieved needs, security, trade-offs

Maintainability

Change management e.g. skills development

Achieved vs. non-achieved needs, security, trade-offs

Maintainability

Change management e.g. skills development

WiP (4)…Cloud-based testing strategy

Pleriminary observations from most recent data (1) There seems to be a correlation between

quality and requirements management Improved requirements management leads to

improvement in quality Global software development

Managing globally distributed teams efficiently

Change management is unique within each organization Different types of changes To introduce change, identify the weakest link

Pleriminary observations from most recent data (2)

Cloud awareness is rising There is need for the benefits of cloud computing to

be clearly understood and appreciated before cloud computing can be taken into use 4 out of 11 organizations are using the cloud 1 has a customer that is using the cloud 1 has a customer that is considering to use

salesForce.com in future Very small companies are more willing to use the

cloud No upfront costs Getting up small functionality up and running quickly

In general, open source tools can be used if deemed necessary