100% Automated Software Testing

Zohaib Iqbal

Assistant Professor, FAST-NU, Islamabad, Pakistan

Research Fellow, Software V&V Lab, SnT, University of Luxembourg

Lead Scientist, QUEST Lab

June 3, 2014

Software Quality Engineering and Testing

(QUEST) Lab

• Quest Lab, Est. 2013, www.questlab.pk

• 18 scientists (Research scientists, PhD candidates, Research Engineers)

• International academic collaborations:

– Interdisciplinary Centre for Security, Reliability and Trust (SnT) Luxembourg

– Simula Research Laboratory, Norway

• International industry partners working in telecom, satellite, and financial sectors

• Industry-relevant research on software quality engineering: testing, model-driven development

• Publishing high quality research

• ICT RDF Grant holder of 14 Million

– The only Software Testing project being funded by ICTRDF

2

QUEST Lab

• Industry-driven research targeting real problems

– Developing novel strategies and tools for quality engineering

and automated testing that are scalable in industrial contexts

• Focus areas

– Automated Software Testing

– Model-driven Software Engineering

– Software Product-line engineering

– Software Engineering for Mobile Applications

– Software Refactoring

3

Automated Software Testing

• For complete automated testing, all of the

following tasks should be automated:

– Test Path/Sequence Generation

– Test Data Generation

– Test Oracle (Verdict) Generation

– Test Execution

• Most testing tools only support automated test execution:

– xUnit framework tools: JUnit, GoogleTest, nUnit, etc

– Record & replay tools: Selenium, Test Complete, QTP, Watir, etc

• Large part of local industry struggling to achieve

automated test execution !!

4

new Calculator(..).add(…)

“Windows”, 2, 2

4

Automated Software Testing – Industry Practice

• What about the tasks other than test execution?

• The test data and test sequences are provided manually

– Typically, random sets of values

• Very basic test oracle

– For example, if the application doesn’t crash, it should be fine

• Problems?

– Human intensive effort

– No systematic approach

– Not scalable, not repeatable, not maintainable

– Limited confidence in testing, How to evaluate effectiveness of

testing?

• Typical solution:

– Testing is driven by availability and capability of the resource

5

What about automation of other steps?

• Useful Test data, sequences, and oracle are

application specific

– Cannot be generalized

– Need to capture specification of the system

under test in a machine analyzable form

– Domain/application specific methodologies and

strategies for testing are required

• One widely accepted solution: Model-based

testing

– Our expertise at QUEST lab & SVV Lab

6

Test Path/Sequence

Test Data

Test Oracle

Model-Based Testing

• Model-based Testing (MBT) uses models of the system to

completely automate the testing process

– The models typically describe some aspects of system under

test

– Widely advocated for complete test automation, i.e.,

generation of test data, sequence, and oracle

• MBT allows automated testing solutions for complex

industrial software systems

– Early

– Systematic

– Scalable

7

Model-Driven Software Engineering

• Model: An abstract description of software artifact, created for a purpose.

• Abstract: Details are omitted. Partial representation. Much smaller and simpler than the artifact being modeled.

• Analyzable: Leads to task automation

Requirements

models

Architecture

models

Behavioural

models

Runtime

models

8

break;

case Unlocked:

switch(e)

{

case Coin:

Thankyou();

break;

case Pass:

s = Locked;

Lock();

break;

}

break;

}

}

Model => Abstraction

9

enum State {Locked, Unlocked};

enum Event {Pass, Coin};

void Unlock();

void Lock();

void Thankyou();

void Alarm();

void Transition(Event e)

{

static State s = Locked;

switch(s)

{

case Locked:

switch(e)

{

case Coin:

s = Unlocked;

Unlock();

break;

case Pass:

Alarm();

break;

}

Software Modeling & The Object Management

Group (OMG)

• A non-profit consortium of leading industrial and academic

organizations

– IBM, Microsoft, Thales, Dell, RedHat, MIT, CMU, Carleton,

and many others

• Focuses on defining modeling standards (programs,

systems and business processes).

• FAST-NU – has the distinction of being the only

Pakistani member organization

– QUEST Lab actively participating in various RFPs, including,

UTP, MARTE, OCL

10

OMG’s Unified Modeling Language (UML)

Requirements Capture

System Structure

System Behavior

11

Use Case Diagrams

+ structured textual description

Sequence Diagrams (+OCL)

Activity Diagrams (+OCL)

State Machines (+OCL)

Class Diagrams (+OCL)

Communication Diagrams (+OCL)

Test Case Generation and Execution

12

Test cases

SW Model

SW Code

Test cases

Compare Oracle

Expected

Results

Results

What modeling

strategy?

What test data & sequence

generation strategy?

What oracle strategy?

Specifications or designs

in UML

Application specific/

control and data flow

analysis

Application

domain specific

Vending Machine Control Software -

State Machine

off

power-on/

light-on; m:=0

power-off/

light-off

not

empty empty

inc/m:=1

dec[m=1]/m:=0

inc/m:=m+1

dec[m>1]/m:=m-1

money

busy idle

coffee

on

coffee[m>0]/start

after(5)/stop

coffee[m>0]/start;dec

13

State coverage

14

off

power-on/

light-on; m:=0

power-off/

light-off

not

empty empty

inc/m:=1

dec[m=1]/m:=0

inc/m:=m+1

dec[m>1]/m:=m-1

money

busy idle

coffee

on

coffee[m>0]/start

after(5)/stop

coffee[m>0]/start;dec

Industrial Applications of MBT @ Quest Lab

• MBT is not something new!

• Well-accepted by international industry

– IBM, Microsoft, Cisco, Schlumberger, etc

• Next, some of the industrial applications of MBT that

Quest members have worked with

15

Industrial Applications of MBT –

Schlumberger & Tomra

• Soft-real time systems: deadlines in the order of hundreds of

milliseconds

– Jitter of few milliseconds acceptable

• Goal: Automation of test cases and oracle generation,

environment simulation

16 Marine Seismic Acquisition System Bottle Recycling Machine

Industrial Applications of MBT –

Schlumberger & Tomra: Solution

• Independent

– Black-box

• Behavior driven by

environment

– Environment model

• One model for

– Environment simulator

– Test cases and oracles

• Notation: Class Diagrams,

State machines, OCL

17

Environment

Simulator

Test cases

Environment Models

Test oracle

Domain Model - Example

Behavioral Model - Example

19

Industrial Applications of MBT –

Cisco Systems

• High End Video Conference Systems

– small personal systems to large office

systems

– Geographically distributed

• Audio and video quality is a concerns

• For these system robustness is every

important

• A typical environment of VCS includes

communication network and other

communicating networking systems

• Goal: Robustness testing

20

Video Conferencing System

Industrial Applications of MBT –

Cisco Systems: Solution

• Functional behavior captured using class diagrams and

state machines

– Average state machine size: 5 states, 11 transitions

• Robustness behavior captured using a UML profile

– Modeling various properties of faults (e.g., severity) to assist

in defining robustness test strategies

– State machines flattened and traverse for test path

generation

• QUEST Lab currently working in collaboration with Simula

Lab to enhance the testing strategies and tool

– QUEST Lab funded by ICT R&D Fund, Pakistan

21

Industrial Applications of MBT –

Luxembourg Tax Department

• Working in collaboration with Software Verification and

Validation Lab, SnT, Luxembourg

• System under test: Tax calculation system

• Focuses on modeling of tax laws and testing the tax

calculation system with a wide range of cases

– Modeling in an analyzable form (Class diagrams & OCL)

• Goal

– Generating test data based on the modeled laws to test the

tax calculation system

– Identification of conflicting laws

22

23

Industrial Application of MBT –

Data Acquisition System

• Being done in collaboration with SnT

• System under test: Data acquisition system

– System gets data from satellites

– Identifies transmission errors and extract useful information

• Model the structure and content of the input data

• Goal: Automated generation of input data (including

erroneous data) to test the acquisition system

24

Research Model @ QUEST

• Unlike other stereotyped academic research, at QUEST lab

research is driven by practical problems faced by

industry

• Strong industrial collaborations to develop and evaluate

candidate solutions

25

Bridging Academia/Industry Gap

• QUEST Lab is working with multiple international

organizations

– Would like to collaborate with local industry as well

• What we need from industry

– Practical problem requiring innovative solution

• e.g., testing software product-lines

– Access to case studies

– In-kind contribution from the industry

• Pakistan Software Testing Board

• ICET 2014 @ FAST-NU

– Industry Experience rack

– Tutorials

26

Thanks