Software testing tools and the selection process

1

Software testing toolsand the selection process

Geoff Quentin

QBIT

TestExpo 17th March 2005

2

Contents

• Introduction

• Specific points about processes

• Specific types of tools

• Implementation

• The way forward

3

Testing - why do it?

• Testing is to add confidence

• Testing is to show compliance

• Testing is to identify where to reduce risk

• Identify defects and then have them fixed

• Face problems if not done

4

FUNCTIONALSPECIFICATION

The simple V Model

CODE

UNITTEST

SYSTEMTEST

ACCEPTANCETEST

TECHNICALSPECIFICATION

USERREQUIREMENTS

UNITTEST

MATERIAL

SYS TESTMATERIAL

UAT TESTMATERIAL

PROGRAMSPECIFICATION

INT TESTMATERIAL

INTEGRATIONTEST

TestExpo © QBIT 2002

5

The project plan showing earlytest planning and creation

Specification ____

Design (user) ____

Design (tech) ______

Design (prog) _____

Code ____

Plan unit test _____QC

Unit test ____QC

Plan integration test _______QC

Integration test _____QC

Plan acceptance _________QC

Acceptance test ____QC

Installation _QC

6

Or for more detail use ISO 12207

• Lots more activities

• Specification paired with design

• System treated separately from software

7

The fit with 12207

Softwaredetaileddesign

Softwarecoding

SoftwareComponent

testing

Softwareintegration

Softwarequalification

testing

Systemintegration

Systemqualification

testing

Systemrequirements

analysis

Systemarchitectural

design

Softwarerequirements

analysis

Softwarearchitectural

design

Acceptance& completion

Specifybusiness

requirements

Software developmentteam (includes software

test team)

Softwarequalification

test team

Systemqualification

test team

The users (includes useracceptance test team)

The acquirerThe supplier

Softwareconfiguration

team

Systemdesignteam

Softwaredesignteam

TestExpo © QBIT 2002

8

Systemrequirements

analysis

Systemarchitectural

design

Acceptance& completion

Specifybusiness

requirements

Useracceptance test

team

Test item transmittalreport

The acceptance tester’s view and ISO 12207

9

Systemqualification

testing

Systemrequirements

analysis

Systemarchitectural

design

Specifybusiness

requirements

Test item transmittalreport

Systemqualification

test team

The system tester’s view and ISO 12207

10

Three documents for you

• This presentationwww.qbit.co.uk/testexpo

• The Generic Test Processwww.qbit.co.uk/gtp

• The Consolidated Test Processwww.qbit.co.uk/ctp

11

Why automate?

• So what is the objective?

• Return on investment

• Achieve a difficult goal

• Improve an existing process

• Implement a new process

• Scale up the process for bigger projects

• Ensure compliance

12

Process focus

• The basic process should be good

• The process upon which the tool is basedshould be good

• Conformity

• Repeatability

• Now seeing tools based on good processes

13

The test process

• Record (set up reporting)

• Create strategy

• Plan tests

• Analyse

• Design

• Schedule

• Execute

14

The testprocessmodel

LOW LEVELTEST PLAN

TESTOBJECTIVES

LIST

TEST EXECUTION

TESTITEM

TESTSUMMARY

REPORT

HIGH LEVELTEST PLAN

LOW LEVELTEST PLAN

LOW LEVELTEST PLAN

LOW LEVELTEST PLAN

TESTSCRIPT

TESTINCIDENTREPORT

TESTSTRATEGY

PROJECTDOC

SPEC

TEST ITEMTRANSMITTAL

REPORT

TESTLOG

TEST ITEMTRANSMITTAL

REPORT

TESTSCHEDULE

4. TEST ANALYSIS

5. TEST DESIGN

6. TESTSCHEDULING

7. TEST EXECUTION

2. TEST STRATEGY

3. TEST PLAN(various)

1. TEST STATUS REPORTING

Copyright © QBIT 2002

15

So which level to apply theprocess?

• Acceptance tests

• System tests

• Integration tests

• Unit tests

So who will use the chosen tool(s)?

16

Roles and responsibilities

• Manager

• Test team leader

• Tool user (the test analyst)

• Tools specialist (to implement)

• Process specialist (to improve the process)

17

Resources required

• Tools specialist

• Trainer

• Sponsor

18

Time table

• Critical to introduce tools early

• Time to familiarise

• Time to tune

• Time to achieve proficiency

• Now to use in earnest

19

So what might be bought?

• Tools to support existing processes

• Tools to implement a new process

• Tools to speed up existing processes

• Tools to enhance, make possible, process

• Tools to make economical a process

20

The tools on offer

• Management of testing

• Capture of tests

• Playback of tests

• Simulation, modelling and prototyping

• Static analysis

• Dynamic analysis

• General purpose tools

21

Tools to support thereview process

• Verification– checking the test object

• Validation– checking context

• Certification

• Defect recording and tracking

• Metrics

22

Management of testing

• Traceability

• Identification of tests

• Storage and retrieval of tests

• Reporting progress of tests

• Repeatability of tests

• Reusability of tests

• Long term support for the testing effort

For the test manager

and team leader

23

System design phase testing

• Capture of tests

• Selection and replay of tests

• Prototyping

• Simulation

• Mathematical modelling

• Change and configuration management

For the team leader

and test analystsystem test and

UAT

24

Software design phase

• Performance prediction

• Performance measurement

• Change and configuration management

For thesystem tester

and the integration tester

25

Programming

• Static analysis

• Dynamic analysis

• Unit test execution

• Change and configuration management

For the programmer

26

Research for tools

• Need to identify where processes need tools

• Need to gather metrics

• Evaluate the return on investment

• Tools to fit the processes

• Process based tools to fit the problem

27

Deployment of the chosen tool(s)

• Small project

• Limit the scope for failure

• Representative

• Support

• Training

28

Support once in use

• Supplier

• Internal

• User groups

• Independent consultants

• Independent trainers

29

Process improvement

• The tool must add value– improve efficiency

– improve effectiveness

– improve coverage

– reduce costs

– reduce duration

– simplify the work

– remove/reduce the chance of errors

30

So what is on offer at thisQBIT tools fair?

• Tools and consultancy

31

Visit• Cresta• Empirix• Vizuri• Compuware• Embarcadero• Princeton Softech• Facilita• Testhouse• Transition• Focus Frame• Propel Systems

• T-Plan

• BCS/iseb

• Sim/SQS

• Contemporary

• QBIT

• Serpico

• ITR

• Original Software

• Books and journals– Artech House

– Better Software

– Professional Tester

32

Summary

• Identify the objective(s)

• Check the processes

• Check the functionality

• Check the overhead of deployment

33

This presentation www.qbit.co.uk/testexpoThe Generic Test Process www.qbit.co.uk/gtpThe Consolidated Test Process www.qbit.co.uk/ctp

Thank you

34