1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group

1

TIME BOXED TESTING

BCS SIGIST

13th July 1998

Graham Thomas - OSI Group

2

Abstract

There is great pressure upon developers today to improve productivity and effectiveness. To achieve this there is a move away from the traditional structured methodologies towards more dynamic, iterative and RAD approaches.

This is being combined with Object and Component based techniques, and delivered with a new generation of IDE’s, to produce thin client, web based, voice and data products.

So how do we test it? Propose Time Boxed Testing Outline an approach Discuss the problems Give a flavour of the fun ?

3

Agenda

Development Lifecycle Testing Lifecycle Testing Helix Testing Stage by Stage Traceability Testing Process How to sell time boxed testing Lessons Learnt

4

The Project

Telephony based project 2 types of proprietary hardware Client Server running Windows NT Interfaces to legacy systems Rational Rose case tool (using UML) Objectory process Visual Studio Was considering Java & thin clients Incremental/Iterative development

5

Incompatible Models?

Sys Test

Classic V Model

Rqmts.

Analysis

Design

Build Unit Test

Int Test

UAT

Breadth

Depth

Iterative (RAD) Model

6

Difficult to Represent Test Stages

Breadth

Depth

unit

unit

unit

ComponentIntegration

ComponentIntegration

System

Acceptance

ComponentIntegration

7

Project Lifecycle

. . .

8

Testing Lifecycle

. . .

9

P

DC

E

A dual spiral life cycle ?

PlanDesignConstructExecute

RqmtsAnalysisDesignBuild

. . .R

AD

B

10

Testing Lifecycle

Unit Testing Component Integration Testing System Testing Acceptance Testing Year 2000 Millenium Compliance Testing Processes and Activities Tools & Traceability Resources Planning

11

Scope of Test Phases

ComponentIntegrationTesting

UnitTesting System

TestingUserAcceptanceTesting

ConicalModel

Iterative Release

12

Tuned Test Lifecycle

System Testing

- Non functional Requirements- Performance Analysis

ComponentIntegrationTesting

UnitTesting

Iterative Release

User Acceptance

Testing- Scenario Testing- Acceptance Criteria

Testing Activities Brought Forward

13

Unit Testing

Unit Testing

StaticAnalysis

DynamicAnalysis

UnitTesting

Iterative Time-Boxed Activity

Specific to delivered components

‘White Box’ testing techniques employed

External Interfaces tested

Use of tools for Static & Dynamic analysis Quality

Assurance

ExternalInterfaceTesting

UnitIntegration

Testing

Sub UnitTesting

14

Unit Testing

SubUnit

Unit

Unit

Unit

Package

Unit Integration Testing

SubUnit

SubUnit

Package

PackageUnit

Unit

Unit

Unit Testing

SubUnit

SubUnit

SubUnitSub

Unit

SubUnit

Sub Unit Testing

SubUnit

Stubs, Drivers, Harnesses

Previously tested & checked-in

15

Component Integration Testing

ComponentIntegration

Non FunctionalRequirements

FunctionalRequirements

Regression Test

ComponentIntegrationTesting

Iterative Time-Boxed Activity

Testing prioritised by Business

End to End process threads where possible

Contains elements of following test phases

Build regression test packs

ScenarioRegression

Test

PerformanceAnalysis

PackageIntegration

16

Package

Component Integration Testing

Package Package Integration must precede Component Integration

Component

Component

Component

Package

Package

Step 1 - Package Integration

17

Component Integration Testing

Component

Component

Component

Component

Component

Drop1

ComponentComponent

Drop2

Component

Component

Component

ComponentComponent

Component

Component

Component

. . .

Step 2 - Component Integration

18

Component Integration Testing

Automated Regression Test Pack

Component

Component

Component

ComponentComponent

Component

ComponentComponent

Component

Component

ComponentComponent

Component

Component

. . .

Component

ComponentComponentComponent

Component

Component

Component

Component

ComponentComponent

Component

Component

ComponentComponent

ComponentComponent

Drop by Drop

19

System Testing

FunctionalRequirements

Regression Test

Non FunctionalRequirements

PerformanceAnalysis

Design Verification

SystemTesting

One off activity Structured testing

techniques End to End process

threads Ties together the

development iterations Regression test

already constructed

20

Acceptance Testing

AcceptanceCriteriaTesting

ScenarioRegressionTesting

BusinessProcessTesting

UsabilityTesting

AcceptanceTesting

Data Take On & Conversion

Testing

One off activity Test that the computer

system integrates with business processes

Structured testing techniques

End to End process threads

Regression test already constructed

21

Y2000 - Millennium Compliance6 Date Scenarios

2 Business Tests

ThisCentury

AcrossCentury

Boundary

Whollynext

CenturyJan 1st Feb 29th Dec 31st

FinancialYear ?

FiscalPeriods ?

Separate test runs Carried out in

sequence Some tests can be

exercised post Y2000

22

Testing Lifecycle TraceabilityHigh Level Function

FunctionalRequirement

Use Case

Scenario

Test Case

Test Script

Non FR

DevelopmentRepository

Test Planning

UserRequirements

23

Testing Processes and Activities

Test Management

Test Reporting Monitor Measure Manage

ExitTest StageSign off

Test Planning Test Design Test Preparation Analysis of Change

Test Execution Test Automation Regression Test

QualityInterface

ProblemMgmt

Conf. Mgmt.Interface

Process

Entry

Activity

24

Processes

Essential Configuration Management

Impact Analysis using a Test Planning repository

Quality Management

Good practice Fault Reporting - on the Intranet - Live

Gathering metric information

25

Tools

Full set of testing tools is required

Static analysis

Dynamic Analysis

Interactive debugging

Test Automation/Capture Replay essential

Test Planning Repository

Performance monitoring (stress/load)

Fault Reporting system

26

Monitoring

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Fault

0

20

40

60

80

100

120

140

Cumulative Error Curve

27

Monitoring

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

0

5

10

15

20

25

30

35

40

45

50

Outstanding

28

Monitoring

1 2 3 4 5 6 7 8 9 10 11 12 13

0

20

40

60

80

100

120

140

Faults by Priority

A

B

C

29

Monitoring

W1 W2 W3 W40

5

10

15

20

25

30

Aged Defect Analysis by Priority

A

B

C

30

Test Resources

UnitTesting

ComponentIntegrationTesting

SystemTesting

UserAcceptanceTesting

TESTTEAM

Testers

BusinessUsersDevelopers

BusinessUsers

Developers

Secondment

31

Test Planning

Definition

Test Strategy

Testing Process

Developm

ent

Non Functional Testing

Test drop

Test drop

Test drop

Decision

Test Approach

Delivery

System Test

Acceptance Test

Y2000 Test

Time-boxed Testing

32

Selling the Approach

Target the following; Test Team

Systems Architect

Development team

Standards & Methods

Business Representatives

Project Sponsor

Via Animated Presentation Test Strategy document

33

So how easy was this?

34

So What went well?

Systems Architect liked it The presentation was well received

35

What was difficult?

Initial cool response from other members of the test team

Development Team were uneasy Dev. Why test anyway, our code is wonderful and our test

harnesses will do all the testing? Business Representatives were unhappy about;

Anything less than 100% testing Bringing their effort forward in the development lifecycle The difference between the ‘Use Case’ modeling they had

done, and the test planning that they would need to carry out

36

What was difficult? (2)

Standards & MethodsSaw nothing wrong with their approach, it was either;

• structured in which case their approach worked fine,

or• it was incremental, in which case their approach worked

fine.• They did not do iterative development, but if they did their

approach would still work.

This took several meetings to gain approval. Project Sponsor

Cancelled the project. (Time Boxed Testing was not cited as the reason !!!)

37

So what fun did we have ?

38

What did I learn from this?

Organisations change their development approach more quickly than their testing approach

There is a reluctance to do anything less than 100% testing

Testing is a key and integral part of any development lifecycle

The more you think about it, the more sense Time Boxed Testing makes

39

Contact Information

Graham Thomas

OSIthe management support company

e-mail : gthomas@osi.co.uk

Phone : 0171 323 3353

Copies of slides including animated gifs at

www.osi.co.uk