System Test Methods TESTTEME The Test Challenge Bottom Up Testing Strategy Integration Test System Test Types of Testing Unit Test = Code-based Testing

System Test Methods

TEST TEME

The Test Challenge

Bottom Up Testing Strategy

Integration Test

System Test

Types of Testing

Unit Test = Code-based Testing

Integration Test = Data-based Testing

System Test = Requirements-based Testing

Testing Interdependencies

Software Testing Strategies

Hierarchical Software Systems

BIG-BANG Strategy

TOP-DOWN Strategy

BOTTOM-UP Strategy

INSIDE-OUT Strategy

OUTSIDE-IN Strategy

BRANCHWISE Strategy

Comparison of the Test Strategies

Program Test Methods

Control Flow Testing (White-Box)

Data Flow Testing (Grey-Box)

Functional Testing (Black-Box)

Testing Heuristics

Roll of Testing in the Life Cycle

1

2

3

6

7

5

4

8

9

10

13

14

15

18

19

17

16

20

21

22

11 23

12 24

• Budget-, Time- and Personnel Constraints

• Great Diversity of SW-Environments

• Applications are becoming increasingly large and complex

• SW-Developers are neither trained nor motivated to test

• Testers are willing but incapable

• Lack of a testing culture

The Testing Challenge

TEST TEME-1

„How do you eat an elephant ... bit by bit“African Proverb

• Class Test– Test of each individual Class as a standalone Unit

• Integration Test– Test of several related classes and components

• System Test– Test of subsystems and the entire system

• Acceptance Test– Test by the User in a live environment

Bottom Up Testing Strategy

TEST TEME-2

„Integration testing exercises interfaces among units within the specified scope to demonstrate that the units are collectively interoperable“

Binder, 1999

• Classes• Class Hierarchies• Packages of related Classes = Components• Subsystems• Client/Server = Frontend/Backend • Entire Application

Integration Test

TEST TEME-3

• The Whole is more than the Sum of the Parts– „Individual verification of components cannot garantee

a correctly functioning system“ - Binder, 1999

• Demonstrating the required Functionality– „If you do not have written objectives for your product,

i.e a specification, or if your objectives are unmeasurable, then you cannot perform system testing“ - Myer, 1976

• Determining when to release a product ?!– Probability of remaining Errors..

System Test

TEST TEME-4

ClassesunderTest

Class Flattener

Test Driver

Tes

t S

tubs

Tes

t M

eth

ods

Integration Testing

Specified Arguments

Message Generator

Input Messages

Componentsunder Test

Return Messages

Message Validator

Unit Testing

ExpectedReturn Values

Replay Capture

Web Pages

Web Client

Web Server

Data

System Testing

DataGenerator

DataValidator

Retrieval Updates

Types of Testing

TEST TEME-5

A Test is always a test against Something

TEST TEME-6

Concept

System

Test Cases Test ResultsTest Cycle

newSystem

Test Cases Test ResultsTest Cycle

currentSystem

Verification = Comparison

In order to verify a system, the requirements must be independent of the code. The task is to test the system against the requirements by extracting test cases from the requirements specification. Verification implies demonstrating that the system fulfills it‘s requirements.

Once a system exists, every new version can be tested against the previous one. In this case the test cases can be extracted from the existing system and enhanced by test cases taken from the change requests. The new version is verified against the old one plus the new requirements.

Initial Testing Regession Testing

Code based Testing

TEST TEME-7

CodeTest inputs Test outcomes

Specification Expected outcomes

Interfaces/Databases

Predicate Paths

SpecifiedPaths

Actual Paths

compare Paths

Corresponds to a Unit Test (White-Box)

Unit Test = Code-based Testing

TEST TEME-8

Block

Block

Block

Block

Block

Block

IF THEN ELSE

DO WHILECASEBlock

IF THEN

41 5

7 8

6

9

10

2

3

= Testpoint

WHITE-BOX-Method

(Modultest)

Test Object:

Data based Testing

TEST TEME-9



compare Data

Representative Values

Interfaces/Files/Tables

Specified Results


Corresponds to an Integration Test (Grey-Box)

Client

Server

Data-server

Interface

Interface

Inte

rfac

e S

imul

atio

n

Inte

rfac

e V

alid

atio

n

GREY-BOX Method

(Program Test)

Input Data

Output Data---------

Integration Test = Data-based Testing

TEST TEME-10

Specification based Testing

TEST TEME-11



compare

Files/Tables/Panels

Specified Results

ExpectedValues


Corresponds to a System Test (Black-Box)

Outputs

SystemTest = Requirements-based Testing

TEST TEME-12

Test Object

Web Pages

Masks

Files

Databases

Parameters(Arguments)

Messages

Web Pages

Masks

Files

Databases

Reports(Results)

Messages

SYSTEM INPUTS SYSTEM OUTPUTS

Application Function

BLACK-BOX Method

(System Test)

Integration Testing Strategies

TEST TEME-13

BIG-BANG

TOP-DOWN

BOTTOM-UP

INSIDE-OUT

OUTSIDE-IN

BRANCH-WISE

Layered Software Systems

TEST TEME-14

Framework

GUI-Components

ErrorhandlingModule

PrintModule

XMLOutputModule

XMLInput

Module

SQLAcess

Module

OrderArt-3

Module

OrderArt-4

Module

OrderArt-1

Module

OrderArt-2

Module

Order Entry Processing System

Sample of a three level Program

ServiceLevel

LogicLevel

PresentationLevel

BIG-BANG Strategy

TEST TEME-15

GUIComponents

ErrorhandlingModule

OrderArt-3

Module

OrderArt-4

Module

OrderArt-1

Module

OrderArt-2

Module

PrintModule

XMLOutputModule

XMLInput

Module SQLAccessModule

TOP-DOWN Strategy

TEST TEME-16

Framework

GUIComponents

Stub StubStub

Stub StubStub Stub

BOTTOM-UP Strategy

TEST TEME-17

Framework

GUIComponents

ErrorhandlingModule

PrintModule

XMLOutputModule

XMLInput

Module

SQLAccessModule

Driver Driver DriverDriver Driver

Test-driven Development

INSIDE-OUT Strategy

TEST TEME-18

Driver

StubStub

OrderArt-3

Module

OrderArt-4

Module

OrderArt-1

Module

OrderArt-2

Module

OUTSIDE-IN Strategy

TEST TEME-19

Framework

GUIComponents

ErrorHandlingModule

PrintModule

XMLOutputModule

XMLInput

Module

SQLAccessModule

Driver DriverDriver Driver

Stub Stub Stub Stub

BRANCHWISE Strategy

TEST TEME-20

Framework

GUI-Components

StubPrint

ModuleStub

XMLInput

Module

SQLAccessModule

Stub StubOrderArt-1

ModuleStub

(Thread Testing)

One Transaction at a timeIs tested through to end

Straight thru testing

Comparison of Test Strategies

TEST TEME-21

Point of View TOP-DOWN BOTTOM-UP INSIDE-OUT OUTSIDE-IN BRANCH-WISE

Division of Labor poor good poor good poor

Controllability poor good poor medium medium

Test Aides required Stubs Drivers Stub&Drivers Stub&Drivers Stubs

Test Case Definition difficult easy medium easy medium

Test Visibility poor good medium good poor

Duration of Test medium long long short short

According to G. Myers: Software Reliability Principles and Practices

Principle SoftwareTest Approaches

TEST TEME-22

Control Flow Testing

Statement Coverage Branch Coverage Path Coverage

Data Flow Testing

Repreäsentative value coverage Boundary value coverage

Equivalence Class Coverage Data Relation Coverage Cause & Effect Analysis

State Coverage

Functional Testing

Result coverage = all desired results Rule coverage = all relevant rules Input oriented = all possible inputs

Control-Flow Testing (White-Box)

TEST TEME-23

Entry

Exit

(1) (2) (3) (5) (6)

(7) (8)

(9)

(10)

(4)

(11)

(12)

( ) = Control- branch

Paths =

1, 2, 3, 4, 6, 111, 2, 3, 5, 6, 111, 2, 7, 8, 10, 111, 2, 7, 9, 10, 111, 12,

Data-Flow Testing (Grey-box)

TEST TEME-24

ProgramInputs/Arguments Outputs/Results

Predicates

Representative ValuesBoundary ValuesEquivalence Classes

Discrete ValuesValue DomainsFunctional ResultsRelations

True/FalseCause/Effect

Functional Testing (Black-Box)

TEST TEME-25

Software SystemSystem SpecificationArgumentsRulesResults

InputsPathsOutputs

Testing Heuristics

TEST TEME-26

The specification of expected results is an essential part of the test.

Programmers are not in a position to test their own programs.

The results of each test case must be verified.

All exception conditions should be tested and confirmed.

It is not enough to confirm that a program does what it should do, but also to confirm that it does not do what it should not do.

Test cases must be repeatable.

Never assume there are no errors, software is always erroneoius.

Errors tend to cluster. If an error is found, other errors are likely to be found around it.

A good test case is a case with a high probability of discovering an error. The goal of testing is to find errors.

A test with no defined end criteria is no test.

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

Documents

System Test Methods TESTTEME The Test Challenge Bottom Up Testing Strategy Integration Test System Test Types of Testing Unit Test = Code-based Testing