BEgInSlIdE 1 Software Testing. bEgInSlIdE 2 Software Life Cycle Sommerville, 1992: D evelopment efforts are typically distributed as follows: Specifications

bEgInSlIdE

1

Software Testing

bEgInSlIdE

2

Software Life Cycle

Sommerville , 1992: Development efforts are typically distributed as follows:

Specifications / Design 30% - 40%

Implementation 15% - 30%

Testing 25% - 50%

bEgInSlIdE

3

Remarks by Bill Gates17th Annual ACM Conference on Object-Oriented Programming,

Seattle, Washington, November 8, 2002

“… When you look at a big commercial software company like

Microsoft, there's actually as much testing that goes in as

development. We have as many testers as we have developers.

Testers basically test all the time, and developers basically are

involved in the testing process about half the time…

… We've probably changed the industry we're in. We're not in

the software industry; we're in the testing industry, and writing

the software is the thing that keeps us busy doing all that

testing.”

bEgInSlIdE

4

Remarks by Bill Gates (cont’d)

“…The test cases are unbelievably expensive; in fact,

there's more lines of code in the test harness than

there is in the program itself. Often that's a ratio of

about three to one.”

“… Well, one of the interesting questions is, when you

change a program, … what portion of these test cases

do you need to run?“

bEgInSlIdE

5

The V-model of development

Requirementsspecification

Systemspecification

Systemdesign

Detaileddesign

Module andunit codeand tess

Sub-systemintegrationtest plan

Systemintegrationtest plan

Acceptancetest plan

ServiceAcceptance

testSystem

integration testSub-system

integration test

Implementation and unit testing

bEgInSlIdE

6

Some popular testing categories

Black box /white box

Static / dynamic

bEgInSlIdE

7

Specification-based testing Reflects true

intention of testing

Does not propagate errors from previous versions

Regression testing Does not need a

specification

Easy to implement

Finds subtle errors

Testing methodologies

bEgInSlIdE

8

How shall we check the I/O relation ? Manually (specification-based)

Table of expected results (specification-based)

Compare results to previous version (Regression testing)

Input OutputSystem under test

Black Box Testing(Behavioral testing)

bEgInSlIdE

9

Testing Input-Output relationships only

Pros

• This is what the product is about.

• Implementation independent.

Cons

• For complicated products it is hard to identify erroneous output.

• It is hard to estimate whether the product is error-free.

• Practically: Choosing input with high probability of error detection is very difficult (e.g. division of two numbers).

Black Box Testing(Behavioral testing)

Input Output

bEgInSlIdE

10

White Box Testing(Operational Testing)

Testing how input becomes output (including algorithms) Pros

• Easier to detect errors.

• Enables to find better tests (direct the tests)

• The only way to check coverage.

Cons

• Implementation weaknesses are not necessarily those of the product.

• Code is not always available

bEgInSlIdE

11

Dynamic testing (Run your program) Predefined tests

• Good for Regression Testing (comparing an old version against a new one)

• Testing the product under extreme conditions

Random tests

“real life” tests

Static testing (Inspect your code) Code analyzers (e.g., tools like lint and purify)

Inspection (code review)

Proofs (by tools, or by mathematical arguments)

Static and Dynamic Testing

bEgInSlIdE

12

Dynamic Testing

In combination with black-box testing

In white-box testing:

Preprocessor controlled code

• The only way for digging into the heart of the code

• Code usually outputs the status of some objects.

• Requires modification whenever the code is modified + compilation.

Specification-based monitoring

Other methods …

bEgInSlIdE

13

Special Testing Methods.

Stress Testing

A product that will work under heavy load (e.g, on-line banking system) should be tested under increasing load - much heavier than expected.

bEgInSlIdE

14

Code analysis Unreachable code

Objects declared and never used

Parameters type/number mismatch

Variable used before initialization

Variable is assigned to twice, without using the first value

Function results not used

Possible array bound violations

Static Testing

bEgInSlIdE

15

Code inspection

Self - The default choice.

• Subtle errors and micro-flaws may be overlooked.

• Wrong conceptions propagate to review…

Code review by others - Very efficient !

Static Testing

bEgInSlIdE

16

One more quote…

Dijkstra:

“Testing can only prove the existence of bugs, not their absence…”

bEgInSlIdE

17

In general – it is undecidable, i.e. can’t be done.

In most cases it is possible, but with manual assistance – the same way we would prove a math theorem.

In some cases properties of software can be proved automatically.

Chances for errors increase with length of text• Write short code (e.g, divide into more functions).

• Provide short proofs for correctness (even if they are informal).

… So why not try to prove correctness?

bEgInSlIdE

18

Estimate how clean is your software

Error Implantation (For measuring the effectiveness of testing)

Introduce errors.

See how many of them are detected.

This gives us an “educated guess” about testing quality.

bEgInSlIdE

19

Estimate how much of the software’s behavior is covered

Coverage is a mean to estimate how rigorous is the testing effort

We can use coverage information in order to guide the process of test generation (some times even automatically)

bEgInSlIdE

20

Statement CoverageExample 1

int a, b, sum;

int list1[10] = {00, 11, 22, 33, 44, 55, 66, 77, 88, 99};

int list2[10] = {99, 88, 77, 66, 55, 44, 33, 22, 11, 00};

cin >> a >> b;

if (a >= 0 && a <= 9)

sum = list1[a];

if (b >= 0 && b <= 9)

sum = sum + list2[b];

cout << sum << "\n";

bEgInSlIdE

21


if (a >= 0 && a <= 9)

sum = list1[a];

if (b >= 0 && b <= 9)


Statement coverage may be achieved by __ test case(s):

bEgInSlIdE

22


if (a >= 0 && a <= 9)

sum = list1[a];

if (b >= 0 && b <= 9)


But statement coverage may not cater for all conditions

such as when a and b are beyond the array size.

bEgInSlIdE

23

Branch CoverageSame Example 1

if (a >= 0 && a <= 9)

sum = list1[a];

if (b >= 0 && b <= 9)


Branch coverage may be achieved by __ test cases:

bEgInSlIdE

24

branch coverageExample

switch (x){

case 1: x = 1; break;

case 2: switch (y){



otherwise: x = 1; }

otherwise: 4;}

branch coverage may be achieved by __ test cases

bEgInSlIdE

26

if (a >= 0 && a <= 9)

sum = list1[a];

else sum = 0;

if (b >= 0 && b <= 9)


else sum = 0;

Path coverage may be achieved by __ test cases:

Path CoverageSame Example 2

bEgInSlIdE

27

Path coverage

subsumes Branch coverage

subsumes Statement coverage

Subsumption Relationships

But can wealways demandpath coverage?

bEgInSlIdE

28

Branch CoverageExample 3

if (a >= 0 && a <= 9)

sum = list1[a];

else sum = 0;

if (b >= 0 && b <= 9)


else sum = 0;

...

if (z >= 0 && z <= 9)


else sum = 0;

Branch coverage may be achieved by __ test cases

bEgInSlIdE

29


if (a >= 0 && a <= 9)

sum = list1[a];

else sum = 0;

if (b >= 0 && b <= 9)


else sum = 0;

...

if (z >= 0 && z <= 9)


else sum = 0;

Path coverage may be achieved by __ test cases

bEgInSlIdE

30


sum = 0;

while (a >= 0 && a <= 9) {

sum = list1[a];

a = a + 1;

};

if (b >= 0 && b <= 9)

sum = list2[b];

else sum = 0;

Statement coverage may be achieved by __ test cases:

bEgInSlIdE

31

Branch CoverageSame Example 4

sum = 0;

while (a >= 0 && a <= 9) {

sum = list1[a];

a = a + 1;

};

if (b >= 0 && b <= 9)

sum = list2[b];

else sum = 0;

Branch coverage may be achieved by __ test cases:

bEgInSlIdE

32

Loop coverage

Skip the loop entirely

Only 1 pass through the loop

2 passes through the loop

n–1, n and n+1 passes through the loop, where n is the maximum number of allowable passes

We only wentthrough theloop once.

bEgInSlIdE

33

Loop CoverageSame Example 4sum = 0;

while (a >= 0 && a <= 9) {

sum = list1[a];

a = a + 1;

};

if (b >= 0 && b <= 9)

sum = list2[b];

else sum = 0;

Loop coverage may be achieved by __ test cases:

bEgInSlIdE

34


sum = 0;

while (a >= 0 && a <= 9) {

sum = list1[a];

a = a + 1;

};

if (b >= 0 && b <= 9)

sum = list2[b];

else sum = 0;


bEgInSlIdE

35

Path coverage

subsumes Branch coverage

subsumes Statement coverage

Subsumption Relationships

But can wealways demandpath coverage?

Path coverage

subsumes Loop coverage

bEgInSlIdE

36

Path CoverageExample 5sum = 0;

while (a >= 0 && a <= 9) {

sum = list1[a];

a = a + 1; };

while (b >= 0 && b <= 9) {

sum = list2[a];

b = b + 1; };

...

while (z >= 0 && z <= 9) {

sum = list26[z];

z = z + 1; };


bEgInSlIdE

37

Path Coverage is not EverythingExample 1

z = x + y;

Path coverage may be achieved by 1 test case

x = 8, y = 0

Cannot detect z = x - y;

x = 2, y = 2

Cannot detect z = x * y;

x = 8, y = 9

Cannot detect

z = 8 + y;

Cannot detect z = x + 9;

We need 2 test cases:

x = 8, y = 9

x = 29, y = 18.

bEgInSlIdE

38

Path Coverage is not EverythingExample 2

int a[10];

if (b > 0)

a[b] = 1;

else …

Same path with b = 5 And b = 12 behave

differently

bEgInSlIdE

39

Condition coverageExample

if (b1 ||(b2 && b3))

a = 1;

else …

Sub-expression coverage may be achieved by __ test cases

Every sub-expression in a predicate should be evaluated to TRUE and FALSE

bEgInSlIdE

40

Multiple condition coverageExample

if (b1 ||(b2 && b3))

a = 1;

else …

Multiple condition coverage may be achieved by __ test cases

Every Boolean combination of sub-expressions in a predicate should be evaluated to TRUE and FALSE

bEgInSlIdE

41

Condition/Branch coverage (MC / DC)

if (!b1 || b2)

a = 1;

else …

MC/DC coverage may be achieved by __ test cases

Union of Condition coverage and Branch coverage

This is an industry standard

bEgInSlIdE

42

Specification-based testing

We will see an example of a system for specification-based testing of real-time applications.

The testing system is called “Logic Assurance”

It monitors the specification, and can also intervene in the execution of the program.

bEgInSlIdE

43

Monitoring

Logic Assurance

SystemUnderDevelopment

Environment Report State

control

Optional

Report stateEvent Reporting

=

control

Overview

bEgInSlIdE

45

A typical specification language: The LA Language (LAL)

Derived from Temporal Logic and C, LAL enables specification of:

Event order.

Relative frequency ("fairness").

Timing demands.

Logical and mathematical operators.

More...

bEgInSlIdE

46

Examples (1/4):

1. Using event names, time directives and messages:

OPEN_DOOR follows CLOSE_DOOR after not more than 10 sec ?: message("Open door is late");

bEgInSlIdE

47

...Examples (2/4)

2. Logical operators and functions:

when time>20: time([CLOSE_DOOR])> time([OPEN_DOOR])+10?: message("CLOSE_DOOR is early");

bEgInSlIdE

48

...Examples (3/4)

3. User-defined functions are used to enhance the language and enable external control:

if JOB_DONE(10) then HEAT(3,5) < 30?: REDUCE_HEAT(5);

bEgInSlIdE

49

...Examples (4/4)

4. Locators are used to scan the log and retrieve event index:

[2nd SEND s.t. Time>=10, packet=5] > 0 ?: REDUCE_HEAT(5);

bEgInSlIdE

50

2. Report events:

From inside the SUD by using the LA Report Facility.

From outside the SUD by using black-box techniques (e.g. OS events)

From the environment (Sensors, etc.)

bEgInSlIdE

51

The LA Report Facility:

The following directive should be inserted where the event occurs in the program:

LA_report_event (

int identifier,

float time-stamp,

user additional data...

)

bEgInSlIdE

52

LA will:

1. Keep an event log.

2. Analyze the rules in real time* (during execution) using the LA Analyzer.

* Off-line analysis is also possible

bEgInSlIdE

53

3. When a rule is violated, do one of the following:

Report the exact place and time the rule was violated.

Invoke a user-defined function.

bEgInSlIdE

54

Demo

A basic communication protocol ("Stop & Wait"):

2 2 23

time-out time-out

Rules:1. Resend packet after time-out…2. Do not resend packet that was acknowledged……

send

1

Ack 1

bEgInSlIdE

55

# event time message index arguments

-- ------ ------- ---------- ------- ------------

1: send 10 A 1 7, 10, 21..

2: Tout 15 A 1 8, 9 ,10..

3: send 16 B 2 20,30,21.. :

Rule 1, event 3: failed to resend packet

‘A’ sent at 10, Ack expected at 13..

Tout for A activated at 15, must send A again..

‘B’ sent at 16, Ack expected at 19..

.

Incoming events: messages:

user screen:

bEgInSlIdE

56

Advantages of real time monitoring:

Tests can be planned, maintained, expanded and applied throughout the development process.

Problems can be detected sooner.

The product is ‘tied’ to the specification.

bEgInSlIdE

57

A generic tool and methodology.

By receiving input from different sources, it enables testing of:

Multiprocessor systems

Dynamic environment systems.


bEgInSlIdE

58

Enables temporal tests.

Enables smarter tests by using information from inside the program.


Documents

BEgInSlIdE 1 Software Testing. bEgInSlIdE 2 Software Life Cycle Sommerville, 1992: D evelopment efforts are typically distributed as follows: Specifications