Whitebox Testing

CS 420 - Spring 2007

A Testing Life Cycle

RequirementSpecs

Design

Coding

Testing

Fault Resolution

FaultIsolation

FaultClassification

Error

Fault

Fault

Fault

Error

Error

incident

Fix

Terminology

• Error– Represents mistakes made by people

• Fault– Is result of error. May be categorized as

• Fault of Commission – we enter something into representation that is incorrect

• Fault of Omission – Designer can make error of omission, the resulting fault is that something is missing that should have been present in the representation

Cont…

• Failure– Occurs when fault executes.

• Incident– Behavior of fault. An incident is the

symptom(s) associated with a failure that alerts user to the occurrence of a failure

• Test case– Associated with program behavior. It carries

set of input and list of expected output

Cont…

• Verification– Process of determining whether output of one

phase of development conforms to its previous phase.

• Validation– Process of determining whether a fully

developed system conforms to its SRS document

Verification versus Validation

• Verification is concerned with phase containment of errors

• Validation is concerned about the final product to be error free

Relationship – program behaviors

Program Behaviors

Specified(expected)Behavior

Programmed(observed)BehaviorFault

OfOmission

FaultOfCommission

Correct portion

Classification of Test

• There are two levels of classification– One distinguishes at granularity level

• Unit level• System level• Integration level

– Other classification (mostly for unit level) is based on methodologies

• Black box (Functional) Testing• White box (Structural) Testing

Relationship – Testing wrt Behavior

Program Behaviors

Specified(expected)Behavior

Programmed(observed)Behavior

Test Cases(Verified behavior)

8 7

5 6

14 3

2

Test methodologies

• Functional (Black box) inspects specified behavior

• Structural (White box) inspects programmed behavior

Functional Test cases

Specified Programmed

TestCases

Structural Test cases

Specified Programmed

TestCases

When to use what

• Few set of guidelines available• A logical approach could be

– Prepare functional test cases as part of specification. However they could be used only after unit and/or system is available.

– Preparation of Structural test cases could be part of implementation/code phase.

– Unit, Integration and System testing are performed in order.

Unit testing – essence

• Applicable to modular design– Unit testing inspects individual modules

• Locate error in smaller region– In an integrated system, it may not be easier

to determine which module has caused fault– Reduces debugging efforts

Test cases and Test suites

• Test case is a triplet [I, S, O] where– I is input data– S is state of system at which data will be input– O is the expected output

• Test suite is set of all test cases• Test cases are not randomly selected.

Instead even they need to be designed.

Need for designing test cases

• Almost every non-trivial system has an extremely large input data domain thereby making exhaustive testing impractical

• If randomly selected then test case may loose significance since it may expose an already detected error by some other test case

Design of test cases

• Number of test cases do not determine the effectiveness

• To detect error in following codeif(x>y) max = x; else max = x;

• {(x=3, y=2); (x=2, y=3)} will suffice• {(x=3, y=2); (x=4, y=3); (x=5, y = 1)} • Each test case should detect different errors

White-Box Testing

• Statement coverage

• Branch coverage

• Path coverage

• Condition coverage

• Data flow-based testing

Statement Coverage

• Statement coverage methodology:– design test cases so that every statement in a

program is executed at least once.

• The principal idea: – unless a statement is executed, we have no

way of knowing if an error exists in that statement

Statement coverage criterion

• Observing that a statement behaves properly for one input value:– no guarantee that it will behave correctly for

all input values.

Example

• int f1(int x, int y){

1. while (x != y){

2. if (x>y) then

3. x=x-y;

4. else y=y-x;

5. }

6. return x; }

Euclid's GCD computation algorithm

• By choosing the test set{(x=3,y=3),(x=4,y=3), (x=3,y=4)}– all statements are executed at least once.

Branch Coverage

• Test cases are designed such that:– different branch conditions is given true and

false values in turn.

• Branch testing guarantees statement coverage:– a stronger testing compared to the statement

coverage-based testing.

Example

• Test cases for branch coverage can be:{(x=3,y=3), (x=4,y=3), (x=3,y=4)}

Condition Coverage

• Test cases are designed such that:– each component of a composite conditional

expression given both true and false values.

• Example– Consider the conditional expression

((c1.and.c2).or.c3):– Each of c1, c2, and c3 are exercised at

least once i.e. given true and false values.

Branch testing

• Branch testing is the simplest condition testing strategy

• compound conditions appearing in different branch statements are given true and false values. (note: the entire condition is given true and false values, not ALL possible sub expressions)

Branch testing

• Condition testing– stronger testing than branch testing:

• Branch testing – stronger than statement coverage testing.

Condition coverage

• Consider a Boolean expression having n components: – for condition coverage we require 2n test

cases.

• practical only if n (the number of component conditions) is small.

Path Coverage

• Design test cases such that:– all linearly independent paths in the program

are executed at least once.

• Defined in terms of– control flow graph (CFG) of a program.

Control flow graph (CFG)

• A control flow graph (CFG) describes: – the sequence in which different instructions of

a program get executed. – the way control flows through the program.

How to draw Control flow graph?

• Number all the statements of a program. • Numbered statements:

– represent nodes of the control flow graph.

• An edge from one node to another node exists: – if execution of the statement representing the

first node can result in transfer of control to the other node.

Example

int f1(int x,int y){

1. while (x != y){

2. if (x>y) then

3. x=x-y;

4. else y=y-x;

5. }

6. return x; }

Example Control Flow Graph

1

2

3 4

5

6

Path

• A path through a program:– A node and edge sequence from the starting

node to a terminal node of the control flow graph.

– There may be several terminal nodes for program.

Independent path

• Any path through the program:– introducing at least one new node or one new

edge that is not included in any other independent paths.

• It may be straight forward to identify linearly independent paths of simple programs. However For complicated programs it is not so easy to determine the number of independent paths.

McCabe's cyclomatic metric

• An upper bound: – for the number of linearly independent paths

of a program

• Provides a practical way of determining: – the maximum number of linearly independent

paths in a program.

McCabe's cyclomatic metric

• Given a control flow graph G,cyclomatic complexity V(G): – V(G)= E-N+2

• N is the number of nodes in G• E is the number of edges in G

Example

• Cyclomatic complexity = 7 – 6 + 2 = 3.

Cyclomatic complexity

• The cyclomatic complexity of a program provides:– a lower bound on the number of test cases to

be designed to get coverage of all linearly independent paths.

– only gives an indication of the minimum number of test cases required.

Path Testing - Test Cases

• Draw control flow graph. (Loops can cause an explosion in the number of independent paths, so use looping values of 0 and 1 to test loops).

• Determine V(G).• Determine the set of linearly independent

paths.• Prepare test cases:

– to force execution along each path

Example Control Flow Graph

1

2

3 4

5

6

Derivation of Test Cases

• Number of independent paths: 4– 1, 6 test case (x=1, y=1)– 1, 2, 3, 5, 1, 6 test case(x=1, y=2)– 1, 2, 4, 5, 1, 6 test case(x=2, y=1)

An interesting application of cyclomatic complexity

• Relationship exists between:– McCabe's metric– the number of errors existing in the code, – the time required to find and correct the

errors.

Cyclomatic complexity

• Cyclomatic complexity of a program: – also indicates the psychological complexity of

a program. – difficulty level of understanding the program.

Cyclomatic complexity

• From maintenance perspective, – limit cyclomatic complexity

• of modules to some reasonable value.

– Some software development organizations: • restrict cyclomatic complexity of functions to some

max number.

Data-flow-based Testing

• Basic idea: test the connections between variable definitions (“write”) and variable uses (“read”)

• Starting point: variation of the control flow graph– Each node represents a single statement, not a chain

of statements

• Set DEF(n) contains variables that are defined at node n (i.e., they are written)

• Set USE(n): variables that are read

Example

Assume y is already initialized

1 s:= 0;2 x:= 0;3 while (x<y) {4 x:=x+3;5 y:=y+2;6 if (x+y<10)7 s:=s+x+y; else8 s:=s+x-y;

DEF(1) := {s}, USE(1) := DEF(2) := {x}, USE(2) :=DEF(3) := , USE(3) := {x,y}DEF(4) := {x}, USE(4) := {x}DEF(5) := {y}, USE(5) := {y}DEF(6) := , USE(6) := {x,y}DEF(7) := {s}, USE(7) := {s,x,y}

DEF(8) := {s}, USE(8) := {s,x,y}DEF(9) := , USE(9) := DEF(10) := , USE(10) :=

1 2 3

4

5

6

7 8

9

10

Reaching Definitions

A definition of variable x at node n1 reaches node n2 if and only if there is a path between n1 and n2 that does not contain a definition of x

DEF(1) := {s}, USE(1) := DEF(2) := {x}, USE(2) :=DEF(3) := , USE(3) := {x,y}DEF(4) := {x}, USE(4) := {x}DEF(5) := {y}, USE(5) := {y}DEF(6) := , USE(6) := {x,y}DEF(7) := {s}, USE(7) := {s,x,y}DEF(8) := {s}, USE(8) := {s,x,y}

1 2 3

4

5

6

7 8

9

10

Reaches nodes 2,3,4,5,6,7,8, but not 9 and 10.

Def-use Pairs

• A def-use pair (DU) for variable x is a pair of nodes (n1,n2) such that – x is in DEF(n1)– The definition of x at n1 reaches n2– x is in USE(n2)

• In other words, the value that is assigned to x at n1 is used at n2– Since the definition reaches n2, the value is not killed

along some path n1...n2.

Examples of Def-Use Pairs

DEF(1) := {s}, USE(1) := DEF(2) := {x}, USE(2) :=DEF(3) := , USE(3) := {x,y}DEF(4) := {x}, USE(4) := {x}DEF(5) := {y}, USE(5) := {y}DEF(6) := , USE(6) := {x,y}DEF(7) := {s}, USE(7) := {s,x,y}DEF(8) := {s}, USE(8) := {s,x,y}

1 2 3

4

5

6

7 8

9

10

Reaches nodes 2, 3, 4, 5, 6, 7, 8, but not 9,10

For this definition, two DU pairs:1-7, 1-8

Data-flow-based Testing

• Identify all DU pairs and construct test cases that cover these pairs– Several variations with different “relative strength”

• All-DU-paths: For each DU pair (n1,n2) for x, exercise all possible paths n1, n2 that are clear of a definition of x

• All-uses: for each DU pair (n1,n2) for x, exercise at least one path n1 n2 that is clear of definitions of x

Data-flow-based Testing

• All-definitions: for each definition, cover at least one DU pair for that definition– i.e., if x is defined at n1, execute at least one path

n1..n2 such that x is in USE(n2) and the path is clear of definitions of x

• Clearly, all-definitions is subsumed by all-uses which is subsumed by all-DU-paths

• Motivation: see the effects of using the values produced by computations – Focuses on the data, while control-flow-based

testing focuses on the control