Something to amuse you…

CS 2212- UWO 1

http://xkcd.com/518

http://www.youtube.com/watch?v=k0xgjUhEG3U 70 minutes

Find the problem (i.e. Is line/statement coverage enough?):

CS 2212- UWO 2

include <stdio>main () { int x, pos, even;

printf("Enter an integer:"); scanf("%d", &x); pos = 1; even = 1; if (x > 0) {pos = 1;} if (x%2 == 0) {even = 1;} printf (“Number: %d, Positive: %d, Even: %d\n", x, pos, even); return 0;}

• This code prints out 1 if the number is even and 1 if the number is positive and 0 for odd and 0 for negative. • I tested it with a minimal test case (x=6) to get 100% line coverage (see image), and I got the correct result but still the program is wrong. WHY???

CS 2212- UWO 3

Structural Testing

Structural testing measures

• Flowcharts

• Structural testing and flowcharts

• Minimal test suites

CS 2212- UWO 4

Program Flowcharts

Example of code:

• Intended to set pos = 1 iff x is positive, even = 1 iff x is evenDraw a graph of the statements and decisions in the code:

• Statements = boxes• Decisions = diamonds

This graph called a “flowchart”

main () { int x, pos, even;

printf("Enter an integer:"); scanf("%d", &x); pos = 0; even = 0; if (x > 0) {pos = 1;} if (x%2 == 0) {even = 1;} printf ("%d, %d, %d\n", x, pos, even); return 0;}

CS 2212- UWO 5

Flowchart Example

Printf()

Scanf()

X>0 Pos=1

X%2=0 Even = 1

Printf()

End

T

T

Pos=0

Even=0

F

F

CS 2212- UWO 6

Flowchart Example

A flowchart is really a labeled directed graph

• Boxes and diamonds = nodes

• Arrows = edges

Edges indicate parts of “paths” which may be followed through code

Labels on edges going out of diamonds indicate what happens when decision in diamond is true/false

CS 2212- UWO 7

Flowcharts: Example 2 (Code)

i = 0;while (s[i] != 0) {

if (s[i] >= ‘a’) && (s[i] <= ‘z’)) {s[i]=s[i] – 32;

}i++;

}

CS 2212- UWO 8

Flowcharts: Example 2 (Flowchart)i=0

i++

End

s[i]=s[i]-32

s[i]!=0?

s[i]>=‘a’&&s[i]<=‘z’?

T

T

F

F

CS 2212- UWO 9

Structural Testing and Flowcharts

What does this have to do with testing?If test suite executes all statements, then test suite visits every box and diamond in the flowchart

• We say the test suite covers every node• We can also say the test suite covers every statement

This does not mean that every arrow (edge) in the flowchart has been followed Example: consider the first example flowchart

• We run it on one test case: x=2• All nodes visited• However, neither F branch has ever been taken

If every arrow in the flowchart has been taken by some case in the test suite:

• We say the test suite covers every edge• We can also say the test suite covers every decision

CS 2212- UWO 10

Edge vs. Node Coverage

Edge cover is more thorough than node coverage

• Catches more problems

Example: consider first example flowchart

• Change statements pos =0; even = 0 in the first column to pos = 1; even = 1

• Now, program always sets pos and even to 1

• But we wanted pos to indicate whether x is positive, even to indicate whether x is even

• If we just test with x = 2

CS 2212- UWO 11

Printf()

Scanf()

x>0 Pos=1

x%2=0 Even = 1

Printf()

End

T

T

Pos=1

Even=1

F

F

Would 100% Statement/Node Coverage Find the Mistake with test case x=2?

Coding mistake made by a novice programmer (initialized incorrectly)

CS 2212- UWO 12

• If we test with both x = 2 and x = -1:—We cover every edge—We also force a failure to happen, program

reports wrong results for x = - 1

Thus, edge coverage can reveal failures where node coverage cannot.

CS 2212- UWO 13

Coverage Terminology

If a test suite executes 100% of statements in program, we say it achieves

• 100% node coverage or 100% statement coverage

If a test suite follows 90% of edges in flowchart, we say it achieves 90% edge coverage (etc)

“Line Coverage”: what gcov measures

• Any line containing code is measured

—Considered covered if any code on line is executed

• Not exactly the same thing as statement coverage

—Can have more than one statement on a line

• However, 100% statement coverage implies 100% line coverage

Edge coverage also called decision coverage

• Tests each decision in an if, while, etc. both ways

CS 2212- UWO 14

Strength of Coverage Measures

100% node coverage implies 100% line coverage

• However, 100% line coverage does not imply 100% node coverage

100% edge coverage implies 100% node coverage

• If followed every edge, must have visited every node• However, as we saw, 100% node coverage does not

necessarily imply 100% edge coverageThus we say:

• Node (statement) coverage is strong than line coverage• Edge (decision) coverage is stronger than node coverage

We can draw a graph:• Arrow from A to B: B is a stronger coverage measure than

ALine

Node(Statement)

Edge (Decision)

CS 2212- UWO 15

Question

Draw a flowchart for the following c code and find the minimum number of test cases to achieve:

• 100% line coverage

• 100% node coverage

• 100% edge coverage

int main (){ int a, b; a = 6; printf("\n\nEnter a value for a: "); scanf ("%d",&a); printf("\n\nEnter a value for b: "); scanf ("%d", &b); if (( a<10) || (a > 15)) if (b ==9) printf ("good\n"); else printf("evening\n"); else printf("madam"); printf("sir\n"); return 0;}

CS 2212- UWO 16

Your Answer

CS 2212- UWO 17

Minimal Test Suites

By throwing more and more test cases into a test suite, we may achieve higher coverage

However, longer test suites take longer to run

Hence, we often want to find some minimal test suite

• Important to define what minimal means i.e. minimal test cases? Minimal number of input used? Minimum time to run?

Example: first flowchart example:

• Test suite: x=2, x=1, x=-2 covers all edges

• However, it has 3 test cases

• Test suite: x=2, x=-1 covers al edges and has only 2 test cases

• Cannot have a minimal test suite which covers all edges and has only 1 test case

—Have to execute each decision both ways

Hence, x=2, x=-1 is a minimal test suite for 100% edge coverage

• Minimal in the sense of needing the fewest test cases

CS 2212- UWO 18

Coverage and Test Suites: Another Example

Consider again the case-conversion code flowchart (Example 2)

Find “minimal” test suite which achieves:

• 100 % node coverage

• 100% edge coverage

Decide what minimal means in this context!

CS 2212- UWO 19

Case Conversion Flowchart: Coverage

Clearly we can achieve 100% node coverage with one-char test case

• Say, for example: “a”This test case also follows both arrows from the top diamond

• This is because:—First time through loop (i==0), T branch taken—Next time condition is evaluated (i==1), s[i] is the null (end

of string) character—Hence, F branch is also taken

However, F branch of other diamond not takenTo cover all edges, we need test suite like

• (A) s= “a”, s=“X”; or • (B) s = “aX”

CS 2212- UWO 20

Case Conversion Flowchart: Minimality

Which of (A) and (B) is “minimal”• Depends on how we define minimality• (A) has 2 test cases, but each has just 1 character• (B) has just 1 test case, but it has 2 characters

We should always define precisely what we mean by minimality for a given test suite

• Here, makes more sense to accept (B) as minimal• Don’t’ have to run program as many times, incur overhead of starting up

process againHence, for this problem, we should say something like:

• Test suite X will be considered smaller than test suite Y if either:—X has few test cases than Y; or—X has the same number of test cases as Y, but the total number of

characters in X is smaller• Saves us from having to accept s = “aaaaaaXXXXXX” as minimal

CS 2212- UWO 21

Stronger Coverage Measures

So far we have been writing compound decisions

• (e.g. (c >=‘a’) && (c <=‘z’)) inside a single diamond

Decision: the whole thing in parentheses after the “if”, while”, etc

• Consider the code:if ((c>=‘a’) && (c<=‘z’)) {..}

• (c >=‘a’) && (c<=‘z’) is the decision

Conditions: the individual terms of the condition, connected by logical operators

• (c>=‘a’) && (c<=‘z’) are the two conditions

If we separate each condition with the decision into a separate diamond:

• We can get a better reflection of what the program does

CS 2212- UWO 22

Splitting Up Diamonds: Flowcharts

Example:

• Instead of writing

we write

• Now we have two new arrows

• To cover all arrows, we have to cover the two ones as well

s[i]>=‘a’ &&

s[i]<=‘z’

s[i]>=‘a’

s[i]<=‘z’

T

T

TF

F

F

CS 2212- UWO 23

Short Circuit Condition Coverage

If we split up all diamonds and then cover all arrows:

• We are achieving more thorough coverage than just edge (decision) coverage can give us

This is called short circuit (SC) condition coverage

• Recall definition of short circuit evaluation

—Evaluation of a Boolean expression is cut short as soon as it can be determined to be true or false

• Applies to language with short circuit evaluation of decisions, e.g. C, C++, Java

Similar but not identical to another notion called just “condition coverage”

• Condition coverage applies to languages without short circuit evaluation of decisions, e.g. Visual Basic

• We will not study this coverage measure

CS 2212- UWO 24

Achieving SC Condition Coverage

Consider the case conversion example

Test case “aX” is sufficient to achieve 100% edge coverage

• Whole decision (c>=‘a’) && (c<=‘z’) is tested both ways: true for ‘a’ and false for ‘X’

Is not sufficient to achieve 100% SC condition coverage

• Have not take the F arrow from the s[i] <= ‘z’ diamond

To take that arrow we need a character that is:

• Greater than or equal to ‘a’

• Greater than ‘z’

We need one such ASCII character

• The test case “aX~” will achieve 100% SC condition coverage

CS 2212- UWO 25

Achieving SC Condition Coverage in General

In general, for a decision A && B to achieve SC condition coverage, we need to design test cases to make:

• A true, B true• A true, B false• A false

We will not even be able to get to the evaluation of B unless A is true as in first two cases (due to SC evaluation)Similarly, for a decision A || B to achieve SC condition coverage, we need to design test cases to make:

• A false, B true• A false, B false• A true

We will not even be able to get to the evaluation of B unless A is false as in the first two cases

CS 2212- UWO 26

SC Condition Coverage vs. Edge Coverage

If we have achieved SC condition coverage:

• We must have evaluated each condition of an “if”, “while”, etc. both ways

• Therefore, we must have evaluated each decision both ways

Therefore, SC condition coverage is stronger than edge (decision) coverage

Is there anything stronger than SC condition coverage?

CS 2212- UWO 27

Path Coverage

“100% path coverage”: every possible path through the flowchart of the code has been executed by some test case

• Path: sequence of nodes visited during an execution

—Can contain repeated elements if execution has visited a node more than once

• The strongest possible coverage measure

For code with no loops, this is achievable

For code with while loops:

• Going around loop a different number of times means we have taken a different path

For some code, we can go around a loop any number of times

Therefore, for some code:

• it is not possible to achieve 100% path coverage

CS 2212- UWO 28

Path Coverage: Example 1

Integer classification example again:

Four paths through the code are possible

• X = 2• X = 1• X = -2• X = -1

Draw the paths for each of the above examples!Therefore we need four test cases for 100% coverage

Printf()

Scanf()

X>0 Pos=1

X%2=0 Even = 1

Printf()

End

T

T

Pos=0

Even=0

F

F

CS 2212- UWO 29

Path Coverage: Example 2

Case conversion example again

Every string of a different length will follow a different path

i=0

i++

End

s[i]=s[i]-32

s[i]!=0?

s[i]>=‘a’&&s[i]<=‘z’?

T

T

F

F

CS 2212- UWO 30

Path Coverage for Loops

Path coverage for many while loops is impossible

Most programs have while loops!

Most have at least one that can be followed an arbitrary number of times:

• e.g. A top level loop where we read a command and execute it

Therefore, for most whole programs, we don’t even attempt 100% path coverage

Path coverage is still useful for small sections of code

CS 2212- UWO 31

Strength of Coverage Measures

Line (weakest)

Node(Statement)

Edge (Decision)

Path (strongest)

Where arrow from A to B means “B is stronger than A”

CS 2212- UWO 32

Testing Loops

If we have loops, we can’t test all possible pathsHowever, just decision coverage may not spot usual kinds of errorsOften code is wrong because programmer didn’t consider what would happen when:

• Loop decision is false right from the start • Loop decision is true once, then false

Sometimes there is a maximum number (max) of times loop can be executed

• Example: loop may stop at end of an array, or when some condition is true

In these cases, code may be wrong because programmer didn’t consider what would happen when

• Loop decision is true max times• Loop decision is true max-1 times

CS 2212- UWO 33

Loop Coverage

It is therefore useful to write test cases which:

• Execute loop 0 times

• Execute loop 1 time

• Execute loop more than 1 time

• Execute loop max times (if max exists)

• Execute loop max -1 times (if max exists)

These tests are often referred to as loop coverage

• However, there is no real formal definition of loop coverage as with node, edge etc. coverage

CS 2212- UWO 34

Loop Coverage: ExerciseProvide adequate loop coverage for the following: (code finds the rightmost dot in a string, used to find extension for filename)

i = strlen(fname) -1;

while (i>0 && fname[i] != ‘.’) {i--;}

To do “loop coverage”, list some of the possible test cases:• (0 times) • (1 time) • (more than 1 time) • (max – 1 times) • (max times) Note: max here is the length of the string minus 1

Note: the code above is almost certainly correct• However, problem may not be with the loop itself, but the code that comes after• For instance, maybe the code that comes afterward:

—Expects a non-empty extension—Expects a non-empty filename before the extension—Can’t handle the case of no extension

• Loop testing will help find those problems too

fname contains: “report1.”fname contains: “report1.t”fname contains: “report1.txt”fname contains: “.txt”fname contains: “report1”