1

Software Testing and Quality Assurance

Lecture 9 - Software Testing Techniques

2

Lecture Outline Mutation Analysis

Theory of Mutation analysis and concept of mutant.

Mutation Operator Equivalent mutants

Comparing coverage criteria.

3

ExampleVoid squeeze (char s[],int c){

int i,j;for (i = j = 0; s[i] != ‘\0’; i++)

if (s[i] != c} {s[ j++ ] = s[i];

}}s[i] = ‘\0’;

}

4

ExampleVoid squeeze (char s[],int c){

int i,j;for (i = j = 0; s[i] != ‘\0’; i++)

if (s[i] != c} {s[ j ] = s[i];

}}s[i] = ‘\0’;

}

Now, function places all references to ‘c’ at position 0 in the array. (FAULT)

5

Test Cases for the Example Test cases (Test suite) is executed on

the faulty version; two possibilities An error is encountered, and the

programmer can debug the program to find the fault and repair it.

An error is not encountered, and all the test cases pass.

6

Coverage Based Testing Techniques To date, we have discussed methods

that aim to find faults by achieving coverage. Little idea as to how good the resulting test

suite is.

7

Mutation Analysis Method for measuring the effectiveness

of test suites; and As a result, we produce new test cases

to be added to that test suite.

A mutant of a program is a copy of the program, but with one slight syntactic

change.

8

Fault is a mutant. In this case, the slight syntactic change is the removal of ++ operator in the array index.

Mutant ExampleVoid squeeze (char s[],int c){

int i,j;for (i = j = 0; s[i] != ‘\0’; i++)

if (s[i] != c} {s[ j ] = s[i];

}}s[i] = ‘\0’;

}

9

Mutation Analysis Given a program, A test input for that

program, and a mutant, We say that test case kills the mutant if and only if The output of the program and the mutant differ

for the test inputs. A mutant that has been killed is said to be dead.

A mutant that has not been killed is said to be alive.

10

Coupling Effect - Theory behind Mutation Analysis A test case that distinguishes a small

fault in a program by identifying unexpected behavior is so sensitive that it will distinguish more complex faults. More complex faults are coupled with

simple faults.

This is our general observation as well. Many of the faults produced by test suites are a result of one

minor fault.

11

Coupling Effect, Mutant & Testing Any test suite that does not uncover the

mutant is inadequate. For example,we test the original squeeze

function with a test suite,and all test cases pass. Now, we deliberately insert the fault (to evaluate

the quality of test suites). If fault is uncovered, mutant is killed. Otherwise, test suites are inadequate.

Mutation Analysis Process

12

Mutation Analysis Used to guide test input generation. For

example, Find a test case that kills every mutant.

It is a way of assessing test suite quality. Test suites that kill more mutants is of

higher quality.

13

Mutation Analysis via Mutant Operator Mutant operator is a transformation rule that,

given a program, generates a mutant for that program.

For example, For ‘if (x < y)’ statement, possible mutants are If (x ≤ y) If (x > y) If (x ≥ y) If (x = y) If (x ≠ y) …..

14

Mutation Analysis via Mutant Operator Mutants are syntactic changes to a

program, so mutant operators are depend on the syntax of programming languages. The mutant operators of one programming

language may not necessarily apply to other languages.

15

Equivalent Mutants Given a program and a mutant of that

program, The mutant is said to be equivalent mutant

if, for every input, the program and the mutant produce the same output.

A major problem with mutation analysis

16

Equivalent Mutants - example Squeeze function

S [ j ++] = s [ i ];

S [ j ++] = s [abs[ i ]];

Absolute value function

Variable i only ever takes on value between 0 and the size of the array. Hence, abs[i] is irrelevant.

This mutant is equivalent can not be killed.In industry, testers aim to kill 95% of the mutants.

17

Comparing Coverage Criteria Which criteria gives the best coverage? Frankl et al. [An applicable Family of Data-Flow Testing Criteria, IEEE Transactions

on Software Engineering, 14(10), 1988] compared paths that each criteria selects.

18

Comparing Coverage CriteriaAll-Paths

All-Du-Paths Multiple condition

All-C-Use/Some-P-Use

All Uses

All-P-Uses/Some C-Uses

All-Defs

All-P-Uses

All-Edges

All-Nodes

Decision/condition

Condition

19

Comparing Coverage Criteria

If (y==1) {

x = 2;

}

If (z==1) {

a = f(x);

}

Multiple condition coverage enforces that y == 1 and z == 1 are both executed for the true and false.

To achieve any data-flow criteria, at leastOne test must execute a definition path

From the statement x = 2 to a = f(x).

20

Key points Mutation analysis is a method of

measuring effectiveness of test suites. And as a result, we produce test cases to

improve program testing. The set of paths that satisfy ‘data-flow’

and ‘coverage based testing’ are not necessarily unique.