Software Testing and Validation SWE 434 Mutation testing

Software Testing and Validation

SWE 434Mutation testing

2

Dynamic Testing• Selection of test data is broadly based on the following

techniques:

– Functional program testing

– Control flow testing

– Data flow testing

• Questions:

– How to assess the effectiveness of the test suite ?

– How many bugs are remaining ?

3

Mutation testing• Create a number of mutants, i.e., faulty versions of program

– Each mutant contains one fault– Fault created by using mutant operators

• Run test on the mutants (random or selected)– When a test case reveals a fault, save test case and remove mutant

from the set, i.e., it is killed– Continue until all mutants are killed

• Results in a set of test cases with high quality• Need for automation

4

Mutation Testing

• Modify a program by introducing a single small change to the code

• A modified program is called mutant

• A mutant is said to be killed when the execution of test case cause it to fail. The mutant is considered to be deadGiven a mutant m M for a ground string program P and a test t, t is said to kill m if and only if the output of t on P is different from the output of t on m.

Mutation Testing

• Mutants

int getMax(int x, int y) {int getMax(int x, int y) { int max; int max;

if (x >y) if (x >=y) max = x; max = x;

else else max = y; max = y; return max; return max;} }

5

int getMax(int x, int y) { int max;

if (x >y) max = x;

else max = x;return max;}

Examples of Mutant Operators

6

Mutant operator In program In mutant

Variable replacement z=x*y+1; x=x*y+1;z=x*x+1;

Relational operator replacement

if (x<y) if(x>y)if(x<=y)

Off-by-1 z=x*y+1; z=x*(y+1)+1;z=(x+1)*y+1;

Replacement by 0 z=x*y+1; z=0*y+1;z=0;

Arithmetic operator replacement

z=x*y+1; z=x*y-1;z=x+y-1;

7

Mutation Testing

• A mutant is an equivalent tot the given program if it always produce the same output as the original program

• A mutant is called killable or stubborn, if the existing set of test cases is insufficient to kill it

• A mutation score for a set of test cases is the percentage of non-equivalent mutants killed by the test suite

• The test suite is said to be mutation-adequate if its mutation score is 100%

(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 8

How mutants survive

• A mutant may be equivalent to the original program– Maybe changing (x < 0) to (x <= 0) didn’t change the

output at all! The seeded “fault” is not really a “fault”.• Determining whether a mutant is equivalent may be easy or

hard; in the worst case it is undecideable

• Or the test suite could be inadequate– If the mutant could have been killed, but was not, it

indicates a weakness in the test suite– But adding a test case for just this mutant is a bad idea.

We care about the real bugs, not the fakes!

9

Mutation Testing: an example

Consider the following program P:

• main(argc,argv)• int argc, r, i;• char *argv[];• { r = 1;• for i = 2 to 3 do• if (atoi(argv[i-1]) > atoi(argv[r-1])) r = i;• printf(“Value of the rank is %d \n”, r);• exit(0); }

Test Case 1: input: 1 2 3 output: Value of the rank is 3Test Case 2: input: 1 2 1 output: Values of the rank is 2Test Case 3: input: 3 1 2 output: Value of the rank is 1

10


Test cases for the program P:

• Test Case 1:

input: 1 2 3

output: Value of the rank is 3• Test Case 2:

input: 1 2 1

output: Values of the rank is 2• Test Case 3:

input: 3 1 2

output: Value of the rank is 1

11


Mutant 1:

• main(argc,argv)• int argc, r, i;• char *argv[];• { r = 1;• for i = 1 to 3 do• if (atoi(argv[i-1]) > atoi(argv[r-1])) r = i;• printf(“Value of the rank is %d \n”, r);• exit(0); }

Test Case 1: input: 1 2 3 output: Value of the rank is ?Test Case 2: input: 1 2 1 output: Values of the rank is ?Test Case 3: input: 3 1 2 output: Value of the rank is ?

12


Mutant 2:

• main(argc,argv)• int argc, r, i;• char *argv[];• { r = 1;• for i = 2 to 3 do• if (i-1) > atoi(argv[r-1])) r = i;• printf(“Value of the rank is %d \n”, r);• exit(0); }


13


Mutant 3:

• main(argc,argv)• int argc, r, i;• char *argv[];• { r = 1;• for i = 2 to 3 do• if (atoi(argv[i-1]) >= atoi(argv[r-1])) r = i;• printf(“Value of the rank is %d \n”, r);• exit(0); }


14


Mutation 4:

• main(argc,argv)• int argc, r, i;• char *argv[];• { r = 1;• for i = 2 to 3 do• if (atoi(argv[r-1]) > atoi(argv[r-1])) r = i;• printf(“Value of the rank is %d \n”, r);• exit(0); }


15

Mutation Testing: an exampleSummary:

• After Execution of modified programs against the test suite, you will get the results:

• Mutants 1 & 3: Programs will pass the test suite, i.e., mutants 1 & 3 are not killable

• Mutant 2: Program will fail test cases 2• Mutant 4: Program will fail test case 1 and test cases 2

• Mutation score is 50%, assuming mutants 1 & 3 non-equivalent

16

Mutation Testing: an exampleImprovements:

• The score is found to be low because we assumed mutants 1 & 3 are nonequivalent

• We need to show that mutants 1 and 3 are equivalent mutants or those are killable

• To show that those are killable, we need to add new test cases to kill these two mutants

17

Mutation Testing: an exampleImprovements:

• First, let us analyze mutant 1 in order to derive a “killer” test. The difference between P and mutant 1 is the starting point

• Mutant 1 starts with i = 1, whereas P starts with i = 2. There is no impact on the result r. Therefore, we conclude that mutant 1 is an equivalent mutant

• Second, if we add a fourth test case as follows:

Test Case 4:

input: 2 2 1

• Program P will produce the output “Value of the rank is 1” and mutant 3 will produce the output “Value of the rank is 2”

• Thus, this test data kills mutant 3, which give us a mutation score 100%

18


Recap:

• Test Case 1:

input: 1 2 3


input: 1 2 1

output: Values of the rank is 2• Test Case 3:

input: 3 1 2


input: 2 2 1

output: Value of the rank is 1

which give us a mutation score 100% !!!

If process is not error-free, fix it

Test Mutants

The Mutation Process

Process

MutationMutationMutation Tests

Test Process

Create Mutants

Yes

Test CompleteNo

Any Live Mutants?

Problem with Tests?

Any Mutations that are caught by tests are killed

New Test Data

Fault Forecasting

20

Mutation testing

Fault ForecastingHow to determine number of remaining bugs?

Definition:Fault forecasting is conducted by performing an evaluation

of the system behavior with respect to fault occurrence or activation

Fault Injection technique:The idea is to inject (seed) some faults in the program and

calculate the remaining bugs based on detecting the seeded faults [Mills 1972]. Assuming that the probability of detecting the seeded and nonseeded faults are the same.

21

Fault Forecasting

22

Ns

Nd

Nr

nsnd

Fault Forecasting Example

23

Tools

24

Mutation testing

© Aditya P. Mathur 2009

Tools for mutation testing

• As with any other type of test adequacy assessment, mutation based assessment must be done with the help of a tool.

• There are few mutation testing tools available freely. Two such tools are Proteum for C from Professor Maldonado and muJava for Java from Professor Jeff Offutt. We are not aware of any commercially available tool for mutation testing. See the textbook (Aditya) for a more complete listing of mutation tools.

Q & A

Documents

Software Testing and Validation SWE 434 Mutation testing