Upload
nicolas-frankel
View
184
Download
2
Embed Size (px)
Citation preview
MUTATION TESTINGTO THE RESCUE OF YOUR TESTS
ΝΙΚΟΛΑΣ ΦΡΑΝΚΕΛ
ME, MYSELF AND I
@nicolas_frankel #mutationtesting
By day
• Consultant ($$$)
By night
•Developer
• Blogger
• Book author
• Teacher/trainer
2
HYBRIS, AN SAP COMPANY
@nicolas_frankel #mutationtesting 3
MANY KINDS OF TESTING
@nicolas_frankel #mutationtesting
Unit Testing
Integration Testing
End-to-end Testing
Performance Testing
Penetration Testing
Exploratory Testing
etc.
4
THEIR ONLY SINGLE GOAL
@nicolas_frankel #mutationtesting
Ensure the quality of the production code
5
THE PROBLEM
@nicolas_frankel #mutationtesting
How to check the quality of the testing code?
6
CODE COVERAGE
@nicolas_frankel #mutationtesting
“Code coverage is a measure used to describe the degree to which the source code of a program is tested”
--Wikipediahttp://en.wikipedia.org/wiki/Co
de_coverage
7
MEASURING CODE COVERAGE
@nicolas_frankel #mutationtesting
Check whether a source code line is executed during a test
•Or Branch Coverage
8
COMPUTING CODE COVERAGE
CC =Lexecuted
Ltotal*100
CC: Code Coverage(in percent)
Lexecuted: Number of executed lines of code
Ltotal: Number of total lines of code
@nicolas_frankel #mutationtesting 9
JAVA TOOLS FOR CODE COVERAGE
@nicolas_frankel #mutationtesting
JaCoCo
Clover
Cobertura
etc.
10
100% CODE COVERAGE?
@nicolas_frankel #mutationtesting
“Is 100% code coverage realistic? Of course it is. If you can write a line of code, you can write another that tests it.”
Robert Martin (Uncle Bob)https://twitter.com/unclebobmartin/status/5596662050966732
8
11
ASSERT-LESS TESTING
@Test
public void add_should_add() {
new Math().add(1, 1);
}
@nicolas_frankel #mutationtesting
But, where is the
assert?As long as the Code Coverage is
OK…
12
CODE COVERAGE AS A MEASURE OF QUALITY
@nicolas_frankel #mutationtesting
Any metric can be gamed!
Code coverage is a metric…
⇒ Code coverage can be gamed
• On purpose
• Or by accident
13
CODE COVERAGE AS A MEASURE OF QUALITY
@nicolas_frankel #mutationtesting
Code Coverage lulls you into a false sense of security…
14
THE PROBLEM STILL STANDS
@nicolas_frankel #mutationtesting
Code coverage cannot ensure test quality
• Is there another way?
Mutation Testing to the rescue!
15
THE CAST
@nicolas_frankel #mutationtesting
William Stryker
Original Source Code
Jason Stryker
Modified Source Code
a.k.a “The Mutant”
16
public class Math {
public int add(int i1, int i2) {
return i1 + i2;
}
}
@nicolas_frankel #mutationtesting
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}
17
STANDARD TESTING
@nicolas_frankel #mutationtesting
✔Execute Test
18
MUTATION TESTING
@nicolas_frankel #mutationtesting
?Execute SAME Test
MUTATION
19
MUTATION TESTING
@nicolas_frankel #mutationtesting
✗
✔Execute SAME Test
Execute SAME Test
Mutant
Killed
Mutant
Survived
20
KILLED OR SURVIVING?
@nicolas_frankel #mutationtesting
Surviving means changing the source code did notchange the test result
• It’s bad!
Killed means changing the source code changed the test result
• It’s good
21
TEST THE CODE
@nicolas_frankel #mutationtesting
✔Execute Test
public class Math {
public int add(int i1, int i2) {
return i1 + i2;
}
}
@Test
public void add_should_add() {
new Math().add(1, 1);
}
22
✔Execute Test
SURVIVING MUTANT
@nicolas_frankel #mutationtesting
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}
@Test
public void add_should_add() {
new Math().add(1, 1);
}
23
TEST THE CODE
@nicolas_frankel #mutationtesting
@Test
public void add_should_add() {
int sum = new Math().add(1, 1);
Assert.assertEquals(sum, 2);
}
✔Execute Test
public class Math {
public int add(int i1, int i2) {
return i1 + i2;
}
}
24
KILLED MUTANT
@nicolas_frankel #mutationtesting
✗Execute SAME Test
@Test
public void add_should_add() {
int sum = new Math().add(1, 1);
Assert.assertEquals(sum, 2);
}
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}
25
MUTATION TESTING IN JAVA
@nicolas_frankel #mutationtesting
PIT is a tool for Mutation testing
Available as
• Command-line tool
•Ant target
•Maven plugin
26
MUTATORS
@nicolas_frankel #mutationtesting
Mutators are patterns applied to source code to produce mutations
27
PIT MUTATORS SAMPLE
Name Example source Result
Conditionals Boundary > >=
Negate Conditionals == !=
Remove Conditionals foo == bar true
Math + -
Increments foo++ foo--
Invert Negatives -foo foo
Inline Constant static final FOO= 42 static final FOO = 43
Return Values return true return false
Void Method Call System.out.println("foo")
Non Void Method Call long t = System.currentTimeMillis() long t = 0
Constructor Call Date d = new Date() Date d = null;
@nicolas_frankel #mutationtesting 28
IMPORTANT MUTATORS
@nicolas_frankel #mutationtesting
Conditionals Boundary
• Potential serious bug hiding there
•if (foo > bar)
29
IMPORTANT MUTATORS
@nicolas_frankel #mutationtesting
Void Method Call• Assert.checkNotNull()
• connection.close()
30
REMEMBER
@nicolas_frankel #mutationtesting
It’s not because the IDE generates code safely that it will never change
• equals()
• hashCode()
31
ENOUGH TALK…
@nicolas_frankel #mutationtesting 32
SH… HAPPENS
@nicolas_frankel #mutationtesting
False positives Imperfect Sloooooooooooooooow
33
PIT IS IMPERFECT
if (p < 0)
...
// changed condition boundary -> survived:
if (p > 0)
...
return 0;
@nicolas_frankel #mutationtesting 34
PIT IS IMPERFECT
void rebootMachine() {
// removed method call:
checkUserPermissions();
Runtime.getRuntime().exec("reboot");
}
@nicolas_frankel #mutationtesting 35
WHY SO SLOW?
@nicolas_frankel #mutationtesting
Analyze test code
For each class under test
• For each mutator
• Create mutation
• For each mutation
• Run test
• Analyze result
• Aggregate results
36
WORKAROUNDS
Increase number of threads
Set a limited a set of mutators
Limit scope of target classes
Limit number of tests
Limit dependency distance
Use incremental analysis
Don’t bind to the test phase
Use scmMutationCoverage
@nicolas_frankel #mutationtesting 37
INCREMENTAL ANALYSIS
Metadata stored between runs
Mutant will not be checked again, if:
• timed out, and class has not changed• killed, and neither class nor test have changed• survived, and there are no new/changed tests for it
@nicolas_frankel #mutationtesting 38
THE SILVER BULLET?
@nicolas_frankel #mutationtesting
Checks the relevance of your unit tests
Points out potential bugs
39
THE REST IS UP TO YOU
@nicolas_frankel #mutationtesting
Validate the assembled application
• Integration Testing
Check the performance
• Performance Testing
Look out for display bugs
• End-to-end testing
Etc.
40
TESTING IS ABOUT ROI
@nicolas_frankel #mutationtesting
Don’t test to achieve 100% coverage
Test because it saves money in the long run
Prioritize:
• Business-critical code
• Complex code
41
Q&A
@nicolas_frankel #mutationtesting
http://blog.frankel.ch/@nicolas_frankelhttp://frankel.in/https://git.io/vznQK
42