Upload
markredeman
View
1.382
Download
0
Embed Size (px)
Citation preview
MUTATIONTESTINGIN PHP WITH HUMBUG
@MARKREDEMANSTUDENT APPLIED MATHEMATICSFREELANCE (WEB) DEVELOPER
Introduction to mutation testingHumbug, a mutation testing framework for PHP.Analyzing code coverage of open source projectsImproving your workflow with mutation testing
What's this talk about?
A tool to analyze stability of a piece of codeSimilar to code coverage, but betterCan find missing tests
WHAT IS MUTATION TESTING?
A quick exampleclass Customer private $orders = 0;
public function __construct($orders) $this>orders = $orders;
public function isGoldCustomer() return $this>orders > 10;
function testIsGoldCustomer() $this>assertFalse((new Customer(0))>isGoldCustomer()); // 100% line coverage :D $this>assertTrue((new Customer(11))>isGoldCustomer());
public function isGoldCustomer() return $this>orders >= 10;
class Customer private $orders = 0;
public function __construct($orders) $this>orders = $orders;
public function isGoldCustomer() return $this>orders > 10;
function testIsGoldCustomer() $this>assertFalse((new Customer(0))>isGoldCustomer()); // 100% line coverage :D $this>assertFalse((new Customer(10))>isGoldCustomer()); $this>assertTrue((new Customer(11))>isGoldCustomer());
Generated mutants aresimilar to real faults
- Andrew, Briand, Labiche, ICSE 2005
Some Definitions
a piece of code that has been changed (mutated) by a mutator
killed if at least 1 test failsescaped if at all test passequivalent if there does not exist a test case which can
distinguish the mutant from the original codeuncovered mutant is not covered by a testfatal mutant produces a fatal errortimout unit tests exceed allowed timeout
Mutation
Killed Mutantclass Customer private $orders = 0;
public function __construct($orders) $this>orders = $orders;
public function isGoldCustomer() return $this>orders < 10; // mutated ">" to "<"
function testIsGoldCustomer() $this>assertFalse((new Customer(0))>isGoldCustomer()); $this>assertTrue((new Customer(11))>isGoldCustomer());
Escaped Mutantclass Customer private $orders = 0;
public function __construct($orders) $this>orders = $orders;
public function isGoldCustomer() return $this>orders >= 10; // mutated ">" to ">="
function testIsGoldCustomer() $this>assertFalse((new Customer(0))>isGoldCustomer()); $this>assertTrue((new Customer(11))>isGoldCustomer());
Equivalent Mutantfunction sum_is_zero(array $values) $sum = 0;
foreach ($values as $value) $sum += $value;
return $sum === 0;
function sum_is_zero(array $values) $sum = 0;
foreach ($values as $value) $sum = $value; // mutated "+" to ""
return $sum === 0;
Increments MutatorDecrements MutatorInvert Negatives MutatorReturn Values Mutator
Math MutatorNegate MutatorConditionals BoundaryMutatorRemove ConditionalsValues Mutator
MUTATOROperator that changes (mutates) a piece of code
Original Mutated+ +* // *% *
MATH MUTATORS
Original Mutated== !=!= ==<= >>= << >=> <=
CONDITIONAL MUTATORS
::
Mutator descriptionNegate Conditionals replace condition by ! (condition)
Remove Conditionals substitute conditional with true or falseIncrements crement numeric values by 1Decrements crement numeric values by 1
Invert Negatives multiply numeric values by 1Return Values remove return statements
MetricsMutation Score Indicator (MSI):
percentage of mutants covered & killed by testsMutation Code Coverage:
percentage of mutants covered by testsCovered Code MSI:
percentage of killed mutants that were coverd by tests$vanquishedTotal = $killedCount + $timeoutCount + $errorCount;$measurableTotal = $totalCount $uncoveredCount; // = $vanquishedTotal + $escapedCount
$msi = round(100 * ($vanquishedTotal / $totalCount));$coveredRate = round(100 * ($measurableTotal / $totalCount));$cc_msi = round(100 * ($vanquishedTotal / $measurableTotal));
Metrics ExampleTests: 361Line Coverage: 64.86%Mutation Score Indicator (MSI): 47%Mutation Code Coverage: 67%Covered Code MSI 70%
653 Mutants were generated284 mutants were killed218 mutants were not covered by tests131 covered mutants were not detected17 fatal errors were encountered3 time outs were encountered
47% of all mutations were detected versus 65% line coverage.
HUMBUGA Mutation Testing framework for PHPMeasures the real effectiveness of your test suites and assist in their improvement.
It eats Code Coverage for breakfast
Git:git clone https://github.com/padraic/humbug.gitcd humbug/path/to/composer.phar installbin/humbug
Phar:wget https://padraic.github.io/humbug/downloads/humbug.pharwget https://padraic.github.io/humbug/downloads/humbug.phar.pubkeychmod +x humbug.phar
Composer:composer global require 'humbug/humbug=~1.0@dev'# And add ~/.composer/vendor/bin to your path to make it easily executableexport PATH=~/.composer/vendor/bin:$PATH
Installation
Configuration$> git clone [email protected]:yourname/yourproject.git# Check if all tests are green$> phpunit$> humbug configure
_ _ _| || |_ _ _ __ | |__ _ _ __ _| __ | || | ' \| '_ \ || / _ ||_||_|\_,_|_|_|_|_.__/\_,_\__, | |___/Humbug version 1.0dev
Humbug configuration tool.It will guide you through Humbug configuration in few seconds.
When choosing directories, you may enter each directory and press return.To exit directory selection, please leave the next answer blank and press return.
What source directories do you want to include? : srcWhat source directories do you want to include? :Any directories to exclude from within your source directories? :Single test suite timeout in seconds [10] :Where do you want to store the text log? [humbuglog.txt] :Where do you want to store the json log (if you need it)? :Generate "humbug.json.dist"? [Y]:Configuration file "humbug.json.dist" was created.
Running Humbug$> humbug
_ _ _| || |_ _ _ __ | |__ _ _ __ _| __ | || | ' \| '_ \ || / _ ||_||_|\_,_|_|_|_|_.__/\_,_\__, | |___/Humbug version 1.0dev
Humbug running test suite to generate logs and code coverage data...
97 [==========================================================] 2 secs
Humbug has completed the initial test run successfully.Tests: 97 Line Coverage: 98.27%
Humbug is analysing source files...
Mutation Testing is commencing on 19 files...(.: killed, M: escaped, S: uncovered, E: fatal error, T: timed out)
E......M.M.......MMTT.....MMS..........................M.M.. | 60 ( 9/19)M..................M....E.......MMMMS
97 mutations were generated: 77 mutants were killed 2 mutants were not covered by tests 14 covered mutants were not detected 2 fatal errors were encountered 2 time outs were encountered
Metrics: Mutation Score Indicator (MSI): 84% Mutation Code Coverage: 98% Covered Code MSI: 85%
Remember that some mutants will inevitably be harmless (i.e. false positives).
Time: 31.39 seconds Memory: 8.75MBHumbug results are being logged as TEXT to: humbuglog.txt
Analyzing Humbughumbug.log.txt
2) \Humbug\Mutator\ConditionalBoundary\GreaterThanDiff on \Carbon\CarbonInterval::__construct() in /Carbon/src/Carbon/CarbonInterval.php: Original+++ New@@ @@ $specDays += $weeks > 0 ? $weeks * Carbon::DAYS_PER_WEEK : 0; $specDays += $days > 0 ? $days : 0;+ $specDays += $days >= 0 ? $days : 0;
$spec .= ($specDays > 0) ? $specDays.static::PERIOD_DAYS : '';
if ($hours > 0 || $minutes > 0 || $seconds > 0) $spec .= static::PERIOD_TIME_PREFIX; $spec .= $hours > 0 ? $hours.static::PERIOD_HOURS : '';
humbug.log.json
"summary": "total": 26, "kills": 23, "escapes": 1, "errors": 0, "timeouts": 0, "notests": 2, "covered_score": 96, "combined_score": 88, "mutation_coverage": 92,"escaped": [ "file": "src\/BroadwayDemo\/Basket\/Basket.php", "mutator": "\\Humbug\\Mutator\\ConditionalBoundary\\GreaterThan", "class": "\\BroadwayDemo\\Basket\\Basket", "method": "productIsInBasket", "line": 101, "diff": " Original\n+++ New\n@@ @@\n \n return isset($this>productCountById[ "tests": [ "BroadwayDemo\\Basket\\AddProductToBasketTest::it_adds_a_product_to_a_basket", "BroadwayDemo\\Basket\\AddProductToBasketTest::multiple_products_can_be_added_to_a_basket" "BroadwayDemo\\Basket\\AddProductToBasketTest::a_product_can_be_added_to_a_basket_multiple_times" "BroadwayDemo\\Basket\\CheckoutTest::it_checks_out_a_basket", "BroadwayDemo\\Basket\\CheckoutTest::it_cannot_checkout_a_basket_that_has_been_emptied" "BroadwayDemo\\Basket\\CheckoutTest::nothing_happens_when_checking_out_a_basket_for_a_second_time" "BroadwayDemo\\Basket\\RemoveProductFromBasketTest::it_removes_a_product_that_was_added" "BroadwayDemo\\Basket\\RemoveProductFromBasketTest::it_does_nothing_when_removing_a_product_that_is_not_in_a_basket" "BroadwayDemo\\Basket\\RemoveProductFromBasketTest::it_only_removes_one_instance_of_a_product" ], "stderr": "", "stdout": "TAP version 13"],
OptionsTimeout:
humbug timeout=10Restricting files:
humbug file=PrimeFactor.phphumbug file=*Driver.php
Incremental analysis:humbug incrementalCan off significant performance boosts by caching previous results in/home/padraic/.humbug..
MutatorsBinary Arithmetic
Boolean SubstitutionConditional BoundariesNegated Conditionals
IncrementsReturn Values
Literal NumbersIf Statements
HUMBUG TEST RESULTS
Package LC MSI MCC CCM Executiontime
carbon/carbon 61% 95% 100% 95% 2.25msymfony/event-dispatcher
85% 54% 69% 78% 20s
hylianshield/validator 100% 75% 89% 85% 1.35mmathiasverraes/money 96% 92% 100% 92% 16.9sthephpleague/fractal 98% 84% 98% 85% 31sbroadway/broadway-demo
96% 88% 92% 86% 5s
broadway/broadway 57% 49% 56% 87% 46s
Line Coverage (LC), Mutation Score Indicator (MSI), Mutation Code Coverage (MCC), Covered Code MSI (CCM)Timout option set to 2 seconds.
TDD workflow with mutation testing
Read the book
Other toolsRuby: MutantJava: PitestC#: NinjaturtlesPython: MutPyMatlab: MatMute
Concluding remarksMutation testing will improve the quality of your testsIs becoming more mainstream over the last yearsWrite small (fast) tests
I have not failed. I've justfound 10,000 ways thatwon't work
- Thomas Edison
Find these slides atmutation.markredeman.nl