View
916
Download
2
Category
Tags:
Preview:
DESCRIPTION
Citation preview
An Analysis and Survey of the Development of Mutation Testing[JH09]
About 30 minutes Tao Heelfinhe@gmail.com
Software Engineering LaboratoryDepartment of Computer Science, Sun Yat-Sen University
With thanks to Yue Jia and Mark HarmanThese slides are mainly extracted from Jia and Harman’s survey work
May 2011Sun Yat-Sen University, Guangzhou, China
1/371/37
[JH09] Yue Jia, Mark Harman (September 2009). "An Analysis and Survey of the Development of Mutation Testing" (PDF). CREST Centre, King's College London, Technical Report TR-09-06.
Outline
Objectives Scope Classification of Research Fundamental Hypotheses Related Concepts Future Trend Tools
Objectives of Mutation Testing
Provide “mutation adequacy score” Measure the effectiveness of a test set in
terms of its ability to detect faults
3/20
Scope of Mutation Testing
The unit level The integration level The specification level As a white-box unit test technique
Theoretical work on Mutation Testing
Hypotheses supporting Mutation Testing Optimization techniques
techniques for reducing computational cost techniques for the detection of equivalent mutants
Practical work on Mutation Testing
Applications of Mutation Testing Development work on Mutation Testing tools Empirical work
Fundamental Hypotheses
Competent Programmer Hypothesis (CPH)
Programs can be corrected by a few small syntactical changes.
Coupling Effect
Test data sets that detect simple types of faults are sensitive enough to detect more complex types of faults A simple fault is represented by a simple mutant
which is created by making a single syntactical change
A complex fault is represented as a complex mutant which is created by making more than one change
A Mutation Operator
A transformation rule that generates a mutant from the original program
Generate ‘realistic faults’
Behaviors of Mutation Operators
Mutation Objects Variables Expressions
Mutation Operation Replacement Insertion Deletion
Problems of Mutation Analysis
Executing the enormous number of mutants Human effort
The human oracle problem The equivalent mutant problem
Cost Reduction Techniques
Classification of Cost Reduction Techniques
Reduction of the generated mutants (may be taken into consideration in our approach for Fault Localization)
Reduction of the execution cost
Mutants Reduction Techniques
Mutant Sampling Random Based on the Bayesian sequential probability ratio
test (SPRT)
Mutant Clustering Based on killable test cases
Selective Mutation By reducing the number of mutation operators applied. Ignore operators ASR and SVR - redundant generation Only using ABS and ROR
Evaluation of Mutants Reduction
A mean mutation score Reduction in the number of mutants Number of equivalent mutants
Execution Cost Reduction Techniques
Strong Mutation Weak Mutation Firm Mutation
by providing a continuum of intermediate possibilities - compare state, which lies between execution (Weak Mutation) and the final output (Strong Mutation).
Equivalent Mutant Detection Techniques
10% to 40% of mutants which are equivalent
Empirical Study
Compare mutation criteria with data flow criteria such as 'all-use'
Compare mutants with real faults
Future Trend
A need for high quality higher order mutants A need to reduce the equivalent mutants A preference for semantics over syntax
mutation An interest in achieving a better balance
between cost and value A pressing need to generate test cases to kill
mutants
Life circle of Mutation Testing
Generate mutants based on specified mutation operations
Reduce mutants Run mutants against a test suite
Mutation Testing Tools
Goals of Mutation to Enhance Fault Localization
Not aim to simulate real faults Analyze the impact of mutation on the
suspiciousness of the mutated statement
Published C Mutation Testing Tools
Name Application Year Character Available Suitable
MILU C 2008Higher Order Mutation, Search-based technique, Test harness embedding
Yes No
MUFORMAT C 2008 Format String Bugs No No
ESPT C/C++ 2008 Tabular No No
CSAW C 2007Variable type optimization
Yes No
ExMAn C, Java 2006 TXL No No
SESAME C, Lustre, Pascal 2006 Assembler Injection No No
Certitude C/C++ 2006 General (Commercial) Commercially No
Plextest C/C++ 2005 General (Commercial) Commercially No
Proteum/IM 2.0 C 2001Interface Mutation,
Finite State MachinesYes Yes
Insure++ C/C++ 1998Source Code
Instrumentation (Commercial)
Commercially No
TUMS C 1995Mutant Schemata
GenerationNo No
Proteum 1.4 C 1993Interface Mutation,
Finite State MachinesNo No
Proteum
Environment Variable: PROTEUMIMHOME li -P pre-filename [-D directory] source-filename LI-filename
Call gcc to preprocess the source code file source-filename.c and produce a file pre-filename.c. Then parse the file pre-filename.c and generate various info files
opmuta [-<operator> n m] [-all n m] source-filename Li-filename Apply mutant operators to the source code and LI files. As ouput,
opmuta produces a description file in a format that muta is able to read and include in the mutant database.
Proteum – An Example: Space
space.c
li ~/smart_debugger/toolkit/proteum/li -P pre-space space li-space
li-space.nli li-space.fun li-space.cgr li-spaec.gfc
source codestrutt.h include
statement info
pre-space.c
function info call graph info def-use pair info
opmuta ~/smart_debugger/toolkit/proteum/opmuta -O Operators pre-one_statement li-one_statement > mutants.txt
mutants.txt
all mutants info
GenetrateMutants.py
Proteum.py
include
Mutants function name by directory
line number by mutant.dsc
mutation operator by mutant.dsc
Thoughts of Self-Made Mutants Generator
I once consider to implement a self-made mutants generator. Only mutate one statement Not aim to simulate real faults
IPO Input
A piece of source code Line number to mutate Mutation operators
Process Preprocess -> Scan -> Parse -> Mutate
Output Mutants
Collection of Compiler Front-end Tools
GCC-XML …failed E.g.
source code: http://www.gccxml.org/HTML/example1in.html
Parse-tree: http://www.gccxml.org/HTML/example1out.html
LLVM … not try, good for Objective-C? http://llvm.org/
Collection of Compiler Front-end Tools
GCC … failed, maybe no parse-tree for GCC -fdump-tree-fixup cfg-lineno
http://stackoverflow.com/questions/697817/how-to-make-gcc-spit-out-a-mapping-from-flow-graphs-to-source-code-line-numbers
-fdump-tree-all -fdump-rtl-all http://stackoverflow.com/questions/1496497/how-can-i-see-parse-tree-intermediate-code-optimization-code-and-assembly-code
-fprofile-arcs -ftest-coverage gcno gcda: not readable for human beings gcov: offer the line number, separated statements
Useful links
http://en.wikipedia.org/wiki/Mutation_testing http://cs.gmu.edu/~offutt/rsrch/mut.html http://www.dcs.kcl.ac.uk/pg/jiayue/repository/
Q & A
31/37
Thank you!Contact me via elfinhe@gmail.com
32/37
Abstract
33/20
Fault localization is a technique that aims to pinpoint faults by analyzing program execution spectrum, while mutation is a testing technique used by generating faulty programs called mutants. Most of existing research in fault localization focuses on the spectrum of the given programs. Yet there are few attempts to introduce mutation’s impact to enhance fault localization techniques. In this paper, we propose a strategy that automatically introduces mutation into fault localization techniques, and present variations of a heuristic method to compute suspiciousness of each statement by considering the impact of mutation. To validate our method, experiments is conducted on benchmark programs, namely Siemens, grep, gzip, sed, space, flex, make, vim, and bash. Results indicate that the method can help programmers locate faults more effectively than methods without mutation.
Recommended