Practical Techniques for Improving the Efficiency and Usability of … · algorithm analysis data ... effectiveness empirical genetic performance understanding using approach commercialofftheshelf

Practical Techniques for Improving theEfficiency and Usability of Mutation Analysis

for Java Programs

Gregory M. Kapfhammer†

Department of Computer ScienceAllegheny College

http://www.cs.allegheny.edu/∼gkapfham/

University of Sheffield – February 3, 2012†Joint with Rene Just and Franz Schweiggert (University of Ulm) and Jonathan Miller Kauffman (Allegheny College)

http://www.cs.allegheny.edu/~gkapfham/

Introduction Mutation Analysis Empirical Evaluation Conclusion

Important Points

Accessing the Presentation

Scan this QR Code with your smartphone!

... or, visit this Web site:

http://is.gd/rekiwo

... or, ask me for a USB drive!

Kapfhammer Allegheny College

Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs

http://is.gd/rekiwo


Important Points

Presenter Introduction: Gregory M. Kapfhammer

test

testing

softwaresuites

prioritization

components

coverageem

piric

allyregressionsuiteapplications

evaluating

algo

rithm

analysis

data

effectiveness

empirical

genetic

perf

orm

ance

understanding

using

approachcommercialofftheshelf

comparison

comprehensive

constrainedcreation

databasecentric envi

ronm

ents

execution

findi

ng

framework

identifying

interactive

javamethods

multiplots

mutation

party

prioritized

reduction

relational

study

third

timeaware

towards

105

adequacy

appr

oach

es

array

automatically

building

callchallenges

chapter

communication

compare

compressing

com

pute

r

conditional

cost

cots

covering

creating

criteria

database

databaseaware

data

base

driv

en

databases

declarativedependable

detection

devices

distributed

distributing

duringsearchbased

dynamic

efficiency

efficient

engi

neer

ing

environment

evaluate

examination

executing

experimental

family

flow

forward

frameworks

free

generation

greedy

gui

hamiltonian

handbook

heaps

implementation

impr

ove

incorporating

incr

ease

information

initial

intranode

invariant

javaspace

javaspacebased

kernel

knapsack

linux

measurement

memory

method

monitoring

operators

paths

poster

potential

preliminary

primitives

prio

ritiz

atio

ns

prio

ritiz

ers

prioritizing

problematic

receive

remote

reports

resource

resourceconstrained

results

role

runtime

science

searchbased

selectionsolutions

solvers

space

studies

stud

yingsupported

synthetic

techniques

transmission

transparently

trees

tuple

unstructured

wrappers




Software Testing

What is a Test Case?

MethodUnder Test




Software Testing


MethodUnder Test

Input




Software Testing


MethodUnder Test

Input Output




Software Testing


MethodUnder Test

TestSet Up




Software Testing


MethodUnder Test

TestSet Up

Input




Software Testing


MethodUnder Test

TestSet Up

Input Output




Software Testing


MethodUnder Test

TestSet Up

Input Output

TestClean Up




Software Testing


MethodUnder Test

TestSet Up

Input Output

TestClean Up

TestOracle




Software Testing


MethodUnder Test

TestSet Up

Input Output

TestClean Up

TestOracle

ExpectedOutput




Software Testing


MethodUnder Test

TestSet Up

Input Output

TestClean Up

TestOracle

ExpectedOutput

TestVerdict




Software Testing


MethodUnder Test

TestSet Up

Input Output

TestClean Up

TestOracle

ExpectedOutput

TestVerdict

ExpectedOutput

Output




Software Testing


MethodUnder Test

TestSet Up

Input Output

TestClean Up

TestOracle

ExpectedOutput

TestVerdict

ExpectedOutput

Output

TestVerdict

The test case passes and the code is correct!




Software Testing


MethodUnder Test

TestSet Up

Input Output

TestClean Up

TestOracle

ExpectedOutput

TestVerdict

ExpectedOutput

Output




Software Testing


MethodUnder Test

TestSet Up

Input Output

TestClean Up

TestOracle

ExpectedOutput

TestVerdict

ExpectedOutput

Output

TestVerdict

The test case fails and a defect is found!




Software Testing

What is a Test Suite?

T1 T2




Software Testing


T1 T2 T3 T4




Software Testing


T1 T2 T3 T4 T5 T6




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2 F3 F4




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2

Requirements R = {R1, . . . ,R6}, Features F = {F1, . . . ,F4}, Bug Fixes B = {B1,B2}




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2


B2B2B2B2




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2


B2B2B2B2B2B2




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2


B2B2B2B2B2B2B2B2B2B2




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2





Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2


How Good is Test Suite T ?




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2



Coverage Analysis




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2



Coverage Analysis Mutation Analysis




Fundamental Concepts

Overview of Mutation Analysis

MutationOperator






MutationOperator

MutationOperator






MutationOperator

MutationOperator

MutationOperator

MutationOperator






MutationOperator

MutationOperator

MutationOperator

MutationOperator






MutationOperator

MutationOperator

MutationOperator

MutationOperator

Methodicallyinject smallsyntacticalfaults into

the programunder test






MutationOperator

MutationOperator

MutationOperator

MutationOperator








MutationOperator

MutationOperator

MutationOperator

MutationOperator








MutationOperator

MutationOperator

MutationOperator

MutationOperator








MutationOperator

MutationOperator

MutationOperator

MutationOperator






Test Case T1 Test Case T2 Test Case T3 Test Case T4

Execute thetest suite after

enabling asingle mutantin the program

under test









under test









under test









under test









under test









under test









under test









under test









under test









under test

The test suitecannot kill themutant – either

a test suiteweakness oran equivalent

mutant!









under test

Repeat thisprocess forall of the

test casesand mutants– calculate

mutation scorewhen finished





Contributions of this Presentation

EfficientMutationAnalysis







Challenges







Challenges

Solutions







Challenges

Solutions

Conditional Mutation







Challenges

Solutions


Syntax TreeTransformation







Challenges

Solutions



Expressionsand Statements







Challenges

Solutions




CompilerIntegrated







Challenges

Solutions




CompilerIntegrated

ComprehensiveEmpirical Study







Challenges

Solutions




CompilerIntegrated

ComprehensiveEmpirical Study

Efficient Technique - Fully Integrated into the Java 6 SE Compiler





Understanding Mutation Analysispublic int eval(int x){

int a=3, b=1, y;

y = a * x;

y += b;return y;

}

public int max(int a, int b){int max = a;

if(b>a){

max=b;}

return max;}

=⇒

=⇒






int a=3, b=1, y;

y = a * x;

y += b;return y;

}


if(b>a){

max=b;}

return max;}

=⇒

=⇒








int a=3, b=1, y;

y = a * x;

y += b;return y;

}


if(b>a){

max=b;}

return max;}

=⇒

=⇒






int a=3, b=1, y;

y = a * x;

y += b;return y;

}


if(b>a){

max=b;}

return max;}

=⇒

=⇒

• y = a - x;

• y = a + x;

• y = a / x;

• if(b < a)

• if(b != a)

• if(b == a)






int a=3, b=1, y;

y = a * x;

y += b;return y;

}


if(b>a){

max=b;}

return max;}

=⇒

=⇒

Unbiasedand powerfulmethod forassessing

oracles andinput values






int a=3, b=1, y;

y = a * x;

y += b;return y;

}


if(b>a){

max=b;}

return max;}

=⇒

=⇒

Unbiasedand powerfulmethod forassessing

oracles andinput values

Useful methodfor fault seeding

during theempirical study

of testingtechniques





Mutation Analysis Challenges

MutantGeneration






MutantGeneration

MutationOperators






MutantGeneration

MutationOperators

Program






MutantGeneration

MutationOperators

Program

Mutants






MutantGeneration

MutationOperators

Program

Mutants

Often Yields aSubstantial Num-

ber of Mutants






MutantGeneration

MutationOperators

Program

Mutants


ber of Mutants

High Time Over-head for Generation






MutantGeneration

MutationOperators

Program

Mutants


ber of Mutants


MutationAnalysis






MutantGeneration

MutationOperators

Program

Mutants


ber of Mutants


MutationAnalysisTests






MutantGeneration

MutationOperators

Program

Mutants


ber of Mutants


MutationAnalysisTests Results






MutantGeneration

MutationOperators

Program

Mutants


ber of Mutants



Individually Executing theMutants is Too Expensive






MutantGeneration

MutationOperators

Program

Mutants


ber of Mutants



Individually Executing theMutants is Too Expensive

PriorSolutions?





Prior Work in Mutation Analysis

Improving Mutation Analysis






Improving Mutation Analysis Offutt andUntch







Do Fewer







Do Fewer

Sampling Selection







Do Fewer Do Smarter







Do Fewer Do Smarter

Distributed Weak Mutation







Do Fewer Do Smarter

Do Faster







Do Fewer Do Smarter

Do Faster

CompilerIntegrated

BytecodeTransformation

MutantSchemata







Do Fewer Do Smarter

Do Faster

Higher OrderMutation

Jia andHarman




Mutation Analysis with MAJOR

Practical Mutation Analysis

Practical (adjective):

1 Of or concerned with the actual doing or use of somethingrather than with theory and ideas

2 (of an idea, plan, or method) Likely to succeed or beeffective in real circumstances; feasible

3 Suitable for a particular purpose





Practical Mutation Analysis





What are the practical techniques that MAJOR employs forimproving the efficiency and usability of mutation analysis?













Encapsulates allmutants withinthe same block








Can be inte-grated withinthe compiler








Transforms theabstract syntax

tree (AST)









Transforms theabstract syntax

tree (AST)

Stmt → Conditional Stmt(if-then-else, switch)

Expr → Conditional Expr(conditional operator ?:)






Transforming the AST

public int eval(int x){int a=3, b=1, y;

y = a * x ;

y += b;return y;

}

⇓

=⇒

=⇒







y = a * x ;

y += b;return y;

}

⇓

ASSIGN

IDENT

y

BINARY

∗

a x

=⇒

=⇒







y = a * x ;

y += b;return y;

}

⇓

ASSIGN

IDENT

y

BINARY

∗

a x

=⇒

=⇒







y = a * x ;

y += b;return y;

}

⇓

ASSIGN

IDENT

y

BINARY

∗

a x

=⇒

=⇒

ASSIGN

IDENT

y

COND-EXPR

THEN

BINARY

+

a x

COND

(M NO ==2)

ELSE

COND-EXPR

THEN

BINARY

-

a x

COND

(M NO ==1)

ELSE

BINARY

∗

a x





Source Code View of Inserting Mutantspublic int eval(int x){

int a=3, b=1, y;

y = a * x ;

y += b;return y;

}

1 Define mutation operators MOP(x ∗ y) = {x − y , x + y , x/y}

2 Determine whether current expression or statement isaffected by mutation

3 Apply mutation operators






int a=3, b=1, y;

y = a * x ;

y += b;return y;

}









int a=3, b=1, y;

y = a * x ;

y += b;return y;

}









int a=3, b=1, y;

y = (M_NO==1)? a - x :a * x ;

y += b;return y;

}









int a=3, b=1, y;

y = (M_NO==2)? a + x :(M_NO==1)? a - x :

a * x ;

y += b;return y;

}









int a=3, b=1, y;

y = (M_NO==3)? a / x :(M_NO==2)? a + x :(M_NO==1)? a - x :

a * x ;

y += b;return y;

}









int a=3, b=1, y;


a * x ;

y += b;return y;

}

Mutants that are not ex-ecuted cannot be killed








Collecting and Using Mutation Coveragepublic int eval(int x){

int a=3, b=1, y;


a * x ;

y += b;return y;

}







int a=3, b=1, y;


(M_NO==0 &&

COVERED(1,3))?

a * x : a * x ;

y += b;

return y;}


Determine coveredmutants with addi-

tional instrumentation






int a=3, b=1, y;


(M_NO==0 &&

COVERED(1,3))?

a * x : a * x ;

y += b;

return y;}


Determine coveredmutants with addi-

tional instrumentation

Only execute and investi-gate the covered mutants





MAJOR’s Compiler

MAJOR’sCompiler





MAJOR’s Compiler

MAJOR’sCompiler

Enhanced StandardJava Compiler





MAJOR’s Compiler

MAJOR’sCompiler


Source Files





MAJOR’s Compiler

MAJOR’sCompiler


Source Files

CommonCompiler Options





MAJOR’s Compiler

MAJOR’sCompiler


Source Files


Domain SpecificLanguage





MAJOR’s Compiler

MAJOR’sCompiler


Source Files


Domain SpecificLanguage

Bytecode withEmbedded

Mutants





Integration into the Java Compiler




















MAJOR’s Domain Specific Language// variable declarationlistCOR={&&, ||, ==, !=};

// Define replacement list

BIN(+)<"org"> -> {-,*};

BIN(*)<"org"> -> {/,%};

// Define own operator

myOp{

BIN(&&) -> listCOR;

BIN(||) -> listCOR;

COR;

LVR;

}

// Enable built-in operator AOR

AOR<"org">;

// Enable operator myOp

myOp<"java.lang.System@println">;







BIN(+)<"org"> -> {-,*};

BIN(*)<"org"> -> {/,%};


myOp{

BIN(&&) -> listCOR;

BIN(||) -> listCOR;

COR;

LVR;

}


AOR<"org">;



Specify mutationoperators in detail







BIN(+)<"org"> -> {-,*};

BIN(*)<"org"> -> {/,%};


myOp{

BIN(&&) -> listCOR;

BIN(||) -> listCOR;

COR;

LVR;

}


AOR<"org">;




Define own mutationoperator groups







BIN(+)<"org"> -> {-,*};

BIN(*)<"org"> -> {/,%};


myOp{

BIN(&&) -> listCOR;

BIN(||) -> listCOR;

COR;

LVR;

}


AOR<"org">;




Define own mutationoperator groups

Enable operators fora specific package,

class, or method





Optimized Mutation Analysis Process

1 Embed and compile all mutants2 Run test suite on instrumented program3 Sort tests according to their runtime4 Perform mutation analysis with reordered test suite






















Compilation Efficiency

Mutant Generation and Compilation

1

2

3

4

5

6

7

8

9

10

11

12

0 20000 40000 60000 80000 100000 120000 140000

Co

mp

iler

run

tim

e in

se

co

nd

s

Number of mutants

apache antjfreechart

itextjava pathfindercommons mathcommons lang

numerics4j

Overhead for generating and compiling mutants is negligible





Mutant Generation and Compilation

1

2

3

4

5

6

7

8

9

10

11

12

0 20000 40000 60000 80000 100000 120000 140000

Co

mp

iler

run

tim

e in

se

co

nd

s

Number of mutants

apache antjfreechart

itextjava pathfindercommons mathcommons lang

numerics4j

Overhead for generating and compiling mutants is negligible





Time and Space OverheadApplication Mutants Runtime of test suite Memory consumption

original instrumented original instrumentedwcs wcs+cov

aspectj 406,382 4.3 4.8 5.0 559 813apache ant 60,258 331.0 335.0 346.0 237 293jfreechart 68,782 15.0 18.0 23.0 220 303

itext 124,184 5.1 5.6 6.3 217 325java pathfinder 37,331 17.0 22.0 29.0 182 217commons math 67,895 67.0 83.0 98.0 153 225commons lang 25,783 10.3 11.8 14.8 104 149

numerics4j 5,869 1.2 1.3 1.6 73 90

• Runtime overhead is application dependent• Larger for CPU-bound applications

• Small for I/O-bound applications

• Even for large projects, applicable on commodity workstations









numerics4j 5,869 1.2 1.3 1.6 73 90












numerics4j 5,869 1.2 1.3 1.6 73 90












numerics4j 5,869 1.2 1.3 1.6 73 90












numerics4j 5,869 1.2 1.3 1.6 73 90







Mutation Analysis Efficiency

Evaluating and Improving Mutation Analysis

0

20

40

60

80

100

120

140

160

180

0 20 40 60 80 100

Nu

mb

er

of

kill

ed

mu

tan

ts

Runtime in seconds

optimized order (using coverage information)

random order (using coverage information)

original order (using coverage information)

optimized order (without coverage information)

random order (without coverage information)

original order (without coverage information)

• Mutation analysis is not feasible without coverage information• Reordering the test suite significantly speeds up the process,

especially if runtimes of tests differ by orders of magnitudeKapfhammer Allegheny College





0

20

40

60

80

100

120

140

160

180

0 20 40 60 80 100

Nu

mb

er

of

kill

ed

mu

tan

ts

Runtime in seconds













0

20

40

60

80

100

120

140

160

180

0 20 40 60 80 100

Nu

mb

er

of

kill

ed

mu

tan

ts

Runtime in seconds











Retrospective

Revisiting Practical Mutation Analysis








Retrospective

Revisiting Practical Mutation Analysis





The evidence suggests that MAJOR is “likely to succeed or beeffective” in real-world software testing circumstances




Retrospective

Reviewing MAJOR’s Contributions

MutationAnalysis




Retrospective


MutationAnalysis

Efficiency: MAJOR has ac-ceptable time and space over-heads and scales to large,real-world programs




Retrospective


MutationAnalysis


Usability: MAJOR’s inte-gration into the Java SEcompiler makes it a no-hassle, drop-in tool




Retrospective


MutationAnalysis


Usability: MAJOR’s inte-gration into the Java SEcompiler makes it a no-hassle, drop-in tool

We will release MAJOR as free and open source software




Conclusions and Future Work

Conclusion

Key Concepts and Features:

• Compiler-integrated solution• Conditional mutation with the abstract syntax tree• Furnishes its own domain specific language• Collects and leverages mutation coverage information

Characteristics of MAJOR:

• Fast and scalable technique• Configurable and extensible mutation tool• Enables an optimized workflow for mutation analysis




Conclusions and Future Work

Conclusion

Key Concepts and Features:

• Compiler-integrated solution• Conditional mutation with the abstract syntax tree• Furnishes its own domain specific language• Collects and leverages mutation coverage information

Characteristics of MAJOR:

• Fast and scalable technique• Configurable and extensible mutation tool• Enables an optimized workflow for mutation analysis



Practical Techniques for Improving theEfficiency and Usability of Mutation Analysis

for Java Programs

Gregory M. Kapfhammer

Department of Computer ScienceAllegheny College

http://www.cs.allegheny.edu/∼gkapfham/

Thank you for your attention!I welcome your questions and comments.

http://www.cs.allegheny.edu/~gkapfham/

Documents

Practical Techniques for Improving the Efficiency and Usability of … · algorithm analysis data ... effectiveness empirical genetic performance understanding using approach commercialofftheshelf