A Guided Genetic Algorithm for Automated Crash Reproduction

MozhanSoltani AnnibalePanichella Arie vanDeursen

Software systems fail!

2

When Debugging, Crash Reproducible Tests are Required!

3

!

4

STAR

Automated Crash Reproduction

JCHARMING MuCrash

Nayrolles etal.,J.Softw.Evol.andProc.,2016

Chen,andKim,TSE2015 Xuanetal.,ESEC/FSE2015

5

STARJCHARMING MuCrash

Automated Crash Reproduction

ASampleStackTrace

6

Automated Crash Reproduction via JCHARMING

JCHARMINGApproach:1) Deriveapartialmodelofthesystem2) Identifythestatementsthattriggerthecrash

Limitation:Requiresexpensivecomputation!

Nayrolles etal.,J.Softw.Evol.andProc.,2016

7

Automated Crash Reproduction via STAR

Approach:1) Computethecrashpreconditions,usingBSE2) Generateatestthatsatisfiesthepreconditions

Limitation:Maynotscaledueto:• Pathexplosion• LimitationsofSMTsolvers

STAR

Chen,andKim,TSE2015

8

Automated Crash Reproduction via MuCrash

Approach:Mutateexistingtestsuntilonereproducesthecrash

Limitation:Lackofeffectivemutationoperators

MuCrash

Xuanetal.,ESEC/FSE2015

9

Crash Reproduction via Search-Based Software Testing!

10

ASampleStackTraceSearchproblem:Findingacrashreproducible unittest!

Metaheuristicoptimization:AGuided GeneticAlgorithm

LeveragingEVSUITE

Crash Reproduction via EvoCrash!

Guided Genetic Algorithm - GGA

11

EvaluateFitness

Selection GuidedCrossover

InitializePopulation

F=0|| T

GuidedMutation

CreatetheNextGeneration

Re-insertion

Guided Genetic Algorithm - GGA

12

EvaluateFitness

Selection GuidedCrossover

InitializePopulation

F=0|| T

GuidedMutation

CreatetheNextGeneration

Re-insertion

Howisthisaguided geneticalgorithm?

Guided Initialization

13

Eachgeneratedtestshallincludeatleastonecalltothetargetfailingmethod!

ASampleStackTrace

Thetargetfailingmethod

Anidealtestcaseleadsto:1. Theexecutionofline1902. TheIllegalArgumentException beingthrown3. Similarstacktracebeinggenerated

14

Fitness Evaluation

ASampleStackTrace

Targetfailingmethod

Guided Genetic Algorithm - GGA

15

EvaluateFitness

Selection Crossover

InitializePopulation

F=0|| T

Mutation

CreatetheNextGeneration

Whatifthereisnocrashreproducibletestinthepopulation?

16

Single-Point Crossover

Parent#1

Parent#2

Offspring#1

Offspring#2

Thetargetcallislost!

17

Parent#1

Parent#2

Offspring#1

Offspring#2

Thetargetcallislost!

What if the target call is lost?

Callincluded?GuidedCrossover

Taketheparents

Takethechildren

18

Mutation

BeforeMutation

Assignanewobject/value

Removeastatement

Addastatement

19

BeforeMutation

Assignanewstatement(constructor)

Removeastatement

Addastatement

What if the target call is lost?

Callincluded?GuidedMutation

MutationAccepted

Is the test useful for debugging?

20

UsefulnessCriterion:

1

34

2

Ausefultestreveals thebug!

5

TheTargetCrashTraceAUsefulCrashReproducibleTest

Chen,N.,&Kim,S.(2015).Star:Stacktracebasedautomaticcrashreproductionviasymbolicexecution.IEEETransactionsonSoftwareEngineering.

Guided Genetic Algorithm - GGA

21

EvaluateFitness

Selection Crossover

InitializePopulation

F=0|| T

Mutation

CreatetheNextGeneration

HowmanycrashescanEvoCrash successfully reproduce?

HowdoesEvoCrash performcomparedtothestate-of-the-artapproachestocrashreproduction?

Empirical Study

Project NumberofCrashes

ApacheCommonsCollections 12

ApacheAnt 20

ApacheLog4j 18

22

ExceptionTypes

NullPointerException

IllegalArgumentException

ArrayIndexOutOfBoundsException

StringIndexOutOfBoundsException

UnsupportedOperationException

IllegalStateException

ConcurrentModificationException

UsefulReproduction

Reproduced

How many crashes can EvoCrashsuccessfully reproduce?

230

5

10

15

20

ACC ANT LOG

• 82%werereproduced,and• ~80%ofthemwereuseful(inaverage)

Results:

20%

28%

34%

90% 71%

88%

24

An Example: ACC-104

Reproductionrate:4/50Challenge: settingboundedFifoBuffer0intherightstate

ThetestbyEvoCrash forACC-104

ThecrashstacktraceforACC-104

How does EvoCrash perform compared to the state-of-the-art approaches?

250

10

20

30

40

50

Total,JCHARMING,EvoCrash Total,STAR,EvoCrash Total,MuCrash,EvoCrash

EvoCrash out-performedthestate-of-the-art!

75% 100%8

50

58%

82%

1266% 83%

ReproductionrateforothertoolsReproductionrateforEvoCrash

Totalnumberofcases

26

EvoCrash Addresses the Limitations!

BackwardSymbolicExecution:• Pathexplosion• LimitationsofSMT solvers

DirectedModelChecking:Expensivecomputations

TestCaseMutation:Limitedmutationoperators

EvoCrash is Available!

27

www.evocrash.org