CarFast: Achieving Higher Statement Coverage Faster

CARFAST: ACHIEVING HIGHER STATEMENT

COVERAGE FASTER

Sangmin Park, Ishtiaque Hussain, Christoph Csallner, Kunal Taneja, B. M. Mainul Hossain, Mark Grechanik, Chen Fu, Qing Xie

CarFast Implementation Evaluation Conclusion

2

Motivation - Achieving High Coverage

Coverage Degree to which program has been tested Measure of confidence

Widely used in industry Avionics industry standard, DO-254 and DO-178B Automotive industry standard, IEC 61508 Other organizations


3

Motivation - Achieving Coverage Fast

Timeout

Current approaches

Goal: Achieve high coverage faster Achieving high coverage fast is difficult

Complex programs Too many test inputs

(e.g., Renters Insurance Program with 78M customer profiles)


4

High level approach

Observation (study we performed) 80% of statements are covered by 20% of branches

(we call those branches "profitable")

Intuition Cover profitable branches fast leading to achieving high

statement coverage quickly

High level approach Use static analysis to find profitable branches Select inputs that direct program execution towards profitable

branches


5

CarFast – Illustrative Example

void foo (int i1, int i2) {

1: if (i1 == 10) { 2: … // branch 1: 300 statements 3: } else if (i2 == 50) { 4: … // branch 2: 600 statements 5: } else { 6: … // branch 3: 100 statements 7: if (i1==20) { 8: if (i2==30) { … } 9: } 10: }

}

300 stmts

600 stmts

T F

FT100stmts

i1==10

i2==50

i1 = 20 and i2 = 20


6

CarFast – Illustrative Example



}

300 stmts

600 stmts

T F

FT100stmts

i1==10

i2==50

i1 = 20 and i2 = 20

600 stmts

DFS search: up to 10%

Branch 2: up to 70%


7

CarFast – Algorithm



}

300 stmts

600 stmts

T F

FT100stmts

i1==10

i2==50

i1 = 20 and i2 = 20

600 stmts

Step 1:Rank

Branches

Step 2:Select

Initial Input

Step 3:Select

Next Input


8




}

Rank Branch # Stmt

1 2 600

2 1 300

3 3 100

4 … …

Step 1: Rank branches• Counts (transitively) branches

by the number of statements they contain

• Resolves method calls• Ranks branches by statements

300 stmts

600 stmts

T F

FT100stmts

i1==10

i2==50


9




}

Step 2: Select a random input• Selects a random input from input

database

Rank Branch # Stmt

1 2 600

2 1 300

3 3 100

4 … …

i1 i2

5 50

20 20

30 30

40 40

Input 1: i1 = 20 and i2 = 20

300 stmts

600 stmts

T F

FT100stmts

i1==10

i2==50


10




}

Rank Branch # Stmt

1 2 600

2 1 300

3 3 100

4 … …

i1 i2

5 50

20 20

30 30

40 40

Input 1: i1 = 20 and i2 = 20

Step 3: Select next input from trace• Executes the program with the input

to collect path condition• Modifies path condition to cover

higher ranked branches• Queries the condition to database

• Selects random input if there are no satisfying input

300 stmts

600 stmts

T F

FT100stmts

i1==10

i2==50


11




}

Input 1: i1 = 20 and i2 = 20

Rank Branch # Stmt

1 2 600

2 1 300

3 3 100

4 … …

i1 i2

5 50

20 20

30 30

40 40

C: (i1!=10)&&(i2!=50)&&(i1==20)&&(i2!=30)





i1==10

600 stmts

T F

FT100stmts

i1==10

i2==50


12

Rank Branch # Stmt

1 2 600

2 1 300

3 3 100

4 … …

i1 i2

5 50

20 20

30 30

40 40




}C’: (i1!=10)&&(i2==50)

Input 1: i1 = 20 and i2 = 20

C: (i1!=10)&&(i2!=50)&&(i1==20)&&(i2!=30)

Input 2: i1 = 5 and i2 = 50





i1==10

600 stmts

T F

FT100stmts

i1==10

i2==50


13

Implementation

Scalability challenges in large applications: up to 1MLOC Large constraints of size up to 5MB Existing tools run out of memory

Execution Engine Initial tool: Concolic execution engine (Dsc) Solution: DSC-Dumper mode

Uses disk instead of memory Removes memory overhead

Test Input Database Initial tool: MSSQL server 2008 Solution: Constraint-based selector

Uses B+ tree based index Provides API to process queries


14

Experiment – Approaches

Random Testing• Random selection of inputs• Black-box approach

Adaptive Random Testing (ART)• Random selection of

evenly distributed inputs• Black-box approach

DART• Concolic execution

approach• Depth-first path exploration• White-box approach

CarFast• Our approach• Static ranking based path

exploration• White-box approach


15

Experiment – Subject Programs

Challenges in selecting programs Programs with various sizes Programs with complex properties Programs without external dependencies

RugRat program generator [WODA 2012] Stochastic-parse-tree based program generation approach Highly configurable option parameters Used in generating 12 programs from 1KLOC to 1MLOC

Test inputs Each program has up to 20 integer inputs Complete combination of inputs for 20 integers = 10020

Pairwise combination of inputs for 20 integers = 1M


16

Experiment – Setup

Study Protocol For statistical significance, ran 30 times Total time = 4 approaches*12 programs*

30 times*24 hours = 34,560 hours

Baseline coverage = min(covi) where i = {Random, ART, DART, CarFast}

Measurement (to achieve baseline coverage) Number of iterations (1 iteration = 1 selection) Elapsed time


17

Programs BaselineCoverage

Appoaches Iterations (mean)

Elapsed Time (mean)

3 (1.2K) 45%

Random 17.1 522.2

ART 17.8 59.8

DART 693.5 1447.0

CarFast 5.9 571.0

5 (2.1K) 78%

Random 1023.2 3162.5

ART 1615.6 5157.7

CarFast 463.9 20040.9

7 (7.8K) 79%

Random 543.1 1736.8

ART 684.1 2217.6

CarFast 380.0 18829

Experiment – Results

* Complete results are in the paper.

13

• Iterations: ICarFast Iother

• Elapsed Time: ERand ECarFast

• DART doesn't scale

2

• Iterations: ICarFast Iother

• Elapsed Time: ERand ECarFast


18

Future Work

Bottleneck Current: Identified modules causing bottlenecks Future: Improve the runtime of CarFast

Fault-detection ability Current: Does not measure fault-detection ability Future: Investigate fault-detection ability

Other test coverage metrics Current: Used static measure on statements Future: Use static measure on branches


20

Contributions

CarFast The first approach to select inputs for achieving statement coverage fast

ImplementationThe tool scales up to 1MLOC

ExperimentThe study shows limitations in popular testing techniques with statistical significance

Tool, subjects, experimental data are available www.carfast.org

BACKUP SLIDES


22

Related Work

Test-case prioritization Test case prioritization: empirical studies [Elbaum,

2002]

Dynamic symbolic execution DART [Godefroid, 2005]

Hybrid concolic testing [Majundar, 2007]

Heuristics for dynamic test generation [Burnim, 2008]

Search-based testing Fitness-guided path exploration [Xie, 2009]


23

CarFast – Preliminary Study

Study Performed on Apache programs Investigated branches and statements Observed power law in results –

20% of branches contain 80% of statements

Hypothesis Assuming the observation holds,

we can steer execution to coverthose 20% of branches

Technology

CarFast: Achieving Higher Statement Coverage Faster