University of Maryland

Efficient Instrumentation forCode Coverage Testing

Mustafa M. TikirJeffrey K. Hollingsworth

University of Maryland

Evaluation of Code Coverage

Measuring code coverage is important to– Identify unexecuted program statements– Verify that each path is taken at least once

Requires extensive instrumentation– Must determine if every statement is

executed

University of Maryland

Instrumentation for Code Coverage

Traditional Approach– Static instrumentation using counters – Instrumentation code remains for entire

execution– Conservative instrumentation of all possibly

needed instrumentation code for all functions

Useless instrumentation wastes time– especially for long running programs such as

servers and enterprise software

Full instrumentation increases setup time

University of Maryland

Our Approach

Insert instrumentation code dynamically– Pre-Instrument all functions at program start– On-Demand instrument at first function call

Periodically remove instrumentation Use dominator trees

– Reduces instrumentation within a function

Use incremental function instrumentation– Insertion of instrumentation on first call– Reduced which function gets instrumented

University of Maryland

Dyninst API

Modify code in a running program Implementations available for

– Alpha,Sparc,Power,Mips and x86 architectures A mutator program

– Generates machine code from high-level code– Transfers machine code to a running mutatee program

Mutatee is the application being instrumented

Base-trampoline contains

– relocated instructions– slots for calling code

Mini-trampoline stores the inserted code

Program

Pre

Post

Relocated Instruction

Base Tramp Mini Tramp

Save Registers

Setup Args

Code SnippetRestore

Registers

University of Maryland

Using Dominator Trees

Definitions:– A dom B if all paths from entry to basic

block B goes through basic block A– A idom B if, for all C, (C != A) and (C dom

B) implies (C dom A)

Fact:– If a basic block, B, is executed

• all basic blocks along the path from B to the root of dominator tree also execute

C

B

A

University of Maryland

Leaf Node Instrumentation

Leaf nodes in dominator tree are instrumented– Coverage of internal nodes will be inferred

Coverage information propagated from leaf nodes to entry

Dominator TreeControl Flow Graph

Entry

13

25

4

Exit

4

2

1

3

Exit

5

Entry

University of Maryland

Non-Leaf Node Instrumentation Leaf node instrumentation is necessary but not sufficient

– Control flow might cause cross edges in dominator tree

Control Flow Graph

Entry

1

32

Exit

Dominator Tree

Entry

2 1

3

We also instrument basic block, A, if A has at least one outgoing edge to basic block, B,

and A does not dominate B

Exit

University of Maryland

Code Coverage Algorithm

At Start– Create CFG and dominator

trees for all functions– Instrument the basic blocks

selected

During Execution– Stop the execution at fixed

time intervals– Delete the executed

instrumentation code

At Termination– Propagate and collect the

coverage information

At Start– Insert breakpoints at each

function entry During Execution

– On breakpoint• Identify the function• Create CFG and

dominator tree• Instrument the basic

blocks selected– At fixed time intervals

delete the executed code At Termination

– Propagate and collect the coverage information

Pre-Instrumentation On-Demand Instrumentation

University of Maryland

Reduction In Instrumentation Points

34-49% with pre-instrumentation 42-79% with on-demand instrumentation

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Ratio

all blocks all blocks OD dominator dominator OD

University of Maryland

SPEC/compress Coverage Curve

Covers 76% in the first 18% of the execution– Most of the basic blocks that will execute are covered

at the beginning of the program– Rest of the run is re-executions

0

10

20

3040

50

60

70

80

0 20 40 60 80 100 120 140 160 180 200Time Interval(sec)

Per

cent

age(

sour

ce li

ne)

University of Maryland

SPEC/compress Execution Time

Purecov is a commercial state of art code coverage tool Our code coverage tools outperform purecov Significant reduction when dynamic code deletion is enabled

– Most of the instrumentation code is deleted at the beginning– Setup time and deletion overhead is insignificant

• A few number of basic blocks to instrument and check

0

1

2

3

4

5

6

7

8

9

10

0 1 2 5 10

15

25

50 0 1 2 5 10

15

25

50 0 1 2 5 10

15

25

50 0 1 2 5 10

15

25

50

PC all blocks all blocks OD dominator dominator OD

Slo

wdow

n R

atio

University of Maryland

PostgreSQL Coverage Curve

Wisconsin benchmark queries– Measure the performance of database systems

• Executes select/join queries repeatedly

0

5

10

15

20

0 10 20 30 40 50 60 70 80Time Interval(sec)

Perc

enta

ge(s

ourc

e lin

e)

University of Maryland

PostgreSQL Execution Time

Using on-demand instrumentation– Our coverage tools outperform purecov almost always

On-demand instrumentation outperforms pre-instrumentation

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

0 1 2 5

10

15

25

50 0 1 2 5

10

15

25

50 0 1 2 5

10

15

25

50 0 1 2 5

10

15

25

50

PC all blocks all blocks OD dominator dominator OD

Slo

wd

ow

n R

atio

Execution Time Setup, Instrumentation and Deletion Time

University of Maryland

Instrumentation Execution Frequency

Overhead of our dynamic code coverage system is bursty Running previously unexecuted code results in the execution

of a significant amount of instrumentation Running previously executed code does not result in the

execution of any instrumentation

PostgreSQL with Wisconsin Benchmark

0

5

10

15

20

25

0 10 20 30 40 50 60 70 80Time(sec)

Covera

ge P

erc

enta

ge

0

1

2

3

4

5

6

7

Execution F

requency

(log10)

University of Maryland

Overall Slowdown Results for our code coverage tool are for 2-second deletion Slowdown using purecov ranges from 1.83 to 19.78 Slowdown using our code coverage tools range

– From 1.002 to 2.6 for on-demand instrumentation– From 1.002 to 4.96 for pre-instrumentation

0

4

8

12

16

20

Slo

wdow

n R

atio

Dominator On-Demand Purecov

University of Maryland

Dyninst Coverage Tool

University of Maryland

Conclusions

Dominator trees– Reduce instrumentation points by 34-49%– Plus on-demand instrumentation reduce

instrumentation points by 42-79%

Combining dominator trees and on-demand instrumentation reduces– Setup time and deletion interval overhead– Runtime overhead by 38-90% compared to

purecov

Dynamic deletion of instrumentation– Computes coverage information faster

University of Maryland

Conclusions (cont)

Code Coverage overhead reduced to about 10%

Code coverage can now be included as part of production code– Information about the execution of

extremely infrequent error cases will be provided

– Reduced overhead for residual testing

Dyninst library + Dyninst Coverage tools – http://www.dyninst.org