1

James A. JonesCollege of Computing

Georgia Institute of Technology

Automation to Assist SoftwareDebugging

Supported by Tata Consultancy Services, NSF, NASABoeing Commercial Airplanes

CERCS IAB Workshop April 2007

Overview

TTTTT

TTTTT

Passing Failing

test

2

CERCS IAB Workshop April 2007

Debugging Phases

• Make changes to the program• Test• Observe failure• Locate bugs• Identify possible fixes• Choose best fix• Implement fix

CERCS IAB Workshop April 2007

Debugging Phases

• Make changes to the program• Test• Observe failure• Locate bugs – Fault Localization• Identify possible fixes• Choose best fix• Implement fix

3

CERCS IAB Workshop April 2007

Current Practice

CERCS IAB Workshop April 2007

mid() { int x,y,z,m;1:read(“Enter 3 integers:”,x,y,z);2:m = z;3:if (y<z)4: if (x<y)5: m = y;6: else if (x<z)7: m = y;8:else9: if (x>y)10: m = y;11: else if (x>z)12: m = x;13:print(“Middle number is:”, m); }

Technique usesDynamic information

• statements executed• outcome (pass/fail)

Statistical analysis• computes suspiciousness of each statement

Tarantula: Fault Localization

Intuition: Statements that areprimarily executed by failedtest cases are moresuspicious than statementsthat are primarily executed bypassed test cases

4

CERCS IAB Workshop April 2007

••••

••

•

Tarantula

mid() { int x,y,z,m;1: read(“Enter 3 numbers:”,x,y,z);2: m = z;3: if (y<z)4: if (x<y)5: m = y;6: else if (x<z)7: m = y; // bug8: else9: if (x>y)10: m = y;11: else if (x>z)12: m = x;13: print(“Middle number is:”, m); }

3,3,

5

1,2,

33,

2,1

5,5,

5

5,3,

42,

1,3

P P P P P F

•••••

•

•••

•••

•

•••

••

•

•

••••

•

•

••••

••

•

Pass Status

safe suspicious

CERCS IAB Workshop April 2007

Coloring statements

3,3,

5

1,2,

33,

2,1

5,5,

5

5,3,

42,

1,3

P P P P P F

••••

••

•

•••••

•

•••

•••

•

•••

••

•

•

••••

•

•

••••

••

•

Test Cases

Pass Status

mid() { int x,y,z,m;1: read(“Enter 3 numbers:”,x,y,z);2: m = z;3: if (y<z)4: if (x<y)5: m = y;6: else if (x<z)7: m = y;8: else9: if (x>y)10: m = y;11: else if (x>z)12: m = x;13: print(“Middle number is:”, m); }

safe suspicious

5

CERCS IAB Workshop April 2007

Tarantula

CERCS IAB Workshop April 2007

0 20 40 60 80 100% of program to be examined to find bug

% o

f bug

s

0

20

40

60

80

100

Tarantula on Space (~6000 LOC)

Results

6

CERCS IAB Workshop April 2007

Overview

TTTTT

TTTTT

Passing Failing

test

CERCS IAB Workshop April 2007

t1 t2 t3 t4 t5 t6 t7 t8 t9 t10

3,3,

51,

2,3

3,2,

25,

5,5

1,1,

45,

3,4

P P P P F

••••

••

•

•••••

•

•••

•••

•

•••

••

•

•

••••

••

•

••••

•

•

Pass/fail Status

3,2,

12,

1,3

5,4,

25,

2,6

•••

•••

•

••••

••

•

••••

••

•

•••

•••

•P P F FF

Tarantula: Fault Localization

mid() { int x,y,z,m;1:read(“Enter 3 integers:”,x,y,z);2:m = z;3:if (y<z)4: if (x<y)5: m = y;6: else if (x<z)7: m = y;8:else9: if (x>y)10: m = z;11: else if (x>z)12: m = x;13:print(“Middle number is:”, m); }

//bug;correct m=y

7

CERCS IAB Workshop April 2007

t1 t2 t3 t4 t5 t6 t7 t8 t9 t10

mid() { int x,y,z,m;1:read(“Enter 3 integers:”,x,y,z);2:m = z;3:if (y<z)4: if (x<y)5: m = y;6: else if (x<z)7: m = y;8:else9: if (x>y)10: m = y;11: else if (x>z)12: m = x;13:print(“Middle number is:”, m); }

3,3,

51,

2,3

3,2,

25,

5,5

1,1,

45,

3,4

P P P P F

••••

••

•

•••••

•

•••

•••

•

•••

••

•

•

••••

••

•

••••

•

•

Pass/fail Status

3,2,

12,

1,3

5,4,

25,

2,6

•••

•••

•

••••

••

•

••••

••

•

•••

•••

•P P P FP

Tarantula: Fault Localization

//fixed

CERCS IAB Workshop April 2007

t1 t2 t3 t4 t5 t6 t7 t8 t9 t10

mid() { int x,y,z,m;1:read(“Enter 3 integers:”,x,y,z);2:m = z;3:if (y<z)4: if (x<y)5: m = y;6: else if (x<z)7: m = y;8:else9: if (x>y)10: m = y;11: else if (x>z)12: m = x;13:print(“Middle number is:”, m); }

3,3,

51,

2,3

3,2,

25,

5,5

1,1,

45,

3,4

P P P P F

••••

••

•

•••••

•

•••

•••

•

•••

••

•

•

••••

••

•

••••

•

•

Pass/fail Status

3,2,

12,

1,3

5,4,

25,

2,6

•••

•••

•

••••

••

•

••••

••

•

•••

•••

•P P P FP

Tarantula: Fault Localization

//bug;correct:m=x;

//fixed

8

CERCS IAB Workshop April 2007

••••

••

•

t1 t2 t3 t4 t5 t6 t7 t8 t9 t10

mid() { int x,y,z,m;1:read(“Enter 3 integers:”,x,y,z);2:m = z;3:if (y<z)4: if (x<y)5: m = y;6: else if (x<z)7: m = x;8:else9: if (x>y)10: m = y;11: else if (x>z)12: m = x;13:print(“Middle number is:”, m); }

3,3,

51,

2,3

3,2,

25,

5,5

1,1,

45,

3,4

P P P P P

••••

••

•

•••••

•

•••

•••

•

•••

••

•

•

••••

••

•

••••

•

•

Pass/fail Status

3,2,

12,

1,3

5,4,

25,

2,6

•••

•••

•

•••

•••

•P P P PP

Tarantula: Fault Localization

//fixed

//fixed

••••

••

•

CERCS IAB Workshop April 2007

Debugging Process

PExecute Debug

testsfail P’

alltestspassExecute Debug

testsfail P’’

Pi isfailure-free

Execute …

• Are all failing tests caused by the same fault?• Are all failing tests caused by the same fault?• Can we associate groups of tests with different faults?

• Are all failing tests caused by the same fault?• Can we associate groups of tests with different faults?• Can we search for different faults simultaneously?

9

CERCS IAB Workshop April 2007

t1 t2 t3 t4 t5 t6 t7 t8 t9 t10

3,3,

51,

2,3

3,2,

25,

5,5

1,1,

45,

3,4

P P P P F

••••

••

•

•••••

•

•••

•••

•

•••

••

•

•

••••

••

•

••••

•

•

Pass/fail Status

3,2,

12,

1,3

5,4,

25,

2,6

•••

•••

•

••••

••

•

••••

••

•

•••

•••

•P P F FF

Tarantula: Fault Localization

mid() { int x,y,z,m;1:read(“Enter 3 integers:”,x,y,z);2:m = z;3:if (y<z)4: if (x<y)5: m = y;6: else if (x<z)7: m = y;8:else9: if (x>y)10: m = z;11: else if (x>z)12: m = x;13:print(“Middle number is:”, m); }

CERCS IAB Workshop April 2007

Debugging Process

PExecute Debug

testsfail P’

alltestspassExecute Debug

testsfail P’’

SequentialDebugging

testsfailExecute

P Debug

Debug

P’Execute

alltestspass

ParallelDebugging P’’ is

failure-freeP’ is

failure-free

Execute …

…

10

CERCS IAB Workshop April 2007

Debugging Process

PExecute Debug

testsfail P’

alltestspassExecute Debug

testsfail P’’

SequentialDebugging

testsfailExecute

P Debug

Debug

P’Execute

alltestspass

ParallelDebugging P’’ is

failure-freeP’ is

failure-free

Execute …

…

Potential benefits:• Reduced time to failure-free program

Potential cost:Multiple debuggers +

Overhead to partition test cases

Potential benefits:• Reduced time to failure-free program• Better utilization of developer effort• Fewer tests executed• Early indication of the number of faults• Less “noise” in locating each fault

Crucial problem:• Partitioning failed tests into groups

of similar behavior—focus ondifferent faults

• fault-focusing clusters

CERCS IAB Workshop April 2007

Fault-focusing Clusters—Overview

t06

t07

t09

t08

t10

t08t07

t04

t09

t02

t05

t01 t03

t10

Fault-focusing clusters:• Clusters of failing test cases• Clusters failing in similar way• Each cluster targeting a different fault

Test Cases

11

CERCS IAB Workshop April 2007

Fault-focusing Clusterst07

t09

t08

t10

t06

t04t02

t05

t01 t03

Test Cases

t06

t04t02

t05

t01 t03

Specialized Test Suites

Developer 1

Developer 2

Specialized test suites:Fault-focusing clusters

combined withpassing test cases

Specialized test suites:Fault-focusing clusters

combined withpassing test cases

Give each specializedtest suite to differentdeveloper (debugger)

CERCS IAB Workshop April 2007

mid() { int x,y,z,m;1:read(“Enter 3 integers:”…2:m = z;3:if (y<z)4: if (x<y)5: m = y;6: else if (x<z)7: m = y;8:else9: if (x>y)10: m = z;11: else if (x>z)12: m = x;13:print(“Middle number is:”… }

//bug

//bug

Visualization of Each Result

mid() { int x,y,z,m;1:read(“Enter 3 integers:”…2:m = z;3:if (y<z)4: if (x<y)5: m = y;6: else if (x<z)7: m = y;8:else9: if (x>y)10: m = z;11: else if (x>z)12: m = x;13:print(“Middle number is:”… }

//bug

//bug

12

CERCS IAB Workshop April 2007

Fault-focusing Clusters

ExecutionClustering

FaultLocalization

failed test casesexecution

information

specializedtest suites

fault locationprediction

Clustering Technique 1: Less aggressive

Clustering Technique 2: More aggressive

CERCS IAB Workshop April 2007

D

0

5

10

15

20

25

30

35

40

D

D 36.26046842 31.14062524 26.16479859

S CT1 CT2

FF

0

5

10

15

20

25

30

35

40

FF

FF 36.26046842 18.28856692 20.78618781

S CT1 CT2

• Parallel less expensive even if debugging sequential• CT1 if time to delivery is most important (low salaries, high urgency)• CT2 if developer cost is most important (high salaries, low urgency)

Summary of ResultsD FF

13

CERCS IAB Workshop April 2007

Conclusion• Technique and visualization to provide support

for finding faults• New approach to debugging—parallelized

debugging• Technique to get fault-focused clusters to

create specialized test suites• Benefits, supported by studies

• less time to failure-free program• better utilization of developer effort• less running of the test suite• indication of the number of faults• less “noise” in locating each fault

CERCS IAB Workshop April 2007

DebuggingTarantula: Fault Localization

• Jones, Harrold, Stasko, Workshop on SoftwareVisualization 2001

• Jones, Harrold, Stasko, International Conference onSoftware Engineering 2002

• Jones, Orso, Harrold, Software Visualization 2003• Jones, ICSE 2004 Doctoral Workshop• Jones, Orso, Harrold, Journal of Information

Visualization 2004• Jones, Harrold, Automated Software Engineering

2005• Jones, Bowring, Harrold, International Symposium

on Software Testing and Analysis, 2007