A Randomized Dynamic Program Analysis for Detecting Real Deadlocks

A Randomized Dynamic Program Analysis for

Detecting Real Deadlocks

Koushik SenCS 265

What is a Deadlock?• An unintended condition in a shared-memory,

multi-threaded program in which:– a set of threads blocks forever– because each thread in the set waits to acquire a

lock being held by another thread in the set• This work: ignore other causes (e.g., wait/notify)

• Example// thread t1sync (l1) { sync (l2) { … }}

// thread t2sync (l2) { sync (l1) { … }}

l1

t1

l2

t2

Motivation• Today’s concurrent programs are rife with

deadlocks– 6,500/198,000 (~ 3%) of bug reports in

Sun’s bug database at http://bugs.sun.com are deadlocks

• Deadlocks are difficult to detect– Usually triggered non-deterministically, on

specific thread schedules– Fail-stop behavior not guaranteed (some

threads may be deadlocked while others continue to run)

• Fixing other concurrency bugs like races can introduce new deadlocks– Our past experience with reporting races:

developers often ask for deadlock checker

Our Goal• Build a deadlock detection tool that– Scales to very large programs– Finds deadlocks quickly– Has no false positives• Shows a real execution exhibiting a deadlock

Related Work: Program Analysis• Static program analysis (e.g., Engler & Ashcraft

SOSP’03; Williams-Thies-Ernst ECOOP’05, Naik et al. ICSE’09)+ Examines all possible program behavior- Often reports many false positives

• Type systems (e.g., Boyapati-Lee-Rinard OOPSLA’02) - Annotation burden often significant• Model checking (e.g., SPIN, Verisoft, Java Pathfinder) - Does not currently scale beyond few KLOC - Not “directed”• Dynamic program analysis (e.g. Goodlock Havelund et

al., Agrawal et al.)+ Usually reports lesser false positives- Has false negatives

Related Work: Testing• Testing

+ Easy to deploy+ Scales to large programs+ No false positives- Same schedule gets tested many times

No effort to control thread scheduler- Often subtle thread schedules that

exhibit deadlocks are not tested

In Summary• Static and dynamic program analyses

have false positives• Testing is simple– No false positives– But, may miss subtle thread schedules

that result in deadlocks

In Summary• Static and dynamic program analyses

have false positives• Testing is simple– No false positives– But, may miss subtle thread schedules

that result in deadlocks• Can we leverage program analysis to

make testing quickly find real deadlocks?

Our Approach• Active Testing• Phase 1: Use imprecise static or

dynamic program analysis to find “abstract” states where a potential deadlock can happen

• Phase 2: “Direct” testing (or model checking) based on the “abstract” states obtained from phase 1

ExampleThread1foo(o1,o2,true)

Thread2foo(o2,o1,false)

void foo(Object l1, Object l2, boolean flag) {

if(flag) { // Long running computations s1: f1(); s2: f2(); } s3: synchronized(l1){ s4: synchronized(l2){ } }

}

Thread 1 Thread 2Thread1foo(o1,o2,true)




}

Testing

f1()

f2()

Lock(o2)Lock(o1)

Unlock(o1)

Unlock(o2)





}

Testing

Lock(o1)

Lock(o2)

Unlock(o2)

Unlock(o1)

Lock(o2)Lock(o1)

Unlock(o1)

Unlock(o2)

f1()

f2()





}

Lock(o1)

Lock(o2)

Unlock(o2)

Unlock(o1)

Lock(o2)Lock(o1)

Unlock(o1)

Unlock(o2)

f1()

f2()

No deadlock detected

Testing




if(flag) { // Long running computations f1(); f2(); } synchronized(l1){ synchronized(l2){ } }

}

Deadlock Directed Testing

Lock(o2)Lock(o1)

Paused

f1()

f2()





}


Lock(o2)Lock(o1)

Paused

f1()

f2()

Lock(o1)

Lock(o2)Paused





}


Lock(o2)

Lock(o1)

Paused

f1()

f2()

Lock(o1)

Lock(o2)Paused





}


Lock(o2)

Lock(o1)

Paused

f1()

f2()

Lock(o1)

Lock(o2)Paused

Deadlock detected !

Preempting threads• How do we know where to pause a

thread ?

Preempting threads• How do we know where to pause a

thread ?– Use existing static or dynamic analyses to

find potential deadlock cycles• Note that these analyses may report false

deadlock cycles– Use “information” recorded for a deadlock

cycle to decide where to pause a thread– We use a modified version of the Goodlock

algorithm (iGoodlock) [Havelund et al, Agarwal et al]

iGoodlock• We consider dynamic instances of

the following statements• c: Acquire(t,l)• c: Release(t,l)• c: Call(t,m)• c: Release(t,m)• c: o = new (t,o’,T)

iGoodlock• We consider dynamic instances of

the following statements• c: Acquire(t,l)• c: Release(t,l)• c: Call(t,m)• c: Release(t,m)• c: o = new (t,o’,T)

Lock Dependency Relation• D is a subset of (T X 2L X L X C*)• (t,L,l,C) is in the lock dependency

relation if

Lock Dependency Relation• D is a subset of (T X 2L X L X C*)• (t,L,l,C) is in the lock dependency

relation if– thread t acquires lock l while holding the

locks in the set L, and C is the sequence of labels of Acquire statements that were executed by t to acquire the locks in L ∪ {l}.

Lock Dependency Chain(t1,L1,l1,C1), (t2,L2,l2,C2), …, (tm,Lm,lm,Cm)

is a lock dependency chain if


is a lock dependency chain if• for all distinct i, j∈[1, m], ti ≠ tj • for all distinct i, j∈[1, m], li ≠ lj • for all i ∈ [1,m − 1], li ∈ Li+1, • for all distinct i,j ∈ [1,m], Li ∩ Lj = ∅.


is a lock dependency chain if• for all distinct i, j∈[1, m], ti ≠ tj • for all distinct i, j∈[1, m], li ≠ lj • for all i ∈ [1,m − 1], li ∈ Li+1, • for all distinct i,j ∈ [1,m], Li ∩ Lj = ∅.• A chain is a deadlock if

Lock Dependency Chain(t1,L1,l1,C1), (t2,L2,l2,C2), …, (tm,Lm,lm,Cm) is

a lock dependency chain if• for all distinct i, j∈[1, m], ti ≠ tj • for all distinct i, j∈[1, m], li ≠ lj • for all i ∈ [1,m − 1], li ∈ Li+1, • for all distinct i,j ∈ [1,m], Li ∩ Lj = ∅.• A chain is a deadlock if• lm ∈ L1





}

Acquire(o1)

Acquire(o2)

Release(o2)

Release(o1)

Acquire(o2)Acquire(o1)

Release(o1)

Release(o2)

Compute D on this trace

f1()

f2()

Computing lock dependency chains

• Compute Dk where D1 = D• for each d ∈D and τ ∈ Di • if d, τ is a lock dependency chain and

a deadlock cycle then report a potential deadlock

• if d, τ is a lock dependency chain and not a deadlock cycle– add d, τ to Di+1

Preempting threads• T1, o1, T2, o2 are runtime objects• How do we identify them across

executions ?


executions ?– Runtime addresses?


executions ?– Runtime addresses?• No, they change across executions !



– Allocation sites where they were created ?• Yes, but different objects created at the

same allocation site will all be treated as the same object



– Allocation sites where they were created ?• Yes, but different objects created at the

same allocation site will all be treated as the same object

– Can we do better?

Abstractions• Based on k-object sensitivity [Milanova et al]• Based on light-weight execution

indexing [Xin et al]

Abstractions

7 class A { 8 void bar() { 9 for(int j = 0; j < 5; j+

+)10 Object l = new

Object();11 } 12 }

1 class C {2 void foo() {3 A a = new

A();4 a.bar();5 }6 }

13 main() {14 C c = new C();15 for(int i = 0; i < 5; i+

+)16 c.foo();17 }

Abstractions

7 class A { 8 void bar() { 9 for(int j = 0; j < 5; j+

+)10 Object l = new

Object();11 } 12 }

1 class C {2 void foo() {3 A a = new

A();4 a.bar();5 }6 }

13 main() {14 C c = new C();15 for(int i = 0; i < 5; i+

+)16 c.foo();17 }

Abstraction of lastobject created (k-object sensitivity) :

[10,3, 14]

Abstraction of last object created (execution indexing) :[(10,5),(4,1),(16,5)]

Implementation• Implemented in a prototype tool

called DEADLOCKFUZZER for Java• Part of the CALFUZZER framework– Active testing [Sen PLDI 08, Park et al.

FSE 08, Joshi et al. CAV 09] of concurrent programs

• Instrument Java bytecode to – observe events during execution– control scheduler

EvaluationProgram

NameLines Of

CodeAverage Runtime in millisecondsNormal iGoodlock DeadlockFuzze

rcache4j 3,897 2,045 3,409 N.A.sor 17,718 163 396 N.A.hedc 25,024 165 1,668 N.A.jspider 10,252 4,622 5,020 N.A.Jigsaw 160,338 - - -Java Logging 4,248 166 272 493Java Swing 337,291 4,694 9,563 28,052DBCP 27,194 603 1,393 1,393Synchronized Lists

17,633 2,862 3,244 7,070

Synchronized Maps

18,911 2,295 2,596 2898

EvaluationProgram

NameLines

Of Code

# Deadlock Cycles Probability of

Reproduction

iGoodlock Real Reproduced

cache4j 3,897 0 0 N.A. N.A.sor 17,718 0 0 N.A. N.A.hedc 25,024 0 0 N.A. N.A.jspider 10,252 0 0 N.A. N.A.Jigsaw 160,338 283 ≥ 29 29 0.214Java Logging 4,248 3 3 3 1.00

Java Swing 337,291 1 1 1 1.00DBCP 27,194 2 2 2 1.00Synchronized Lists

17,633 27 27 27 0.99

Synchronized Maps

18,911 20 20 20 0.52

Evaluation

Limitations• If DEADLOCKFUZZER does not

reproduce a deadlock, we cannot say if the deadlock is a false positive or not

• Jigsaw– Deadlocks reported by iGoodlock : 283– Deadlocks reproduced by

DEADLOCKFUZZER : 29– Deadlocks confirmed as false positives :

18– Rest : 236

Limitations• If DEADLOCKFUZZER does not

reproduce a deadlock, we cannot say if the deadlock is a false positive or not

• Jigsaw– Deadlocks reported by iGoodlock : 283– Deadlocks reproduced by

DEADLOCKFUZZER : 29– Deadlocks confirmed as false positives :

18– Rest : 236

Cannot say if they are real deadlocks or false warnings

Conclusion• DEADLOCKFUZZER is a practical

deadlock detection tool that– Scales to large programs– Finds deadlock quickly– Finds real deadlocks

• Complements static and dynamic analyses– Automatically confirms some of the real

deadlocks

Documents

A Randomized Dynamic Program Analysis for Detecting Real Deadlocks