View
218
Download
0
Embed Size (px)
Citation preview
CHESSFinding and Reproducing
Heisenbugs
Tom Ball, Sebastian BurckhardtMadan Musuvathi, Shaz Qadeer
Microsoft Research
Interns: Gerard Basler (ETH Zurich),Katie Coons (U. T. Austin),
Tayfun Elmas (Koc University),P. Arumuga Nainar (U. Wisc. Madison),
Iulian Neamtiu (U. Maryland, U.C. Riverside)
Concurrency is HARDRare thread interleavings can result in bugs
These bugs are hard to find, reproduce, and debugHeisenbugs: Observing the bug can “fix” it !
A huge productivity problemDevelopers and testers can spend weeks chasing a single
Heisenbug
Demo
Let’s find a simple concurrency bug
CHESS motivationToday:
concurrency testing == stress testing
Stress increases the interleaving variety, butNot predictable → HeisenbugsNot systematic → poor coverage of interleavings
Don’t stress, use CHESSBasic primitive: Drive a program along an interleaving
of choiceInterleaving can be decided by a program or a userDoing this today is surprisingly hard
Use model checking techniques to systematically enumerate thread interleavings
CHESS architecture
CHESSScheduler
MemoryModelbugs
Monitors
Coverage
Repro
TestingDataraces
Debugging Visualization
UnmanagedProgram
Windows
ManagedProgram
.NET CLR
• Record the interleaving executed• Drive the program along an interleaving
Talk outlineIntroduction
Preemption bounding [PLDI ‘07]Tackling state space explosion
Fair stateless model checking [PLDI ‘08]Handling cycles in states spaces
CHESS architecture details [OSDI ‘08]
Enumerating thread interleavings
x = 1;y = 1;
x = 2;y = 2;
2,1
1,0
0,0
1,1
2,2
2,22,1
2,0
2,12,2
Thread 1 Thread 2
1,2
2,0
2,2
1,1
1,1 1,2
1,0
1,2 1,1
y = 1;
x = 1;
y = 2;
x = 2;
Stateless model checking [Verisoft]Systematically enumerate all paths in a state-space graph
Don’t capture program states Capturing states is extremely hard for large programsState = globals, heap, stack, registers, kernel, filesystem,
other processes, other machines,…
Very effective on acyclic state spaces Termination is guaranteed
Potentially revisits program statesPartial-order reduction alleviates redundant exploration
x = 1; … … … … … y = k;
State space explosionThread 1 Thread n
x = 1; … … … … …y = k;
…
n threads
k steps each
Number of executions = O( nnk )
Exponential in both n and kTypically: n < 10 k > 100
Limits scalability to large programs
Goal: Scale CHESS to large programs (large k)
x = 1;if (p != 0) { x = p->f;}
Preemption bounding Prioritize executions with small number of preemptions
Preemption is a context switch forced by the scheduler Unexpected by the programmere.g. Time-slice expiration
Hypothesis: most concurrency bugs result from few preemptions
x = p->f;}
x = 1;if (p != 0) {
p = 0;
Thread 1 Thread 2
preemption
non-preemption
Polynomial state spaceTerminating program with fixed inputs and deterministic threads
n threads, k steps each, c preemptionsNumber of executions <= nkCc . (n+c)!
= O( (n2k)c. n! )
Exponential in n and c, but not in k
x = 1; … … … … …y = k;
x = 1; … … … … … y = k;
Thread 1 Thread 2
x = 1; … … … …
x = 1; … … …
…y = k;
… …
y = k;
• Choose c preemption points
• Permute n+c atomic blocks
Find lots of bugs with 2 preemptionsProgram Lines of code Bugs
Work Stealing Q 4K 4
CDS 6K 1
CCR 9K 3
ConcRT 16K 4
Dryad 18K 7
APE 19K 4
STM 20K 2
TPL 24K 9
PLINQ 24K 1
Singularity 175K 2
37 (total)
Acknowledgement: testers from PCP team
Good coverage metricWhen CHESS completes search with c preemptionsAny remaining bug requires c+1 or more preemptions
Two preemptions sufficient to reproduced all stress-test failures, reported so far
Talk outlineIntroduction
Preemption bounding [PLDI ‘07]Tackling state space explosion
Fair stateless model checking [PLDI ‘08]Handling cycles in states spaces
CHESS architecture details [OSDI ‘08]
Concurrent programs have cyclic state spaces
SpinlocksNon-blocking algorithmsImplementations of synchronization primitivesPeriodic timers…
L1: while( ! done) { L2: Sleep(); }
M1: done = 1;
Thread 1 Thread 2 ! done L2
! doneL1
done L2
doneL1
A demonic scheduler unrolls any cycle ad-infinitum
! done
done! done
done! done
done
while( ! done){ Sleep();}
done = 1;
Thread 1 Thread 2
! done
Depth bounding
! done
done! done
done! done
done! done
Prune executions beyond a bounded number of steps
Depth bound
Problem 1: Ineffective state coverage
! done
! done
! done
! done
Bound has to be large enough to reach the deepest bug Typically, greater than 100
synchronization operations
Every unrolling of a cycle redundantly explores reachable state space
Depth bound
Problem 2: Cannot find livelocksLivelocks : lack of progress in a program
temp = done;while( ! temp){ Sleep();}
done = 1;
Thread 1 Thread 2
Fair stateless model checking
Make stateless model checking effective on cyclic state spacesEffective state coverageDetect livelocks
Key idea
This test terminates only when the scheduler is fairFairness is assumed by programmers
All cycles in correct programs are unfair A fair cycle is a livelock
while( ! done){ Sleep();}
done = 1;
Thread 1 Thread 2
! done! done
donedone
Key idea
This test terminates only when the scheduler is fairFairness is assumed by programmers
CHESS should only explore fair schedules
while( ! done){ Sleep();}
done = 1;
Thread 1 Thread 2
! done! done
donedone
What notion of fairness?
Weak fairnessForall t :: GF ( enabled(t) scheduled(t) )A thread that remains enabled should eventually be
scheduled
A weakly-fair scheduler will eventually schedule Thread 2Example: round-robin, FIFO wait queues
while( ! done){ Sleep();}
done = 1;
Thread 1 Thread 2
Weak fairness does not suffice
Lock( l );While( ! done){ Unlock( l ); Sleep(); Lock( l );}Unlock( l );
Lock( l );done = 1;Unlock( l );
Thread 1 Thread 2
en = {T1, T2}
T1: Sleep()T2: Lock( l )
en = {T1, T2}
T1: Lock( l )T2: Lock( l )
en = { T1 }
T1: Unlock( l )T2: Lock( l )
en = {T1, T2}
T1: Sleep()T2: Lock( l )
Strong Fairness Forall t :: GF enabled(t) GF scheduled(t) A thread that is enabled infinitely often is scheduled infinitely often
Thread 2 is enabled and competes for the lock infinitely often Example: a round-robin scheduler with priorities [Apt & Olderog ‘83]
Lock( l );While( ! done){ Unlock( l ); Sleep(); Lock( l );}Unlock( l );
Lock( l );done = 1;Unlock( l );
Thread 1 Thread 2
Constructing a strongly fair schedulerA round-robin scheduler is not strongly fair
It is only weakly fair
Extend a round-robin scheduler with priorities [Apt & Olderog ‘83]
CHESS also needs to be demonicCannot generate all fair schedules
There are infinitely many, even for simple programs
It is sufficient to generate enough fair schedules to Explore all states (safety coverage)Explore at least one fair cycle, if any (livelock coverage)
Do it without capturing the program states
Fair stateless model checkingGiven a concurrent program Q and a safety property P
Q does not necessarily have an acyclic state space
Determine Q satisfies P and Q is fair-terminating (livelock-free)
Without capturing program states
(Good) Programs indicate lack of progress
Good Samaritan assumption:Forall threads t : GF scheduled(t) GF yield(t)A thread when scheduled infinitely often yields the processor infinitely
often
Examples of yield:Sleep(), ScheduleThread(), asm {rep nop;}Thread completion
while( ! done){ Sleep();}
done = 1;
Thread 1 Thread 2
Robustness of the Good Samaritan assumptionA violation of the Good Samaritan assumption is a
performance error
Programs are parsimonious in the use of yieldsA Sleep() almost always indicates a lack of progressImplies that the thread is stuck in a state-space cycle
while( ! done){ ;}
done = 1;
Thread 1 Thread 2
Fair demonic scheduler (outline)Maintain a priority-order (a partial-order) on threads
A < B means that A will not be scheduled in a state where B is enabled
Threads get a lower priority only when they yieldScheduler is fully demonic on yield-free paths
A thread loses its priority once it executesRemove all edges t < A when A executes
Four outcomes of the semi-algorithmTerminates without finding any errorsTerminates with a safety violationDiverges with an infinite execution
that violates the GS assumption (a performance error)that is strongly-fair (a livelock)
In practice: detect infinite executions by a very long execution
CoverageTheorem: The algorithm achieves full coverage
if every state is reachable by a yield-free path, and Exists a fair cycle iff exists a fair cycle with at most one
yield per thread
Results: Achieves more coverage faster
With fairness
Without fairness, with depth bound
20 30 40 50 60
States Explored 1726 871 1505 1726 1307 683
PercentageCoverage 100% 50% 87% 100% 76% 40%
Time(secs) 143 97 763 2531 >5000 >5000
Work stealing queue with one stealer
Livelocks in Singularity(both fixed)A thread needlessly burns its CPU quantum in a spin-
loop “it's a bug that we think we have seen in practice, but
that would have been very difficult to find through normal means” [Dean Tribble]
An infinite loop in the Promise implementationManifested as a non-reproducible problem in an existing
stress-testCHESS found the bug in a simple test harness with a
repeatable error-trace
Talk outlineIntroduction
Preemption bounding [PLDI ‘07]Tackling state space explosion
Fair stateless model checking [PLDI ‘08]Handling cycles in states spaces
CHESS architecture details [OSDI ‘08]
CHESS architecture recap
CHESSScheduler
MemoryModelbugs
Monitors
Coverage
Repro
TestingDataraces
Debugging Visualization
UnmanagedProgram
Windows
ManagedProgram
.NET CLR
• Record the interleaving executed• Drive the program along an interleaving
Capture the ‘happens-before’ graph Happens-before graph captures all communication between threads in
a concurrent execution
Abstracts time: For a given input, two executions that result in the same happens-before graph are behaviorally equivalent
x = 1
t = x;
wait(e)
setEvent(e)
Enforce a single-threaded executionHappens-before graph is a partial-order
can be converted to a totally-ordered single threaded execution
Big performance winData-accesses are automatically ordered by synchronization
eventsDon’t need to instrument data-accesses
Cannot explore non-sequentially consistent executions of a programResulting from relaxed memory model of the hardwareSober: A tool that detects the presence of such executions [CAV
‘08]
Directing the executionGiven a happens-before graphBlock the execution of a synchronization if it produces
an edge not in the graph
Need to understand the precise semantics of synchronization operations
ConclusionMessage to concurrency programmers
Think seriously about interleaving coverage
Message to system/PL researchersConcurrency APIs should have a clear specification of the
nondeterminism exposed
Don’t stress, use CHESSCHESS binary available for academic use
http://research.microsoft.com/CHESSCHESS will be shipped for commercial use, very soon
http://msdn.microsoft.com/devlabs
CHESS is extensibleUse CHESS scheduler for concurrency toolsPlug in new search algorithms
Questions
A stress test fails…
CHESS reproduces the bug in 2 mins