Upload
prema
View
43
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Breakpoints and Halting in Distributed Systems. Presented by Abhishek Saxena CS 739 Distributed Systems Spring 2002. References. Detecting Relational Global Predicates in Distributed Systems by Alexander I. Tomlinson and Vijay K. Garg, 1993 - PowerPoint PPT Presentation
Citation preview
Breakpoints and Halting in Distributed Systems
Presented by
Abhishek Saxena
CS 739 Distributed Systems
Spring 2002
2
References
• Detecting Relational Global Predicates in Distributed Systems by Alexander I. Tomlinson and Vijay K. Garg, 1993
• Breakpoints and Halting in Distributed Programs by Barton P. Miller and Jong-Deok Choi, 1992
• Restoring Consistent Global States of Distributed Computations by Goldberg et al., 1991
3
Presentation Layout
• Introduction• Motivation• Halting in Distributed Systems• Detecting Breakpoints for:
• Conjunctive/Disjunctive/Linked Predicates• Relational Predicates
• Applications to Research• Relevance to papers read• Conclusions
4
Introduction
• General problems of:– Halting distributed programs– Detecting breakpoints – Validating resource conflicts– Recording, restoration and replay of program
sequences
5
Motivation
• Why halt?– Interactive debugging– Issues in distributed systems:
• No single global notion of time• Unpredictable communication delays• How to issue instant command to all processes?• Command to simultaneously reach all processes?
6
Halting
• 2 pertinent questions:– How to halt a distributed program?
• Halting Algorithm
– When to halt?• Breakpoint Detection
7
Halting Algorithm
• Extends Chandy & Lamport’s algorithm• Sending rule:
– Increments last_halt_id– Send halt marker containing this value to
outgoing channels
• Receiving rule:– Compare the halt_id with its last_halt_id &
update – Send halt marker like sender
8
Receiving process Q
Process T
Process U
Halt marker
Sending process P
Process R
Process S
Halt markerHalt marker
The Halting Algorithm
Halt marker Halt marker
9
The Halting Algorithm
• Intuitive extension to Chandy & Lamport’s Algorithm[1]
• Leads to a global consistent state since:– Process states same as recorded process
states in [1]– Undelivered messages same as recorded
channels states in [1]
10
Problems with this Algorithm
• Processes that infrequently interact with other computation processes• Long halting time
• Acyclic network connection
P Q
Producer Consumer
Communication Channel
11
A Solution…• Centralized debugger process:
d
qp
Debugger process
12
Problems with this solution
• Communication overheads
• Possible change in execution of program
• Complex to build
13
Detecting Breakpoints
• Breakpoints & Predicates
• Predicate satisfaction = breakpoint detection
• Distributed processes’ system needs: – Simple predicates– Disjunctive predicates– Linked predicates…interesting!– Conjunctive predicates…very interesting!
14
Simple Predicates
• Encapsulate single process behavior
• Detect simple events:– Entered procedure– Message sent / received– Channel created / destroyed– Process created / destroyed
15
Disjunctive predicates
• Form:
DP ::= SP [ U SP ]*
• Satisfied when any SP is satisfied
• Initiate halting when DP is true
16
Linked Predicates
• Specify sequences of events
• Form:
LP ::= DP [ ->DP ]*
• Debugger process sends the LP {DP1->...} to processes involved in DP1
• Upon DP1, strip off DP1 & send stripped LP to processes involved in DP2
17
Process S
Process P
Linked predicates’ implementation
Debugger process
Process Q
Process R
Processes involved in DP1
Processes
involved in DP2
DP1->DP2DP1->DP2DP1->DP2
Start Halting
Process T
DP2DP2
Start halting
Start halting
18
Conjunctive Predicates
• Form:
CP ::= SP [ ∩ SP ]*• Hardest to detect! • No single time reference across machines• Interpretation based on virtual time:
– Consider processes P1, P2 with virtual time axes T1, T2
– Define
SCP = { (t1, t2) | t1ε T1, t2ε T2, SP(t1) ∩ SP(T2) }
19
Conjunctive predicates
• Split SCP into:– Ordered-SCP:
{ (t1, t2) | (t1, t2)ε SCP, ((SP1) i -> (SP2) j) U ((SP2) i ->(SP1) j) }
– Unordered-SCP:{ (t1, t2) | (t1, t2)ε SCP, (t1, t2) € ordered-SCP }
20
Conjunctive Predicates
t11
t12
t13
t21
t22
t23
unordered- SCP pair
ordered-SCP pair
21
Conjunctive Predicates
• Detecting unordered-SCP events difficult
• Requires:– Global information gathering process– Time delay!– Cannot preserve meaningful process states
22
Detecting Relational Global Predicates
• Resource conflict validation problems undetectable by earlier predicate classes
• Form:
( x0 +…+ xn > C )– xi: resource usage at Pi– C: total resource available
• Undecomposable into earlier classes of predicates
23
How to detect such predicates?
• 2 algorithms:– Decentralized: runs concurrently– Centralized: decoupled from the target
program
24
Model & Notation
• Partial ordering on S = { S0, …, Sn } where, Si <= Sj, for 0 <= i,j <= n
• Happens-before relation: “->”
• pred.u.i: Intuitively, is the state just preceding u in process i
• succ.u.i: The state just succeeding u in process i
25
Concurrent States & Intervals
Deterministic event
Non-deterministic event
Local state
P Q
State Interval
Receive Interval
2
3
411
10
9
26
Concurrent Intervals
1, lo1
0, lo0 0, i 0, hi0 KEY
1, j 1, hi1
pred relation
P1
P0
27
Concurrent Intervals
• Intervals (0,i) & (1, j) concurrent iff
KEY exists in P0 or P1 s.t.,
lo0 < i <= hi0 & lo1 < j <= hi1,
where,
the lo0, lo1, hi0, hi1 as defined by the previous diagram
28
Overview of algorithms
• Gather information– What?– How?
• Consider 2 processes P0 & P1
• Gather concurrent interval sequences: – { lo0 to hi0 } at P0 & { lo1 to hi1 } at P1
• Check resource violations at all possible pairs of states in these sequences!!
29
Algorithms contd…
• Representation of
(0, lo0) (0, hi0)
(1, lo1) (1, hi1)
as a 2x2 Matrix clock• Row i of Pi’s matrix clock = Pi’s vector clock• Current interval at Pk = (k, Mk[ , ])• Row k of Mk…pred() of current interval at Pk• Row i<>k…pred.pred() of current interval at Pk
30
Maintaining Matrix Clocks
• Initialize– Initialize matrix to 0– If k=0 or k=1 Mk[k, k] ++
• Send message tagged with Mk[., .] ; Increment Mk[k,k] for k=0 V 1
• Upon message receive update matrix clock; Increment Mk[k,k] ; – Mk[k, ]= diagonal(Mk)
31
Matrix Clock Example
1 00 0
0 00 1
0 00 2
2 12 3
2 10 1
3 10 1
0 0
0 1
2 1
0 1
P0
P1
32
Decentralized Algorithm
• Consider process P0
• Upon mesg receive evaluate lo0, lo1, hi0, hi1
• Find min value of resource(x) at P0
• Send debug mesg (min_x0, lo1, hi1) to P1
• P1 detects the predicate :
(min_x0 + min_x1 > C)
33
Overheads & Complexity at P0
• Message overheads:– (# of receive intervals at P0)* sizeof ( 3
integers)………………..Debug mesgs– Sizeof(4 integers)…………Application mesgs
• Memory:– # intervals at P0; min_x for each interval
• Computation:– (# intervals at P0)*( # debug mesgs sent +
received)
34
Centralized Algorithm
• Checker process runs concurrently or, post-mortem
• Consider the latter: processes P0 & P1– Processes keep trace files containing:
• min_x for each interval• an array of {lo0, lo1, hi0, hi1} for each interval
– Runs a check algorithm• Builds heaps by inserting the min_x values for all
concurrent interval sequences at P0 & P1 • Use these heap-tops to detect the predicate
35
Overheads & Complexity for P0
• Memory:– 4 integers for matrix clock each application process
• Computation:– Monitor local variables– Rest offloaded to checker– O(R0 + M0logM0 + M1logM1)
Where, R0 & M0 = # rec intervals & total intervals at P0
36
Major Practical Problems
• Reduced complexity from exp to O(nlogn) but still…
• Large overheads even for 2 processes
• Lots of messages!
• Lots of memory space!
• Lots of computation!
37
Applications to Research
• Development of distributed debugging environment– Recording of execution sequences– Rollback– Replay– Exploration of new execution scenarios
• Command of mission-control distributed systems
38
Relevance to Papers Read
• The S/Net’s Linda kernel:– Debugging distributed tuple space– Detecting race conditions, deadlocks, probe
effects
• Chandy & Lamport’s paper explores the detection of stable predicates and Garg’s paper explores unstable predicate detection
39
Conclusions
• Distributed debugging still challenging
• No efficient algorithm
• Hard to do away with overheads
• Need for efficient event monitoring & manipulation tools
• Message sequence chart generators
• Program flow analysis for more independent program splitting