Distributed Snapshots: Determining Global States of Distributed Systems Joshua Eberhardt Research...

Distributed Snapshots: Determining Global States

of Distributed Systems

Joshua Eberhardt

Research Paper: Kanianthra Mani Chandy and Leslie Lamport

Background What is a distributed system?

Set of autonomous computers Communication network Software that integrates it into a single entity

Figure 1

Overview Introduction Model of a Distributed System Global-state Detection Algorithm

Motivation Termination

Stability Detection

Overview Introduction Model of a Distributed System Global-state Detection Algorithm

Motivation Termination

Stability Detection

Processes in Distributed Systems Process is an instance of a computer

program being executed. Processes in a distributed system

communicate by sending and receiving messages.

A process can record its own state and the message it sends and receives.

Global States and Processes To determine a global state, a process p

must cooperate with other processes to record their own states and send them to p.

Main problem is to devise an algorithm to record global states.

Global State Detection Problems Let y, be a predicate function defined over

the global states of the a distributed system D. (In other words, y(S) is true or false for a global

state S of D) The predicate y is a stable property of D if

y(S) implies y(S’) for global states S’ of D reachable from S of D

Going Further Many distributed system problems can be

formulated as the general problem of creating an algorithm by which a process in a distributed system can determine whether a stable property y holds.

Examples Deadlock Detection Termination Detection

Structure of Distributed Algorithms

Structured as sequence of phases. Transient Part Stable Part

Stability needs to be detected so that one phase can be terminated and another initiated. Termination of a Computational Phase vs.

Termination of a Computation

Termination Phase The overall problem can be partitioned into

the problems of detecting the termination of one phase and initiating a new phase.

Example of a stable property The kth computational phase has terminated

where k = 1, 2, 3, … Thus we can determine the termination of the kth

phase for any given k.

Overview Introduction Model of a Distributed System Global-state Detection Algorithm

Motivation Termination

Properties Stability Detection

Channels A distributed system consists of a finite set

of processes and a finite set of channels. Properties of channels.

Infinite buffers Error-free Deliver messages in order sent.

Linking the Terms State of a channel

Sequence of messages sent along the channel. Process

Defined by a set of states, including the initial state and a set of events.

Event An atomic action that may change the state of a

process and the state of at most one channel that is incident of the process.

Figure 2

Distributed system with processes p, q, r and channels C1, C2, C3, C4.

Events Can be defined by

Process p in which the event occurs State s of p before the event State s’ of p after the event Channel c whose state is altered by the event Message M sent along channel c

Based on these definitions we can define event e into a 5-tuple. <p, s, s’, M, c>

Expanding to Global States Global state of a distributed system is a set

of component process and channel states. Initially, all of the states are at their initial state,

and as a consequence all of the channels would be the empty sequence.

Occurrences of events may change the global state.

Events and Global States Remember e = <p, s, s’, M, c> We can say e can occur in a global state S:

The state of p in S is s If c is directed towards p, then the state of c in S

is a sequence of messages with M at the head. If c is directed away from p, then the state of c in

S is a sequence of messages with M at the tail.

Going Further If c is directed towards p, then the state of c

in S is a sequence of messages with M at the head. Define a function next where next(S, e) is the

global state immediately after the occurrence of event e in global state S.

The value of next(S, e) is defined only if event e can occur in global state S.

Computational Model Let seq = (ei: 0 < i < n) be a sequence of

events in component processes of a distributed system.

Si+1 = next(Si, ei) for (0 < i < n) where S0 is the initial global state.

We can say seq is a computation of the system iff ei can occur in Si

Example: Single Token Conversation (Deterministic)

Simple distributed system

State Transition Diagram of a Process

Example: Single Token Conversation (Deterministic)

Example: Message Passing (Nondeterministic)

New State Transition Diagrams

Example: Message Passing (Nondeterministic)

More then one way to change the initial global states, all subsequent states would then be different.

Overview Introduction Model of a Distributed System Global-state Detection Algorithm

Motivation Termination

Properties Stability Detection

Motivation How it works:

Each process records its own state and the 2 processes that a channel is incident on cooperate in recording the channel state.

Algorithm is to be superimposed on the underlying computation.

Next example will show how we can record the state of a channel instantaneously. Let c be a channel from p to q.

Single Token Example

Assume the state of process p is recorded as “in p”. Now assume that the global state transitions to “in c”. Suppose the states of c, c’, and q were also recorded in the global state “in c”.

This global state shows that there are two tokens! This shows inconsistency because the state of p was recorded

before p sent the message along c and the state of c is recorded after p sent the message.

Notation Let n be the number of messages sent

along c before p’s state is recorded. Let n’ be the number of messages sent

along c before c’s state is recorded. In our example, this inconsistency shows

that n < n’ or (0 < 1)

Another scenario

Suppose the state of c is recorded in global state “in p”. The system then transitions to the global state “in c” and the states of c’, p

and q are recorded in the global state “in c”. The recorded state shows no tokens in the system! This shows inconsistency when the state of c is recorded before p sends a

message along c and the state of p is recorded after p sends a message along c. Other words n > n’ (1 > 0)

To maintain consistency, n = n’

In Relation to Messages Received Let m be the number of messages received along

c before q’s state is recorded. Let m’ be the number of messages received along

c before c’s state is recorded. To show consistency, m = m’ So for every state the number of messages

received along a channel can’t exceed the number of messages sent along that channel. In other words n > m and n’ > m’.

Bank Example

Bank Example

Bank Example

Important Details to Not e The state of channel c that is recorded must be

the sequence of messages sent along the channel before the sender’s state is recorded.

If n’ = m’, the recorded state of c must be the empty sequence.

If n’ > m’, the recorded state of c must be the (m’ + 1)st…… nth messages sent by p along c.

Markers From these conditions we can devise an

algorithm by which q can record the state of the channel c.

Process p sends a marker after the nth message it sends along c and before sending any messages further along c.

The state of c is the sequence of messages received by q after q records its own state and before q sends the marker along c.

To ensure n > m, q must record its state after receiving a marker along c and before q receives further messages along c.

Algorithm Outline Marker Sending Rule for a Process p

For each channel c, incident on and directed away from p:

p sends a marker along c after p records its state and before p sends further messages along c.

Algorithm Outline Marker Receiving Rule for a Process q

On receiving a marker along a channel c: if (q hasn’t recorded its state)

record qq records c as the empty sequence

elseq records the state of c as the

sequence of messages received along c after q’s state was recorded and before q received the marker along c

Overview Introduction Model of a Distributed System Global-state Detection Algorithm

Motivation Termination

Properties Stability Detection

Termination of the Algorithm The marker receiving and sending rules

guarantee that if a marker is received along every channel, then each process will record its state and the states of all incoming channels.

Finite Time To ensure that the global state recording

algorithm terminates in finite time, each process ensures No marker remains forever in an incident input

channel. It records its state within finite time of initiation of

the algorithm.

Finite Time If process records its state and there is a

channel from p to q, then q will record its state in finite time.

Termination in finite time is ensured if for every process q, q records its state or there is a path from p which records its state to q.

Overview Introduction Model of a Distributed System Global-state Detection Algorithm

Motivation Termination

Stability Detection

Stability Detection Motivation

It is a paradigm for many practical problems, such as distributed deadlock detection.

Can be defined as follows Input: A stable property of y Output: Boolean value definite with the property

(y(Si) definite) or (definite y(Sf)) where S i

represents the global state when initiated and S frepresents the global state when it is terminated.

What this means Input of the algorithm is based on the function of

y. During execution of the algorithm the value y(S)

for any global state S may be determined by a process in the system.

With the output of the algorithm stored in the boolean value definite, we mean that Process p enters and thereafter remains in some special

state to signal that definite = true or false.

Definite value Definite = true

Implies the stable property holds when the algorithm terminates.

Definite = false Implies the stable property doesn’t hold when the

algorithm is initiated.

Solution begin

record a global state S*;definite := y(S*);

end. Correctness of the stability detection algorithm

S* is reachable from Si

Sf is reachable from S* (Theorem) y(S) y(S’) for all S’ reachable from S (definition of stable

property)

Conclusion Distributed systems are applied to many

applications used today, especially in database applications.

Its important to know how each of the processes interact with each other and to know the global state of the system to ensure it is consistent.

References Chandy, K. M. and Lamport L. Distributed

Snapshots: Determining Global States of Distributed Systems

http://www.eecs.ucf.edu/~dcm/Teaching/COT4810-Spring2011/Literature/ChandyAndLamport.pdf

Llewellyn M. Intro to OS: (Distributed Process Management)

http://www.cs.ucf.edu/courses/cop4600/sum2010/distributed%20process%20management%20-%20part%202%20(12).pdf