Distributed Snapshot

Distributed Snapshot

Think about these

-- How many messages are in transit

on the internet?

-- What is the global state of a distributed

system of N processes?

How do we compute these?

One-dollar bank

0

1

2

(0,1)

(1,2)(2,0)

Let a $1 coin circulate in a network of a million banks.

How can someone count the total $ in circulation? If

not counted “properly,” then one may think the total $

in circulation to be one million.

Importance of snapshots

Major uses in

- deadlock detection

- termination detection

- rollback recovery

- global predicate

computation

Consistent cut

(a ∈ C) ⋀ (b a) ≺ ⇒ b ∈ C

If this is not true, then the cut C is inconsistent

A cut is a set of events. If a cut C is consistent then

time

Consistent snapshot

The set of states immediately following the events

(actions) in a consistent cut forms a consistent

snapshot of a distributed system.

• A snapshot that is of practical interest is the most

recent one. Let C1 and C2 be two consistent cuts

and C1 ⊂ C2. Then C2 is more recent than C1.

• Analyze why certain cuts in the one-dollar bank are

inconsistent.

Consistent snapshot

How to record a consistent snapshot? Note that

1. The recording must be non-invasive.

2. Recording must be done on-the-fly.

You cannot stop the system.

Chandy-Lamport Algorithm

Works on a

(1) strongly connected graph

(2) each channel is FIFO.

An initiator initiates the

algorithm by sending out a

marker ( )

White and red processes

Initially every process is white.

When a process receives a marker,

it turns red if it has not already done so.

Every action by a process, and every

message sent by a process gets the

color of that process.

Two steps

Step 1. In one atomic action, the initiator (a) Turns red (b) Records its own state (c) sends a marker along all outgoing channels

Step 2. Every other process, upon receiving a marker for the first time (and before doing anything else) (a) Turns red (b) Records its own state (c) sends markers along all outgoing channels

The algorithm terminates when (1) every process turns red, and (2) Every process has received a marker through each incoming channel.

Why does it work?

Lemma 1. No red message is received in a white action.

Why does it work?

Theorem. The global state recorded by Chandy-Lamport algorithm is equivalent to the ideal snapshot state SSS.

Hint. A pair of actions (a, b) can be scheduled in any order, if there is no causal order between them, so (a; b) is equivalent to (b; a)

SSSEasy conceptualization of the snapshot state

All white All red

Why does it work?

Let an observer observe the following actions:

w[i] w[k] r[k] w[j] r[i] w[l] r[j] r[l] … w[i] w[k] w[j] r[k] r[i] w[l] r[j] r[l] … [Lemma 1]w[i] w[k] w[j] r[k] w[l] r[i] r[j] r[l] … [Lemma 1]w[i] w[k] w[j] w[l] r[k] r[i] r[j] r[l] … [done!]

Recorded state

Example 1: Count the tokens

Let us verify that Chandy-Lamport snapshot algorithm correctly counts

the tokens circulating in the system

A

B

C

How to account for the channel states? Use sent and received variables for each process.

token no token

token

token

no token

no token

A

B

C

no token

no token

token

Are these consistent cuts?

1

2

3

Example 2: Communicating State Machines

ch1

ch2

i j

up

down

up

state machine

i

state machine

j

send

M

send

M'

down

global state i ch1 j ch2

S0 down φ down φ

1 S up M down φ

2 S up M up M'

3 S down M up φ

receive

M'

receive

M

Something unusual

Let machine i start Chandy-Lamport snapshot before it

has sent M along ch1. Also, let machine j receive the

marker after it sends out M’ along ch2. Observe that

the snapshot state is

down ∅ up M’

Doesn’t this appear strange? This state was never

reached during the computation!

Understanding snapshot

S0

S1

i sends M

j sends M'

j receives M

j sends M'i receives M'

S1'

S2

S2'

i sends M

j receives M

i receives M'

i receives M' j receives M

S3

S0

S3'

recorded

state SSS

Understanding snapshot

The observed state is a feasible state that is reachable

from the initial configuration. It may not actually be visited

during a specific execution.

The final state of the original computation is always

reachable from the observed state.

Discussions

What good is a snapshot if that state has never been visited by the system?

- It is relevant for the detection of stable predicates.

- Useful for checkpointing.

Discussions

What if the channels are not FIFO?

Study how Lai-Yang algorithm works. It does not use any marker

LY1. The initiator records its own state. When it needs to send a

message m to another process, it sends a message (m, red).

LY2. When a process receives a message (m, red), it records its state

if it has not already done so, and then accepts the message m.

Question 1. Why will it work?

Question 1 Are there any limitations of this approach?

Food for thought

Distributed snapshot = distributed read.

Distributed reset = distributed write

How difficult is distributed reset?

Distributed debugging(Marzullo and Neiger, 1991)

observer

Distributed system

e, VC(e)

Distributed debugging

Distributed debugging

Possibly ϕ: At least one consistent global state S is reachable from the initial global state, such that φ(S) = true.

Definitely ϕ: All computations pass through some consistent global state S such that φ(S) = true.

Never ϕ: No computation passes through some consistent global state S such that φ(S) = true.

Definitely ϕ Possibly ϕ ⇒

Examples

ϕ = x+y =12 (true at S21) Possibly ϕ

ϕ = x+y > 15 (true at S32) Definitely ϕ

ϕ = x=y=5 (true at S40 and S22) Never ϕ

*Neither S40 and S22 is a consistent state*

Global State Collection

Global state collection

Some applications

- computing network topology

- termination detection

- deadlock detection

Chandy-Lamport algorithm does a partial job. Each process

generates a fragment of the global state, but these pieces have

to be “stitched together” to form a global state.

A simple exercise

Once the pieces of a consistent

global state become available,

consider collecting the global state via

all-to-all broadcast

At the end, each process

will compute a set V, where

V= {s(i): 0 ≤ i ≤ N-1 }

i

k

j

l

s(i) s(j)

s(k) s(l)

All-to-all broadcast

V.iW.i

V.kW.k

(i,k)

Acts like a “pump”

Assume that the topology is a strongly connected graph

V.jW.j

(j,i)