Understanding Distributed Snapshots and Consistent Cuts

Distributed Snapshot Distributed Snapshot

Introduction Introduction Global state Set of local states of all the processes. A distributed snapshot collects the global state of a distributed system. Isn t this easy? Just record the local states of the processes? NO ! Note that you can t freeze the system to read the global state.

One One- -dollar dollar bank bank Let a $1 coin circulate in a network of a million banks. How can someone count the total $ in circulation? If not counted properly, then the count may appear to be anything between $0 and $1M .

Importance of snapshots Importance of snapshots Some examples of distributed snapshot are - deadlock detection - termination detection - counting the total number of messages in transit on a large network - checkpointing and rollback recovery after crash - global predicate computation (like, can the system ever be in this global state?) (useful for distributed debugging)

Consistent cut Consistent cut time p1 p2 p3 Cut 1 is consistent, but Cut 2 is not consistent.

Consistent snapshot Consistent snapshot The set of states immediately following the events (actions) in a consistent cut forms a consistent snapshot of a distributed system. A snapshot that is of practical interest is the most recent one. Let C1 and C2 be two consistent cuts and C1 C2. Then C2 is more recent than C1. Analyze why certain cuts in the one-dollar bank are inconsistent.

Recording Consistent snapshot Recording Consistent snapshot How to record a consistent snapshot? Note that 1. The recording must be non-invasive. 2. Recording must be done on-the-fly. You cannot stop the system.

Chandy Chandy- -Lamport Algorithm Lamport Algorithm Works on a (1) strongly connected graph (2) each channel is FIFO. An initiator initiates the algorithm by sending out a marker ( )

White and red processes White and red processes Initially every process is white. When a process receives a marker, it turns red if it has not already done so. Label each action by a process, and every message sent by a process by the color of that process. So, white action = action by a white process red action = action by a red process white message = message sent by a white process red message = message sent by a red process

Two steps Two steps Step 1. In one atomic action, the initiator (a) Turns red (b) Records its own state (c) sends a marker along all outgoing channels Step 2. Every other process, upon receiving a marker for the first time (and before doing anything else) (a) Turns red (b) Records its own state (c) sends markers along all outgoing channels The algorithm terminates when (1) every process turns red, and (2) Every process has received a marker through each incoming channel.

Why does it work? Why does it work? Lemma 1. No red message is received in a white action.

Why does it work? Why does it work? All white All red SSS Easy conceptualization of the snapshot state Theorem. The global state recorded by Chandy-Lamport algorithm is equivalent to the ideal snapshot state SSS. Hint. A pair of actions (a, b) can be scheduled in any order, if there is no causal order between them, so (a; b) is equivalent to (b; a)

Why does it work? Why does it work? Let an observer observe the following actions: w[i] w[k] r[k] w[j] r[i] w[l] r[j] r[l] w[i] w[k] w[j] r[k] r[i] w[l] r[j] r[l] [Lemma 1] w[i] w[k] w[j] r[k] w[l] r[i] r[j] r[l] [Lemma 1] w[i] w[k] w[j] w[l] r[k] r[i] r[j] r[l] [done!] Recorded state

Example 1: Count the tokens Example 1: Count the tokens Let us verify that Chandy-Lamport snapshot algorithm correctly counts the tokens circulating in the system 2 token token no token C A no token token no token B A no token token no token C 1 3 B Are these consistent cuts? How to account for the channel states? Use sent and received variables for each process.

Example 2: Communicating State Machines Example 2: Communicating State Machines

Something unusual Something unusual Let machine i start Chandy-Lamport snapshot before it has sent M along ch1. Also, let machine j receive the marker after it sends out M along ch2. Observe that the snapshot state is down up M Doesn t this appear strange? This state was never reached during the computation!

Understanding snapshot Understanding snapshot

Understanding snapshot Understanding snapshot The observed state is a feasible state that is reachable from the initial configuration. It may not actually be visited during a specific execution. The final state of the original computation is always reachable from the observed state.

Discussions Discussions What good is a snapshot if that state has never been visited by the system? It is relevant for the detection of stable predicates (Stable predicates are those that once become true, remain true thereafter indefinitely. Examples include program termination, deadlock, token loss etc) It is useful for checkpointing. (Checkpoints are consistent global states that are periodically recorded and saved. Once a system crashes, the administrator rolls the system back to a recent checkpoint, and then restart.)

Distributed Snapshot Distributed Snapshot

Think about these Think about these Global state Set of local states of all the processes. A distributed snapshot collects the global state of a distributed system. Isn t this easy? Note that you can t freeze the system to measure it ! Example problems How many messages are in transit on the internet? Has the distributed computation terminated? How do we compute these?

One One- -dollar dollar bank bank Let a $1 coin circulate in a network of a million banks. How can someone count the total $ in circulation? If not counted properly, then one may wrongly count the total $ in circulation to be one million.

Importance of snapshots Importance of snapshots Some important uses are - deadlock detection - termination detection - checkpointing and rollback recovery after crash - global predicate computation (like, can the system ever be in this global state? - Distributed debugging

Consistent cut Consistent cut A cut is a set of events. It contains at least one event per process that has taken an action. A cut is consistent when the following property holds: Let a, b be two events. Then a consistent cut C (b a) b ? time p1 p2 p3 Cut 1 is consistent, but Cut 2 is not consistent.

Consistent snapshot Consistent snapshot The set of states immediately following the events (actions) in a consistent cut forms a consistent snapshot of a distributed system. A snapshot that is of practical interest is the most recent one. Let C1 and C2 be two consistent cuts and C1 C2. Then C2 is more recent than C1. Analyze why certain cuts in the one-dollar bank are inconsistent.

Recording Consistent snapshot Recording Consistent snapshot How to record a consistent snapshot? Note that 1. The recording must be non-invasive. 2. Recording must be done on-the-fly. You cannot stop the system.

Chandy Chandy- -Lamport Algorithm Lamport Algorithm Works on a (1) strongly connected graph (2) each channel is FIFO. An initiator initiates the algorithm by sending out a marker ( )

White and red processes White and red processes Initially every process is white. When a process receives a marker, it turns red if it has not already done so. Every action by a process, and every message sent by a process gets the color of that process. So, white action = action by a white process red action = action by a red process white message = message sent by a white process red message = message sent by a red process

Two steps Two steps Step 1. In one atomic action, the initiator (a) Turns red (b) Records its own state (c) sends a marker along all outgoing channels Step 2. Every other process, upon receiving a marker for the first time (and before doing anything else) (a) Turns red (b) Records its own state (c) sends markers along all outgoing channels The algorithm terminates when (1) every process turns red, and (2) Every process has received a marker through each incoming channel.

Why does it work? Why does it work? Lemma 1. No red message is received in a white action.

Why does it work? Why does it work? All white All red SSS Easy conceptualization of the snapshot state Theorem. The global state recorded by Chandy-Lamport algorithm is equivalent to the ideal snapshot state SSS. Hint. A pair of actions (a, b) can be scheduled in any order, if there is no causal order between them, so (a; b) is equivalent to (b; a)

Why does it work? Why does it work? Let an observer observe the following actions: w[i] w[k] r[k] w[j] r[i] w[l] r[j] r[l] w[i] w[k] w[j] r[k] r[i] w[l] r[j] r[l] [Lemma 1] w[i] w[k] w[j] r[k] w[l] r[i] r[j] r[l] [Lemma 1] w[i] w[k] w[j] w[l] r[k] r[i] r[j] r[l] [done!] Recorded state

Example 1: Count the tokens Example 1: Count the tokens Let us verify that Chandy-Lamport snapshot algorithm correctly counts the tokens circulating in the system 2 token token no token C A no token token no token B A no token token no token C 1 3 B Are these consistent cuts? How to account for the channel states? Use sent and received variables for each process.

Example 2: Communicating State Machines Example 2: Communicating State Machines

Something unusual Something unusual Let machine i start Chandy-Lamport snapshot before it has sent M along ch1. Also, let machine j receive the marker after it sends out M along ch2. Observe that the snapshot state is down up M Doesn t this appear strange? This state was never reached during the computation!

Understanding snapshot Understanding snapshot

Understanding snapshot Understanding snapshot The observed state is a feasible state that is reachable from the initial configuration. It may not actually be visited during a specific execution. The final state of the original computation is always reachable from the observed state.

Discussions Discussions What good is a snapshot if that state has never been visited by the system? It is relevant for the detection of stable predicates (Stable predicates are those that once become true, remain true thereafter indefinitely. Examples include program termination, deadlock, token loss etc) It is useful for checkpointing. (Checkpoints are consistent global states that are periodically recorded and saved. Once a system crashes, the administrator rolls the system back to a recent checkpoint, and then restart.)

Discussions Discussions What if the channels are not FIFO? Either implement a FIFO channel on top of the non-FIFO channel, or use Lai-Yang algorithm -- it is an adaptation of the Chandy-Lamport algorithm, but does not use any marker. Two major steps: LY1. The initiator records its own state. When it needs to send a message m to another process, it sends a message (m, red). LY2. When a process receives a message (m, red), it records its state if it has not already done so, and then accepts the message m. Q1. Why will it work? Since it guarantees that no red message is received by a white action. Q2. Are there any limitations of this approach? Yes! Until the messages reach all processes, the recording of the snapshot remains incomplete.

Checkpointing Mechanism for (backward) error recovery. Transaction states are periodically stored on stable (incorruptible) storages. Following a failure, the transaction rolls back to the nearest checkpoint retrieved from the stable storage. Independent (unsynchronized) vs. coordinated (synchronized) checkpointing

Classification of checkpointing Coordinated Checkpointing takes only consistent snapshots. This has some overhead. Uncoordinated checkpointing apparently has no overhead, but some efficiency issues. See below To which checkpoint will it roll back to after crash ?

Food for thought Food for thought Distributed snapshot = distributed read (read the state of a distributed system on the fly, without stopping the system). Distributed reset = distributed write (update the state of a distributed system on the fly, without stopping the system) How difficult is distributed reset?

Distributed debugging Distributed debugging (Marzullo and Neiger, 1991) e, VC(e) observer Distributed system

Distributed debugging Distributed debugging

Distributed debugging Distributed debugging Possibly : At least one consistent global state S is reachable from the initial global state, such that (S) = true. Definitely : All computations pass through some consistent global state S such that (S) = true. Never : No computation passes through some consistent global state S such that (S) = true. Definitely Possibly

Examples Examples = x+y =12 (true at S21) Possibly = x+y > 15 (true at S31) Definitely = x=y=5 (true at S40 and S22) Never *Neither S40 nor S22 are consistent states*