Upload
blanche-freeman
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
Experiences with Formal Specifications of Fault-Tolerant
File Systems
Roxana Geambasu (University of Washington)
Andrew Birrell (Microsoft Research)
John MacCormick (Dickinson College)
22
Fault-Tolerant File Systems (FTFSs) FTFSs are crucial components in today’s datacenters They underlie most of what we do on the Web Dependability & correctness of FTFSs are paramount
Google File System (GFS)
Niobe Dynamo
Google Earth
Google Analytics
Amazon services
Web services
3
FTFSs Are Extremely Complex Contain sophisticated protocols for:
replica consistency, recovery (replica addition to compensate for failures), reconfiguration (replica removal due to failure), load balancing, etc.
Hence, FTFS protocols and implementation are hard to get right
Formal Methods (FM) Formal methods have been used extensively to
increase trust in complex systems Formal specification languages are unambiguous Model checking and formal proofs are reliable
However, FTFS designers still rely solely on prose and intuitive reasoning Prose may be ambiguous, inaccurate Intuitive reasoning may be faulty
FTFS Design and Analysis Challenges
Without formal methods, it is hard to: Understand FTFS behavior and semantics
Intuitive reasoning is hard and error-prone
Explore alternative designs Alternative designs may affect semantics in complex ways
Compare various FTFSs Prose is ambiguous and code bases are huge (tens of
thousands of lines of code)
Goal: Convince FTFS Builders to Use FM Previous studies showed how and for what purposes
to use FM for many classes of systems, e.g.: Local/distributed FSs, processor caches, TCP congestion
Our work: Shows how and for what purposes to use FM for
another specific class of important systems:
fault-tolerant file systems
Identifies convenient ways in which FM help in understanding, designing & comparing FTFSs
77
Our Experience We wrote TLA+ specifications for three protocols:
Chain replication (Cornell University) Niobe (Microsoft) GFS (Google)
Our experience shows that FM help solve FTFS challenges:
1. Comparing system mechanisms & tradeoffs
2. Understanding and proving semantics
3. Exploring alternative designs
88
Outline
Specification effort
Experiences with formal specifications for FTFS:
1. Comparing system mechanisms
2. Understanding and proving semantics
3. Exploring alternative designs
Conclusions
Specification Effort
Question: How hard is it to build specifications?
Answer: Moderately precise specifications are reasonably easy to produce
Chain Niobe GFS
Time to write 3 weeks 1 week 2 weeks
1010
Outline
Specification effort
Experiences with formal specifications for FTFS:
1. Comparing system mechanisms
2. Understanding and proving semantics
3. Exploring alternative designs
Conclusions
11
1. Comparing System Mechanisms
Case study: GFS vs. Niobe
From prose, they seemed very different systems GFS: trades some consistency for throughput Niobe: designed for strong consistency
Our TLA+ specifications highlight significant mechanism overlap and also key differences
11
12
Capturing Similarities & Differences
Common
Niobe GFS
(291 lines)
(287 lines)(189 lines)
single-master, primary-secondary replication
More than half of the TLA+ code-base is common Specifications are small due to TLA+ expressiveness
Compare their total sizes to the tens of thousands of LOC of the systems’ implementations
13
Differences Stand Out Clearly in TLA+
1
w
32
w
w
w
ACK ACK
4
Example: Write completion in GFS and Niobe
14
Differences Stand Out Clearly in TLA+
Group reconfiguration
1
w
32
w
w
w
ACK ACK
4 1
32
w
w
w
ACK ACK
w
4
Example: Write completion in GFS and Niobe
15
Understanding Tradeoffs
Smaller latency, but writes may leave
group inconsistent
A write never leaves replica group in
inconsistent state
Example: Write completion in GFS and Niobe
Tradeoff:
16
Lesson: Formalism Helps in Comparison
Formal specifications distill key differences and similarities between systems
Understanding the key differences enables us to understand tradeoffs
1717
Outline
Specification effort
Experiences with formal specifications for FTFS:
1. Comparing system mechanisms
2. Understanding and proving semantics
3. Exploring alternative designs
Conclusions
18
2. Understanding FTFS Consistency Hard to prove consistency models for FTFSs
For weakly consistent systems, it can be even harder
Solution: use refinement mapping1. Reduce system to a really simple model
2. Prove the correctness of the reduction
3. Reason about the SimpleStore
For convenience, we use model-checking instead of full manual proofs at Step 2
System
SimpleStore
consistency model
reduction
consistency model
19
SimpleStores capture only client-visible behaviors and abstract out all protocol mechanisms
SimpleStores are easy to reason about
SimpleStores for the Three FTFSs
Chain_SS Niobe_SS GFS_SS
Chain BlueNiobeChain
reductionreductionreduction
GFS
20
Chain_SS
Chain
reduction
20
Chain’s Consistency Semantics
Using convenient methods, we gained reliable insight into Chain’s consistency model
linearizablelinearizable Proof is straightforward (half a page)
2121
Niobe’s Consistency Semantics
Chain_SS
Chain
reduction
linearizable
linearizable
Niobe_SS
Niobe
reduction
linearizable
linearizable
Similar experience as with Chain Thus, formal methods help in verifying standard
consistency models for strongly-consistent FTFSs
GFS_SS
??
GFS
GFS’ Consistency Semantics Formal methods proved helpful in several ways
An interesting conclusion (details in the paper): Using refinement mappings, we were able to show
that, under a small set of assumptions, GFS has regular-register semantic
GFS_SS
GFSassumptions
reduction
regular register
regular register
well-defined intermediate-level consistency model
23
Lesson: Formalism Helps Understand Semantics
Refinement mappings help in understanding & reliably verifying consistency models of FTFS
They are useful for both strongly consistent and weakly consistent FTFSs
2424
Outline
Specification effort
Experiences with formal specifications for FTFS:
1. Comparing system mechanisms
2. Understanding and proving semantics
3. Exploring alternative designs
Conclusions
2525
3. Exploring Alternative Designs
Exploring alternative designs is much easier using our framework (TLA+ specs, SimpleStores, reductions)
SystemSimpleStore
System model
reduction
26
Case-Study: Changing Niobe’s Design Currently, Niobe’s clients read from primary only Reading from any replica may improve throughput
Design question:
What happens to Niobe if it adopts read-any policy?
ChainGFSassumption
Niobe_SS
linearizable
regular register?
GFS_SS
regular register
Nioberead-any
regular register
Conclusions FTFSs are extremely important in today’s Web We showed how formal methods can help improve
our understanding and trust in FTFSs
Lessons from our experience with three FTFSs: Writing formal specifications is relatively easy Formal methods enable:
Insightful comparison of mechanisms & tradeoffs Reliable verification of consistency properties Convenient investigation of alternative designs
2828
Appendix
29
Related Work FM are extensively used to reason about software [Bickford,
et.al., 96] and hardware [Shimizu, et.al., 02] However, FTFS builders have not adopted them yet By sharing our experience, we hope to convince FTFS builders of the
utility of specifying their systems formally
Using FM to improve understanding and trust in systems: Previous works apply FM to various classes of systems: [Chkliaev, et.al., 00], [Crow, et.al., 98], [Joshi, et.al., 03], [Houston, et.al., 91] The closest works are those looking at distributed FS (AFS, Coda)
[Sivathanu, et.al., 05], [Wing, et.al., 97], [Yang, et.al., 04] We show how to apply them in the specific context of FTFS
Reducing complex systems to simple ones in order to reason about semantics has been used before [Joshi, et.al., 03]
We apply this method to FTFSs
GFS Assumptions
If:
1. A write never crosses chunk boundaries GFS client library offers chunk-level operations
2. A write never goes to a stale replica Implement this assumption using a lease mechanism
Then:GFS_SS
GFSassumptions
reduction
regular register
regular register
Standard Consistency Models Linearizability (Atomic register semantic)
Any client-visible history H generated by the system is equivalent to a legal sequential interleaving S
The sequential interleaving S preserves the real-time ordering of operations from H
Serializability Any client-visible history H generated by the system is equivalent to a
legal sequential interleaving S
Regular register semantic Read not concurrent with any write returns most recently written value Read concurrent with some writes returns either the value of the in-
process writes or the most recently written value
Safe register semantic Read not concurrent with any write returns most recently written value Read concurrent with some writes can return anything
3232
Summary of Contributions Identified a new important class of extremely complex
systems: FTFSs
Showed three aspects of FTFS design & analysis for which FM prove especially valuable Mechanism comparison, semantics understanding, and
design space exploration
Showed how to apply specific FMs to FTFSs Showed how to construct SimpleStores and what can be
learned from them SimpleStores are reusable between systems
We believe that our study, tailored toward FTFSs, can be more relevant to FTFS designers than more general studies
Lessons from Our Experience Building high-level specifications for FTFS is relatively easy
It is also remarkably useful for understanding system
The exercise of writing specifications exposes similarities in seemingly dissimilar systems (GFS, Niobe)
Formal specifications also distill the key design differences
Specifications enable convenient verifications of consistency for both strongly and weakly consistent systems Niobe and Chain are both linearizable GFS can be upgraded to regular register via a clear set of assumptions GFS’ design to read from any replica heavily influences its consistency
Intuition can fail often times Niobe seemed to be reducible to Chain_SS, but actually was not
3434
Chain SimpleStore
write channel
drop(w7)
commit(w5)
SerialDB
reads
read()
read channel
Chain_SS
r2
r1
r3
w7
w6
w5
Requestswrites
Responseswrites
reads
3535
The Temporal Logic of Actions (TLA+) Formalism that combines a temporal logic with a
logic of actions Especially designed for specification of distributed
asynchronous systems TLA+ specifications model the system as a state
machine: Define system variables (state) Model actions that the system can take as state transitions
36
Understanding Tradeoffs
Smaller write latency, but writes may leave group inconsistent
A write never leaves replica group in
inconsistent state
1
32
4 1
32
4
Error
readread
Old value