Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert

Bob-the-Builder vs. Fix-it-Felix

Maintaining Overlays in Dynamic Graphs

Seth Gilbert

What is an overlay?Overlay Networks

Given a collection of (changing) servers:

• Choose a good subset of the edges.




• Subgraph has low degree.

• Subgraph has low diameter.






• Maintain edges as servers come and go.

Can we build it?

Yes we can!

We can fix it!

Bitcoin is broken(So why are you wasting your time here when you could be busy hacking bitcoins?)

What goes wrong?

Bitcoin is a peer-to-peer overlay network• Overlay is used for all communication.• Overlay is assumed to be reliable.

What goes wrong?

Bitcoin is a peer-to-peer overlay network• Overlay is used for all communication.• Overlay is assumed to be reliable.• Nodes arrive and leave all the time.• Malicious/greedy users can partition the

network.

What goes wrong?

Basic overlay maintenance idea:• Connect to a few arbitrary neighbors.• Maintain a fixed number of neighbors.• Accept incoming connection requests.

Details: Eclipse Attacks on Bitcoin’s Peer-to-Peer Network by Heilman, Kendler, Zohar, Goldberg

Bitcoin is a peer-to-peer overlay network• Overlay is used for all communication.• Overlay is assumed to be reliable.• Nodes arrive and leave all the time.• Malicious/greedy users can partition the

network.

Rapid and significant churn• Users constantly arriving and departing.• Network changes continuously.• No stable state.

Malicious / greedy users• Some users may not follow the protocol.• Or all users may be greedy!

Long-lived system

BitcoinWhy is this a hard problem?

I am not going to solve the Bitcoin problem today.

A few existing overlay networks…

ChordKademlia

Pastry Tapestry

Skip+

HSkip+

Patricia Tries

DexXHeal

Forgiving Tree

PGrid

Skipnet

RN Protocol

dHamiltonianCycles

Avatar

Ca-Re-Chord

HyperCubes

HyperRing

Chameleon

Re-Chord

Tiara

Corona

What do we know?Discussion of existing approaches.Is the problem solved?Are the existing solutions practical?

Ongoing workSome ideas that we are thinking about.

Challenge for us to work onTo work on after the workshop is over.

Today’s GoalsA Workshop Talk

Collaborators: Gopal Pandurangan, Peter Robinson, and Amitabh Trahan

Overlay Networks

Overlay NetworksGround Rules

Underlying network:• Collection of nodes.

• Nodes arrive (join). Joining node is connected to someone.

• Nodes leave (fail).

• Precise model TBA.

Overlay NetworksGround Rules

Communication:• Every node has an address.

• Every node has an address book.

• A node can send a message to every node in the address book.

• A node can send an address to another node.

Joe

Sue

JoeSue

< Sue >

Overlay NetworksGoals

Overlay network:• Low degree constant or logarithmic.

• Low diameter logarithmic or polylogarithmic

Note: every existing solution guarantees these properties.


Routable:• There exist short paths…

• … and we can find short paths.

Note: random graphs may not be good!


Other properties:• Good expansion

• Good conductance

Random walks converge quickly

Diameter is small

Overlay NetworksIssues

Churn:• Nodes arrive.

• Nodes leave.

Note: Not considering Byzantine/malicious failures today!


Synchrony vs. Asynchrony:• Synchronous: computation proceeds in rounds.

• Asynchronous: arbitrary message/computation delays

Today: assume computation proceeds in rounds.


Oblivious vs. Adaptive scheduler/adversary:• Oblivious: schedule/arrivals/failures fixed in advance.

• Adaptive: schedule/arrivals/failures depends on execution.

Questions: • Do actions of the algorithm correlate with crashes?

• Is there an attacker using knowledge of the system?

Attacker can crash critical nodes?

Sending many messages can overload a link?

Overlay NetworksMetrics

Rate of churn:• How fast can nodes join and leave?

• What is the maximum rate that an algorithm can tolerate?


Rate of recovery:• What happens when something goes wrong?

• How fast does the algorithm reconstruct a good overlay?


Costs:• Message complexity how many messages per round?

• Communication complexity how many bits of communication?

• Quiescent complexity what happens if changes stop?

• Adaptive complexity how do costs relate to the changes?

Overlay NetworksOverlay Networks





• Maintain edges as servers come and go.

OverlaysA Play in Three Acts

Act I : Fix-it-Felix and the Half-Life-Hobgoblin

Wherein we meet our hero Felix, as he races to keep up with the Half-Life-Hobgoblin, fixing the overlay as fast as it is being destoryed.

Act II : Bob-the-Builder and the Destabilizing-Demon

Wherein we meet our hero Bob, as he counters the Destabilizing-Demon, rebuilding the overlay no matter how badly damaged.

Act III : Here Be Dragons

Wherein we explore the dangerous and unknown path forward and away from the hobgoblins and demons of our despair.

Basic Statistics• Home: Niceland

• Role: Good Guy

• Antagonist: Ralph

• Tool: magic hammer

• Powers: fixing anything

Introductions

Fix-it-FelixI can fix it!

Fix it:• On insert: fix it!

• On departure: fix it!

• On any change: fix it!

Basic philosophy:• Fix changes as soon as

they occur.

• Fix it right then and there.

• Fixes are local and immediate.

Fix-it-Felix Approach

How to cope with changes?I can fix it!

Approach:• Start with good

overlay.

• Bound rate of churn.

• Fix the overlay faster than the churn destroys it.

• Maintain good properties throughout.


How to cope with changes?I can fix it!

Measuring ChurnMetric: per-period-churn

In every round:• At most k nodes join every r

rounds.

• At most k nodes leave every r rounds.Ex: k=1, r=log n At most 1 node joins / leaves every log(n) rounds.

2520151050

arrive arrivearrive

departdepart depart

departdepart

Measuring ChurnHalf-life of a System

Minimum time in which either:

• Number of nodes doubles.

• Number of nodes halves.If half life is H and there are n nodes at time t, then between [t, t+H], there are at least (n/2) and at most 2n nodes in the system.

2520151050

arrive arrive arrive arrive departarrive

departdepartarrive

H = 5n = 6

Measuring ChurnHalf-life of a System

Ideal Theorem:

If the half-life of the system is at least θ(log n), then the overlay is always good.

If half life is H and there are n nodes at time t, then between [t, t+H], there are at least (n/2) and at most 2n nodes in the system.

Minimum time in which either:• Number of nodes doubles.

• Number of nodes halves.

Tolerating ChurnChord

Advantages:• Tolerates high rate of churn.

• Overlay has many good properties.

• Simple to implement.

Disadvantages: • Oblivious adversary

• Fragile

• Limitations on where nodes can join?

• Only supports one topology.

Once the ring is disrupted, all is lost!

Easy to attack, at risk of correlated failures.

[Liben-Nowell, Balakrishnan, Karger, PODC‘02]

Basic overlay:• Ring topology

• Nodes distributed randomly on ring.

• Edges:- successor- predecssor- “fingers”

• Approximates a hypercube.

Tolerating Churn

Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]

Maintenance:

• Ignore “fingers.”

- Only matters for routing/lookup.

- Easy to rebuild.

• Maintain successor/predecessor.

• Ensures that ring remains a ring.

Tolerating Churn


Assume adversary is oblivious:

• Adversary does not know where nodes live on the ring.

• New nodes are inserted “randomly.”

• Nodes are deleted “randomly.”

Tolerating Churn


Think about crashes:

• Assume at most n/2 nodes crash.

• Ring is disconnected?

Tolerating Churn



• Assume at most n/2 nodes crash.

• Maintain connection to log(n) successors.

• With high probability, at least 1 successor survives.

Tolerating Churn


Think about joining:

• Assume at most n/2 nodes join.

• Joining node is attached “randomly.”

• Creates appendages hanging off ring.

• At most O(log n) joining nodes in one place.

Tolerating Churn


Repair mechanism:

• Query successors for their successors.

• Query successors for their predecessor.

• Update local state.

• Notify successors of changes.

Note: I’m being a bit imprecise.

Tolerating Churn


Repair mechanism:

• Query successors for their successors.

• Query successors for their predecessor.

• Update local state.

• Notify successors of changes.

Note: I’m being a bit imprecise.

Tolerating Churn


Beware: what if this node is deleted?

Time analysis:

• Every O(1) steps (in expectation), one insert/delete is resolved.

• Within O(log n) steps, with high probability, all inserts/deletes from the last half-life are resolved.

Tolerating Churn


Analysis ideas:

• Define an almost good state.

• Show that in the almost good state, the overlay has good properties.

• Show that if the half-life length Ω(log n), then the overlay is always almost good.

Tolerating Churn


Tolerating ChurnChord Summary

Advantages:• Tolerates high rate of churn.

• Overlay has many good properties.

• Simple to implement.

Disadvantages: • Oblivious adversary

• Fragile

• Limitations on where nodes can join.


Once the ring is disrupted, all is lost!

Easy to attack, at risk of correlated failures.

*Re-Chord (KKS11) yields O(n log n) stabilization.


• Within O(nc) steps, log(n) consecutive nodes crash.

• If you wait long enough, bad events happen.

• Then the ring is disconnected.

• It will take a long time to fix!

Tolerating Churn

Chord is fragile?

Recent work by Pamela Zave has found bugs/imprecision in original Chord descriptions:

Using lightweight modeling to understand Chord

Why the Chord Ring-Maintenance Protocol is not

Correct How to make Chord correct

Conclusion: Chord works, but only if you really get it right.

Tolerating ChurnChord details are tricky

Approach:• Start with good

overlay.

• Bound rate of churn.

• Fix the overlay faster than the churn destroys it.

• Maintain good properties throughout.


Chord tolerates churn.I can fix it!

Tolerating ChurnAlternate Fix-it-Felix approach

Load-balanced hypercube:• At most O(log n) arrivals/departures per round.

• Basic idea: map O(log n) nodes to each vertex of a hypercube.

• Load-balance among hypercube vertices to cope with churn.

• Grow/shrink hypercube as needed.

[Kuhn, Schmid, Wattenhofer, IPTPS’05]


Load-balanced hypercube:• At most O(log n) arrivals/departures per round.

• Basic idea: map O(log n) nodes to each vertex of a hypercube.

• Load-balance among hypercube vertices to cope with churn.

• Grow/shrink hypercube as needed.

[Kuhn, Schmid, Wattenhofer, IPTPS’05]

Interesting ideas: • Map real nodes to a virtual topology.

• Use load balancing to keep real nodes well distributed.

• Tolerates adaptive (and omniscient) adversary.

Note: (unavoidable) weaker per-round adversarial limit.


Random (dynamic) graph:• Maintain a random constant-degree graph.

• Up to θ(n/log n) churn per round.

• Based on random walks.

[Augustine, Pandurangan, Robinson, Roche, Upfal, FOCS’15]

Disadvantages: • Not routable.

• Oblivious adversary

• Network size remains fixed.

• Limitation on where new nodes can be attached.


XHeal:• Fixes any graph that has deletions (and insertions).

• Carefully replaces missing nodes with small expanders.

• Preserves good properties of original graph:- Bounded stretch: log(n)- Bounded degree-growth

[Pandurangan, Trehan, PODC’11]


XHeal:• Fixes any graph that has deletions (and insertions).

• Carefully replaces missing nodes with small expanders.

• Preserves good properties of original graph:- Bounded stretch: log(n)- Bounded degree-growth

[Pandurangan, Trehan, PODC’11]

Interesting aspects: • One change at a time.

• Local repair.

• Works for any topology/graph.

Fix-it-Felix ApproachMoral of the story:

General ideas:• Local repair.• Simple updates.• Maintain approximate structure.

Several existing solutions:• Oblivious adversary.• Varying rates of churn.• Subtle dependence on the joining model.

Challenges:• Adaptive adversary • Tricky analysis, since failures are ongoing.• Hard to keep costs proportional to changes.

I can fix it!








Basic Statistics• Home: Sunflower

Valley

• Role: Building Contractor

• Antagonist: None.

• Tool: Distributed team of building machinery.

• Powers: Teamwork.

Introductions

Bob-the-Builder Can we build it? Yes we can!

Rebuild good overlay:• On insert: rebuild!

• On departure: rebuild!

• On any change: rebuild!

Basic philosophy:• Assume network is in

some arbitrary topology.

• Build a good overlay.

• Build it.. fast!

Bob–the-Builder Approach

How to cope with changes? Can we build it? Yes we can!

From any initial state:• Build a good overlay.

• Fast construction.

• No ongoing churn.

Self-stabilization:• Arbitrary initial

state corruption.

• Converges to a good state.


How to build an overlay? Can we build it? Yes we can!

Dynamic Graph ModelAssume an arbitrary initial state:

Initially:• Graph is given in an arbitrary connected topology.

• State of the nodes may be corrupted.

Dynamic Graph ModelConstruct a good overlay:

In every round:• Exchange messages with neighbors.

• Adjust edges.

• Improve the overlay.

No further joins/leaves allowed (until overlay is constructed).

Dynamic Graph ModelStabilization:

Eventually:• Overlay is constructed.

• Good properties are guaranteed.

• In good state, joins and leaves may be supported.

From any initial state:• Build a good overlay.

• Fast construction.

• No ongoing churn.

Self-stabilization:• Arbitrary initial

state corruption.

• Converges to a good state.


How to build an overlay? Can we build it? Yes we can!

Self-Stabilizing OverlaysSkip+ Overlay Graphs:

Advantages:• Stabilizes quickly.

• Supports efficient joins and leaves.

• Simple rules.

Disadvantages: • Large message / communication complexity (in the worst

case).

• Large degree during construction (in the worst case).

• Oblivious adversary (for departures).

• Churn??


[Jacob, Richa, Scheideler, Schmid, Taubig, PODC’09]

Skip+ OverlaysClassical Skip List

011010 001000 010000 010001 001111 011


011010 001000 010000 010001 001111 011

Advantages:•Fast search / insert in log(n) time.•Fault-tolerant


011010 001000 010000 010001 001111 011

Disadvantage: congestion•Only one root.•Load is not balanced over nodes.

Skip+ OverlaysSkip Graph

011010 001000 010000 010001 001111 011

Skip+ OverlaysSkip+ Graph

011010 001000 010000 010001 001111 011

Skip+ OverlaysHow to build it?

Overlay guarantees:• Fast searches / routing.

• Fast joins / leaves.

• Small diameter.

• Small degree.

Overlay construction issues: • Initially, diameter of the graph may be large.

• How do nodes find their neighbors efficiently?

• How do nodes sort themselves properly into linked lists?

• Leverage parallelism? Not one insertion at a time!

Skip+ OverlaysSimple trick: pointer doubling

In every round:• Send your entire address book to all your neighbors.

• In every round, your “knowledge diameter” doubles.

• Note: each rounds squares the adjacency matrix.

Within log(n) rounds, graph is a clique.

42 8 12


Once you have a clique:• Delete edges not in the overlay.

• Final graph matches good topology.

Within log(n)+1 rounds, graph is Skip+.

[Example: Skip List]


Bad news:• Degree grows very large: n-1.

• Messages grow very large: θ(n).

• Number of messages per round is large: θ(n2).

Conclusion: fast, but inefficient, overlay construction.

42 8 12

Skip+ OverlayMore efficient construction:

Basic idea:• Minimize expensive doubling steps.• Route nodes directly using existing overlay, if

possible.

Local view:• Each node uses its local view to determine who its

neighbors should be in the final overlay.

Stable edges:• An edge (u,v) is stable if it appears to be in the final

Skip+ overlay according to both the views of u and v.• Otherwise, an edge is temporary.

Skip+ OverlaysSimple rules:

Rule 1: Introduce Friends • Notify stable neighbors of neighbors they should

know about (including yourself).

Rule 2: Forward Temporary Edges (Routing)• Forward a temporary neighbor to a stable neighbor

with the largest shared prefix.

Rule 3a: Introduce All (Pointer Doubling)• If your set of stable neighbors changes (e.g., due to a

change in views), then introduce all your neighbors to each other.

Rule 3b: Linearize• Help organize your neighbors at the same “level” into

a linked list.

• d

Skip+ OverlayHow does it work?

In general:• Guarantees fast O(log n) overlay construction.• Best effort to restrain message size / degree during

construction.

Locally checkable:• Nodes can locally determine whether the overlay is

correct.• If every node thinks it is correct, then it is!

Bad news:• Still can lead to large degree, large messages, and

many messages per round, in the worst case.


Advantages:• Stabilizes quickly.

• Supports efficient joins and leaves.

• Simple rules.

Disadvantages: • Large message / communication complexity (in the worst

case).

• Large degree during construction (in the worst case).

• Oblivious adversary (for departures).

• Churn??



Many improvements, similar ideas:• HSkip+ : Heterogeneous bandwidth

• Corona : Deterministic

• TCF-Skip+: Generalization, local detection analysis, etc.

011010 001000 010000010001 001111 011

Self-Stabilizing OverlaysAlternate Bob-the-Builder approach

Patricia-Tree-like Overlay:• Forms a tree from a weakly connected graph.• Asynchronous.• Communication-efficient:

• Small messages, O(1) per round per nod.• Low contention

[Angluin, Aspnes, Chen, Wu, Yin, SPAA’05]

Disadvantages: • Initially: every node has low degree.

• Builds a tree.

• Later version (SSS’07) builds better overlay.

Bob-the-Builder ApproachMoral of the story:

General ideas:• Deterministic, exact final structure.• Converge to final structure.• Graph square or component merging

to converge fast.

Several existing solutions:• Oblivious adversary.• No joins/leaves during stabilization.

Challenges:• How to keep messages small?• How to avoid graph squaring?• How to tolerate ongoing changes?

Can we build it? Yes we can!








Good overlay:• Small diameter / small degree: O(log n).• Routable / good expansion.

Fast and robust construction:• Rapidly formation from any initial state.• Self-stabilizing.

Churn tolerant:• Tolerates large fractions of nodes joining and leaving

every round.• Oblivious adversary (?).

Efficient construction/maintenance:• Small messages (e.g., size O(log n)).• Message-efficient (e.g., 1 message per node per round).

Big PictureGoals:

Collaborators: Gopal Pandurangan, Peter Robinson, and Amitabh Trahan

OverviewPlan:

Virtual overlays:• Define good topologies.• Map virtual nodes to real servers.

Merging two good overlays:• Start with two good overlays, connected by one link.• Build new good overlay.

Construction algorithm:• Divide-and-conquer.• Merge and collapse components.

Robustness:• Self-stabilization.• Churn.

Virtual OverlaysWhat is a good topology?

Sequence of graphs G1, G2, …, Gn

• Graph Gj has [j, 2j] nodes.

• Graph Gj has O(log j) degree and diameter.

Expandable:• There exists a mapping from each node in Gj to 1 or 2

nodes in G2j.

Random sampling:• Supports a mechanism for randomly sampling nodes.• E.g., fast random walks.

Permutation routing:• Supports efficiently routing permutations.

Note: many (deterministic) expanders satisfy these requirements.

Virtual OverlaysMap virtual topology to real nodes

For each real node:• Assign O(1) virtual nodes to it.

• Connect two real nodes if any of their virtual nodes are connected.

Inspiration: The Forgiving Tree [HST’12], DeX [PRT’14] and [KSW’05]

Virtual Real

OverviewPlan:





MergingCombining two overlay networks

Assume two overlay networks:• Each overlay is a properly mapped virtual topology.

• The two overlays are connected by one edge.

Key challenge: merge the two overlay networks.


Step 1: Grow the virtual topology• Double one overlay (using topology mapping Gn G2n).

• Creates excess virtual nodes.


Step 1: Grow the virtual topology• Double one overlay (using topology mapping Gn G2n).

• Creates excess virtual nodes.

Step 2: Create new edges• Use permutation routing.


Step 3: Distribute excess virtual nodes • Send new virtual nodes on a random walk of old

topology.• (Use random sampling of target topology.)




Problem: only one bridge.



topology.• (Use random sampling of target topology.)• Every success creates a new bridge.• Number of bridges doubles (i.e., exponential growth).




Problem: eventually, hard to find an empty node.


Step 3: Distribute excess virtual nodes • Full nodes send new virtual nodes on random walk of old

topology.

If many nodes are empty, they are easy to find.• Empty nodes send requests on random walk of the old

topology.

If most nodes are full, then there are lots of bridges.

Step 4: Rebalancing and clean up• Balance/reduce the virtual nodes (maybe).• Drop the old topology. (Or keep it as a backup.)

End result: a new instantiation of the virtual topology.


OverviewPlan:





Overlay ConstructionDivide-and-Conquer Algorithm

1. Collections: maintain a collection of overlays.• Initially, each node is its own collection.• Collections are connected by edges between (real) nodes.

2. Matching: find a matching in the graph of collections.

• Pair up collections.

3. Merge: combine pairs of collections.• Merge components.• Combine two components into a single new collection.

4. Repeat: while there is > 1 collection, go to Step 2.

Inspired by [Angluin, Aspnes, Chen, Wu, Yin, SPAA’05]






4. Repeat: while there is > 1 collection, go to Step 2. Problem: matching may be

small!



2. Sparsify: reduce degree of graph of collections.

• Create constant degree graph.• Each node organizes its “children” into a line.




5. Repeat: while there is > 1 collection, go to Step 2.


Initially: connected graph


Sparsify: reduce the degree


Repeat: match and merge









Divide-and-Conquer:• Collection has small diameter O(log n) cost to coordinate.• Collection merging: O(polylog n) cost per merge step.• Number of iterations: O(log n) matchings• Overall: O(polylog n) time to form overlay.

Overlay ConstructionEfficiency Analysis

Merging:• Random walks: O(log n) cost• Bridge doubling: O(log n) iterations• Overall: O(polylog n) time to merge topologies.

OverviewPlan:





Self-StabilizationDetecting Trouble


Local detection:• If the topology is bad, some node can detect it.• Virtual topology is deterministic.• Can verify assignment of virtual nodes to real nodes, since

each real node is assigned 1 or 2 virtual nodes.

Detection diameter:• Assume there is a topology problem.• Detection diameter = max distance of any node from a

node that can detect the problem.• In any expander, detection diameter is O(log n), even if

real graph diameter is much larger!



each real node is assigned 1 or 2 virtual nodes.

[Defined by [Berns, Ghosh, Pemmaraju: SSS’11]

Detection diameter:• Assume there is a topology problem.• Detection diameter = max distance of any node from a

node that can detect the problem.• In any expander, detection diameter is O(log n), even if

real graph diameter is much larger!



each real node is assigned 1 or 2 virtual nodes.Conclusion:

Within O(log n) time, every node learns of a topology error.

Self-stabilizatinoBasic idea:

If a node detects an error:• Flood notification to everyone.• Time: O(log n).

On notification of error:• Restart overlay construction from scratch.• Within O(polylog n) time, new overlay.

Self-stabilizatinoBasic idea:

If a node detects an error:• Flood notification to everyone.• Time: O(log n).

On notification of error:• Restart overlay construction from scratch.• Within O(polylog n) time, new overlay.

Remaining issue: synchronization• Different nodes begin rebuilds at different times.• Components synchronize as they merge.

OverviewPlan:





ChurnDepartures and Failures


What types of failures?• Graceful: execute an exit protocol.• Oblivious: adversary decides failures in advance. • Random: adversary fails nodes at random.• Adaptive: adversary chooses failure on-line.• Connected: adversary never disconnects graph.

Overlay tolerates failures• Expanders are highly fault-tolerant.• For “good” target topologies, overlay maintains good

properties, even if nodes fail. • As long as not too many nodes fail, all is good.


What types of failures?• Graceful: execute an exit protocol.• Oblivious: adversary decides failures in advance. • Random: adversary fails nodes at random.• Adaptive: adversary chooses failure on-line.• Connected: adversary never disconnects graph.

ChurnRepairing failures

Option 1. Local repair • If a node leaves gracefully, hand-off virtual nodes.• If a node crashes, neighbors regenerate virtual nodes.• Rebalance (via random walks).

Conjecture: tolerates small changes.

[as in Kuhn, Schmid, Wattenhofer, IPTPS’05]

Option 2. Periodic Rebuild• Every so often, initiate a complete rebuild.• Assume failures oblivious or random (in some sense).• As long as:

“half-life” > “rebuild time”

then everything continues to work.

ChurnRepairing failures

Option 1. Local repair • If a node leaves gracefully, hand-off virtual nodes.• If a node crashes, neighbors regenerate virtual nodes.• Rebalance (via random walks).

Conjecture: tolerates small changes.

[as in Kuhn, Schmid, Wattenhofer, IPTPS’05]

ChurnKey Question:

Does the construction procedure work when there are ongoing failures?

1. Merging fails if links connecting collections fail:

• Random/oblivious failures not too many links fail / iteration.

2. Collection coordination is more difficult• Can no longer rely on a leader or an aggregation tree in

the collection.• Matching is still doable.• Sparsification is harder.

Joining is easy• Find an extra virtual node.• Route virtual node to new entrant.• Or wait until the next rebuild.

ChurnJoining nodes

How do nodes join?• Random: introduced to a random node.• Oblivious: introduced to a node chosen by the

adversary in advance.• Limited adversarial: introduced to a node, but

adversary cannot introduce more than one at each location.

OverviewPlan:












Wrap-upOverlay Networks

Many approaches:• One zillion

different base topologies.

• Many different repair techniques.

• Real-world implementations.

• Algorithms and theory.

A few existing overlay networks…

ChordKademlia

Pastry Tapestry

Skip+

HSkip+

Patricia Tries

DexXHeal

Forgiving Tree

PGrid

Skipnet

RN Protocol

dHamiltonianCycles

Avatar

Ca-Re-Chord

HyperCubes

HyperRing

Chameleon

Re-Chord

Tiara

Corona

Wrap-upA few things I have learned

Churn analysis is hard:

• Relatively few papers really allow failures to happen at any time.

• Analysis is handwavy, and sometimes wrong (see, e.g., Chord).

• Can we do a better job of understanding / proving results regarding churn?

We can fix it!


Self-stabilization is also hard:

• System can start in any state! Easy to ignore corner cases.

• Still, easier because there are no ongoing failures.

• Tends to be expensive, due to lots of local checking and rebuilding.

Can we build it?

Yes we can!


Integrating both churn and self-stabilization:

• Hard to stabilize if failures are ongoing!

• Competing demands: Self-stabilization

treats every failures as a disaster!

Churn-tolerance tries to ignore/repair small failures.


Communication costs:

• Many existing solutions rely on “graph squaring” to handle worst-case.

• Churn-tolerance is relatively cheap (when it works).

• Self-stabilization tends to be much more (communication) expensive.


No one has any idea how to deal with Byzantine/malicious or greedy participants in a dynamic overlay network!

Exception: see “Brahms: Byzantine resilient random membership sampling” by Bortnikov, Gurevitch, Keidar, Kliot, Shraer

Exception: see “Self-stabilizing and Byzantine-Tolerant Overlay Network” by Dolev, Hoch, and van Renesse


Many interesting ideas.

Many interesting techniques.

No one “right” solution yet.

Can we build it?

Yes we can!

We can fix it!

Documents

Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert