1
Addressing and Routing in Multi-substrate Overlay
Networks
Jorg Liebeherr
University of Toronto
Joint work with: Majid Valipour and Boris Drazic
2
Substrate (“underlay”) network
Overlay network
What is an overlay anyway?
• An overlay network is a virtual network of nodes and logical links built on top of an existing network
• A virtual link in the overlay corresponds to a path in the underlay
Multi-substrate Network
• Colors indicate type of substrate• Overlapping circles and links indicate bidirectional connectivity
Resulting Topology
4
Approaches to Routing
1. “Vanilla” Ad-hoc routing– Extensively explored
– Not cognizant of substrates
– Scalability limits
2. Single structured overlay– Simple addressing and routing
– Scales to large networks
– Not always viable
3. Hierarchical– Within and between groups of
same color
– Needs address and routing solutions
5
6
Overlays over multiple substrates
• Goal: Improve ability to build an overlay over multiple substrates
• Problem: How to efficiently propagate information about address bindings?
• Solution: Protocol mechanisms for exchanging address bindings Cross Substrate Advertisement (CSA)
Address Bindings in Single-substrate Overlay
• Address binding: [A; SA(A)]
7
C
A DB
Substrate Network
SA(A)SA(C)
SA(B)SA(D)
Overlay nodeidentifier
Substrate address
C
A DB
SA(A) SA(B)
Substrate S3Substrate S2Substrate S1
SAS1(A)
SAS1(C)
SAS2(C)
SAS2(B) SAS3(A)
SAS3(B)
SAS3(D)
Address Bindings in Multi-substrate Overlay
• More complex address binding: [A; SAS1(A), SAS3(A)]
8
Overlay nodeidentifier
Substrate address
9
Cross Substrate Advertising (simplified)- Outgoing -
Overlay SocketNode
Node Adapter
Cross-Substrate Advertisement
Adapter
Interface 1
Interface 2
Interface 3
ID = A
Source: A
SAS1(A) SAS2(A) SAS3(A)
SAS1(A)
SAS2(A)
SAS3(A)
List of substrate addresses
Source: A SAS1(A) SAS2(A) SAS3(A)
10
Cross Substrate Advertising (simplified)- Incoming -
Overlay SocketNode
Node Adapter
Cross-Substrate Advertisement
Adapter
Interface 2
Interface 3
ID Substrate addresses
A SAS1(A), SAS1(A), SAS1(A)
Source: A
Source: A SAS1(A) SAS1(A) SAS1(A)
12
Evaluate Methods for Relayed Address Exchange
Exchange address lists periodically Attach address lists to
protocol messages
Add address list to each message with info on node
Request address list when info is needed
Add preferred address
to each message with info on node
“Push” “Pull”“Push Single”
Gossip Protocol-driven dissemination
Experimental Evaluation
• Local Emulab Testbed• 20 Linux nodes
• Software:• Hypercast with CSA
• Delaunay Triangulation protocol
• Multiple UDP/IP substrates
CPU 2xQuad-Core Xeon 5400 (2 Ghz)
RAM 4G DDR2
Interface 8x1Gbps (4 Intel card, 4 NetFPGA)
Mapping of Nodes to Substrates
• K x K substrates (K+1) x (K+1) regions• Nodes distributed uniformly across regions
K=2
R11
R22
R12 R13
R23
R31
R21
R33R23
Performance metrics
• Stability: Do nodes reach a stable state?– % of nodes satisfying stability criterion for local neighborhood
• Connectivity: Do nodes form a single overlay?– # of partitioned topologies
15
A single stable overlay topology has formed when (1) 100% of nodes are stable; and(2) there is one topology
Stability
K=8 (64 substrates), 648 nodes
Push/Pull
None
Push-single
Gossip
Push-single and gossip
250 ms
500 ms
1000 ms
Connectivity
K=8 (64 substrates), 648 nodes
Num
ber
of
part
ition
s
Num
ber
of
part
ition
s
Protocol overhead
Received Traffic (average per node): K=17 (289 substrates), 2592 nodes
Approaches to Routing
1. “Vanilla” Ad-hoc routing– Extensively explored
– Not cognizant of substrates
– Scalability issues
2. Single structured overlay– Simple addressing and routing
– Scales to large networks
– Not always viable
3. Hierarchical– Within and between groups of
same color
– Needs address and routing solutions
19
Reachability Domain
20
• Substrate: Set of compatible attachment points• Reachability Domain: Maximal set of nodes with path in same substrate
Nodes in the same reachability domain must be able to agree on the scope and identifier of a reachability domain.
Hierarchical routing: First find destination RD, then node within RD
Dynamic Reachability Domains
21
• Number and location of reachability domains are dynamic
New
reachability domain
Location of
reachability domain
changes
Keeping consistent routing tables may not be feasible
Strawman: Landmark Domain Routing
22
• Inspired by Landmark Routing and Compact Routing
• Key concepts: – Use least volatile reachability domains as landmark domains
– Route to destination via landmark domain• Nodes know how to route to landmark domain• Locator is reverse path from landmark domain
– Locator has one entry per traversed reachability domain
Reachability Domain
23
• Route from A K
• K’s Locator: {RD6; H, J}
Step 1: Use next-hop information to reach node in landmark domain
Step 2: Use locator for exit node in landmark domain
Step 3: Follow reverse path to K
Preliminary Evaluation
24
• Random static topology• 100 nodes per substrate with up to 70 reachability domains• Multiple landmark domains
Related Works
• Adhoc Routing– Hierarchies or clustering for scalability
e.g., [Ramanathan & Steenstrup `98] [Pei et.al. `00] [Eriksson et.al.`04]
– Structured overlays for ad-hoc routing e.g., [Viana et.al. ‘04]
• Scalable Routing– Landmark e.g., [Francis `88]
– Compact Routing e.g., [Thorup & Zwick `01]
– Metric Spaces e.g., [Krioukov `10]
• Overlay-based Routing– Identifier-based e.g, [VRR ’06] [IHR ’08] [ROFL ’06]
– Locator-based + directory e.g., [SEATTLE `08] [SpoVNeT ’08]
25
Summary
• Self-organizing overlay protocols over multiple substrates
• Cross-Substrate Advertisement improves ability to form a structured overlay over multiple substrates
• Hierarchical routing between reachability domains
• Landmark domain routing awaits full exploration:– Provable bounds on path stretch– Scalability vs. dynamics