Embed Size (px)
Citation preview
Stochastic Analysis and Improvement of theReliability of DHT-based Multicast
Guang Tan and Stephen A. JarvisDepartment of Computer Science, University of Warwick
Coventry, CV4 7AL, United KingdomEmail:
�gtan,saj � @dcs.warwick.ac.uk
Abstract— This paper investigates the reliability of application-level multicast based on a distributed hash table (DHT) in ahighly dynamic network. Using a node residual lifetime model, wederive the stationary end-to-end delivery ratio of data streamingbetween a pair of nodes in the worst case, and show throughnumerical examples that in a practical DHT network, thisratio can be very low (e.g., less than 50%). Leveraging theproperty of heavy-tailed lifetime distribution, we then considerthree optimizing techniques, namely Senior Member Overlay(SMO), Longer-Lived Neighbor Selection (LNS), and ReliableRoute Selection (RRS), and present quantitative analysis of datadelivery reliability under these schemes. In particular, we discussthe tradeoff between delivery ratio and the load imbalance amongnodes. Simulation experiments are also used to evaluate themulticast performance under practical settings. Our model andanalytic results provide useful tools for reliability analysis forother overlay-based applications (e.g., those involving persistentdata transfers).
I. INTRODUCTION
Overlay multicast [12] is an effective paradigm to providelarge-scale data dissemination over the Internet. There are twobasic approaches to organizing multicast groups. The first is tomake all multicast members self-organize into a group accord-ing to some kind of topology (e.g., tree or mesh); the multicastmembers need to locate upstream nodes and assume linksmaintenance [11] [20]. In the second approach [10] [31] [21],the multicast protocol is layered based on a distributed hashtable (DHT) protocol that supports application-layer routingbetween overlay nodes. The different routes provided by theDHT from receiving nodes to the source node automaticallyform a tree topology. Two main advantages of the DHT-basedapproach are (1) multicast applications can easily exploit theDHT’s routing and failure recovery functions to organize themulticast group, obviating the need to handle network dynam-ics and maintain neighbor sets themselves, and (2) the sameDHT-based overlay (e.g., openDHT [23]) can be shared bymany overlay applications and multicast trees simultaneously.As DHTs have been increasingly deployed and used as abuilding block of distributed systems, developing multicastbased on DHTs can considerably simply software developmentand thus becomes an appealing scheme for multicast. Anumber of projects have been or are being undertaken usingthis technique (e.g., Splitstream [9], RSSDHT [1], MOOD [2],and QStream [23]).
In this paper we study the reliability property of DHT-basedmulticast in the context of low-bit-rate streaming applications,
such as text/voice streaming and distributed white board.Reliability is one of the major concerns for any overlay-basedmulticast protocol. In an overlay network, nodes are highlytransient and the data streaming between two end-points cansuffer frequent interruptions, which may last in the order oftens of seconds [11]. As a result, the nodes may either receiveonly a small proportion of the data or have to heavily relyon some kind of error recovery mechanism. This problemis particularly critical for a DHT-based multicast protocol, asthe DHT routes often pass through some non-multicast-groupnodes, which leads to longer data delivery paths and hencepoorer transfer reliability.
Our analysis begins with a stochastic model for nodelifetime and data delivery over a set of nodes. Specifically, weconsider the distribution of node residual lifetime, which playsa fundamental role in the reliability analysis throughout thepaper. With this result, we then obtain the worst-case stationarydata delivery ratio between the source node and a receivingnode. The numerical examples show that this ratio can be verylow (e.g., less than 50%).
Using the fact that a node’s lifetime generally follows aheavy-tailed distribution [8] [28] [25] [26], which itself impliesthat the longer-lived nodes are likely to be more stable thanshort-lived ones, we then consider three optimization schemes.In the first scheme, called Senior Member Overlay (SMO), thenodes above a certain threshold age are organized into a specialoverlay, which takes the responsibility of all the forwardingtasks in data delivery. The second scheme, called Longer-Lived Neighbor Selection (LNS), leverages the flexibility ofneighbor selection provided by some DHT algorithms to makeevery node choose relatively stable nodes as their neighbors,thus improving the stability of the data delivery paths. In thethird scheme, called Reliable Route Selection (RRS), the DHTalgorithm no longer progresses through the ID space in agreedy manner; instead it will choose stable next hop nodesfrom the available options under the constraint that the pathlength remains unchanged. We examine the data delivery ratiounder these schemes and discuss their implications on the loadimbalance among nodes. To obtain insight into DHT-basedmulticast under more realistic settings, we also conduct sim-ulations to evaluate their performance using practical metrics.
For illustration purposes, we use Chord [27] and Scribe [10]as examples of the DHT and multicast group managementprotocols, respectively. We will also discuss the applicability
of certain techniques to a number of representative DHTalgorithms, including Pastry [24], CAN [22], Kademlia [16]and De Bruijn [18]. Due to space limitation, we do notdiscuss other multicast group management protocols, such asBayeux [31] and the one proposed in [21], as well as theircombinations with the different underlying DHT algorithms,but the analysis would be easily applied to these variations.
We confine our study to applications for which the band-width constraint of nodes is not a big issue. Handling band-width constraints [5] poses major challenges to the formalanalysis and complicates the design of optimization schemes.We leave this as a subject of future research.
The paper proceeds as follows: Section II establishes thestochastic model; based on the model, Section III analyzes thedata delivery ratio of an overlay path under the plain DHT;Section IV introduces and analyzes the three optimizationtechniques; Section V presents the simulation results; Sec-tion VI documents some related work and finally Section VIIconcludes the paper.
II. MODEL AND PERFORMANCE METRICS
We assume a simple multicast application that transfers datain the following manner: in a multicast group, the data issent from the source infinitely at a constant data rate, withoutbuffering data at any node, and no retransmission or recoverymechanism is present in the system.
It has been widely observed that application endpoints’lifetimes (in terms of node uptime, user session time orfile transfer time) often follow a heavy-tailed distribution.Typical applications exhibiting such characteristics include filesharing [8] [28], multimedia streaming [6] [25] [26] [30], etc.We therefore use a shifted Pareto distribution [14] ���������� � � ��� ������������������� �"!#��$%���&��� to model nodes’lifetimes ' . The shape parameter ! is assumed to be greaterthan 2 so that the finite mean ( and variance ) exist. We alsouse the density function *+�,�����-.��!������/� �0� ������� ���1��2 �3�4��1�"!5�6$+� . It is easy to verify that (#7�,�1��! �8� � and)�9:;!���9<�%=>��! �?� ��9+�@! � $+�BA .
Other assumptions are as follows. Nodes enter the system ina Poisson process (PP) [28] at a constant rate C , with node D ’sarrival time at EGF ; during the evolution of the network, nodes’lifetimes follow the same distribution � � �3��� . The overlaynetwork has a very large number of nodes (e.g., one million)and has entered a steady state. We further assume a H -bit( HGIKJ ) DHT identifier and that $ML is much larger than thenumber of nodes in the system.
Throughout the paper, we use delivery ratio as a primarymetric for the reliability of the multicast. It is defined as theproportion of data units successfully received by a node fromthe source [4]. Under the assumed transfer mode, the deliveryratio is approximately the fraction of time during which areceiving node finds that all the forwarding nodes between thesource and itself function normally. We focus on the worst-case stationary delivery ratio between two nodes, which isachieved when the two nodes have a maximum path length N(e.g., $O� for two nodes $+9"P �Q� distance apart). The quantity N is
determined by the two nodes’s IDs and the routing algorithmof the DHT. Assuming a fixed maximum path length allowsus to obtain an exact lower bound for the stationary deliveryratio between two general nodes.
A second metric in our model is the node stress, defined asthe number of children supported by a node. In our model, akey tradeoff is between delivery ratio and the load balancingof nodes. Since the minimum node stress is zero, the maximumnode stress (MNS) of all nodes, which reflects the variation inthe range of individual node stresses, is used as a metric forload balancing. When the number of multicast groups R&� �
,the MNS of a DHT is the per-group MNS times R . UsuallyR is independent of the DHT and multicast protocols, so weoften omit this factor (i.e., assume RS �
) when comparingthe MNSs of various schemes.
Note that in a practical network, the dynamics of nodesmay introduce a substantial number of undesirable non-DHTlinks among the nodes [5], which complicates the analysis ofMNS. We therefore assume that the multicast protocol canautomatically correct these non-DHT links (by, for example,periodically re-establishing the application-layer connectionsaccording to the DHT routing table), so that a node’s stresscan be approximated by the number of DHT nodes pointingto it (i.e., its in-degree in the DHT graph).
III. DATA DELIVERY RATIO IN A PLAIN DHT
In this section we derive the worst-case delivery ratiobetween two nodes in a DHT network. We first examine themean residual lifetime of a randomly chosen node, and thenobtain the stationary delivery ratio using renewal theory.
A. Node Residual Lifetime
Given the current time T , we let random variables U and Vdenote a (randomly chosen) node’s age and residual lifetime,respectively. This is illustrated in Figure 1(a), where the sumof U and V is equal to ' . We first examine the property ofU , and then consider the joint distribution of �3UW�"VX� , whosemarginal distribution will characterize our variable of interestV .
As assumed in the previous section, the nodes enter theoverlay network as a homogeneous Poisson process (PP) withrate C . Let
��Y �3Z��[�"� \]Z_^?T � be the corresponding countingprocess [19] formed by the arrival of all the nodes that haveever entered the system, and
��YW` �3Z����a�Q\]Z_^?T � the countingprocess formed by the arrival of all nodes that are presentat the network at time T . In Figure 1(b), the former processis depicted by a sequence of solid points along the top timeaxis, while the latter process is formed by the sequence ofcircles on the bottom time axis. Indeed,
��Y ` ��Z����a�b\cZd^cT �is a non-homogeneous Poisson process (NPP), as stated by thefollowing lemma. (The proofs of some lemmas and theoremsare provided in our technical report [29].)
Lemma 1: The counting process��Y ` ��Z����"�G\eZ^fT � is a
non-homogeneous Poisson process (NPP) with rate functiong �3Z��h;Ci� �-� � � ��T � Z��"���"�Q\?Z_^]T-j
T
L
T
0
N(t) = PP(tau)
0 T
X Y
L
X: age Y: residual lifetime L: lifetime
N'(t) = NPP(lambda( )) .
(a) (b)
.
Fig. 1. The point processes formed by the arrival of nodes.
Next, we apply the generalized campbell theorem [19]to
Y ` �3Z�� to capture the characteristics of the arrival times(thus the ages) of present nodes. This theorem is restated inLemma 2.
Lemma 2: [19, page 227] Let E 2 �aE 9 � j/j/j �aE � be theevent times in an NPP � g ��� ��� ; let
�2 ��9 � j/j/j/�
� � be � i.i.d.random variables with distribution Pr
� � F�\K� �d��d�3���"���d�3Td� ,where �:�����;��P g �� ����� , and
�2 ��9 �/j j/j[�
� � the orderstatistics of
�2 ��9 � j/j/j �
� � . Then, conditioned onY �3Z�� ����
�@E 2 �aE 9 � j/j/j/�aE � ���f� � 2 � � 9 � j/j j/� � � �[jLemma 2 states that the joint distribution of the arrival times
of the present nodes is in distribution equivalent to that of�2 ��9 � j/j/j/�
� � , which has a marginal distribution which iseasy to manipulate. With this lemma, we can proceed to obtainthe density function of U .
Lemma 3: As T���� , the density function of the age ofa randomly chosen node is given by
*��������h �( =� � � � �3��� A j (1)
Proof: Assume that at time T there are � active nodesin the system, whose arrival times are E `
2 � E`9 �/j j/j/� E
`� , re-spectively. Define � i.i.d. random variable
� F���D� � � $%�/j j/j������and their order statistics as in Lemma 2. Let indicator �����:�represent one when event � occurs and zero otherwise. In anordinary DHT, the selection of a node for general purposessuch as finding its neighbors is independent of the node’sproperties such as arrival time and age, so the arrival timeof a selected node can be seen as equiprobable for all E `F ��D�� �a$1�/j j/j[����� . Thus,
Pr� U �?��� Y ` �3T:���� � "! ��
�#F%$ 2
���3T � E `F �]���&� Y ` �3T:����(',jApplying Lemma 2 to the NPP
��Y ` ��Z����"Z*)?� � and observingthat all
� F ’s are symmetric, we have
Pr� U �K�"� Y ` �3T:� +� � "! ��
�#F,$ 2
���3T � � F �K���-' Pr
� �2 ^]T � � �d �:��T � ����d�3Td� j
Hence,
Pr� U ^?� �d � � �d�3T � ����d�3T:� �.. � �
g �3Z��-�%Z�.P g �3Z����1Z �P = � � � � �3Z��BA/�%Z
�.P = � � � � �3Z��BA/�%Z �which gives
* � �3���h �0� �,� �3��� .P = � � � � �3Z�� A0�%Z j
Letting T1�2� and noting that 43P = � � � � �3Z�� A0�%Z ( wehave *�� �3���d = �:� � � �����BA���(�� which establishes the lemma.
With the above conclusion, we can solve the distribution ofV , the node’s residual lifetime.
Lemma 4: As T+�5� , the density function of the residuallifetime of a randomly chosen node is given by
*�6 �871� �( =� � � � ��71�BABj (2)
Moreover,
= V A� (i9 � )�9$<( j (3)
Notice that * 6 ��7%� * � �3��� , which means that a node’sresidual lifetime is in distribution equivalent to a node’sage. Also from Eq.(3), it can be seen that = V A � = '�Ad��)�9 � (i9 �"�+$<(89 = VX9[A3�+$<(�� � , indicating that, providedthe network model, the mean residual lifetime of participatingnodes is even greater than the node’s mean lifetime. Thissomewhat anti-intuitive result explains the measurement ob-servation in [28]: that in a steady-state network, a majority ofparticipating nodes in the system are long-lived nodes, whilethe remaining short-lived nodes turn over at a high rate. Animportant implication of this fact is that, while a reliability-ignorant multicast protocol may have a poor delivery ratiodue to a few highly transient nodes on the path, the existenceof the many long-lived nodes provides the opportunity toachieve a high delivery ratio without occurring significant loadimbalance, if the underlying DHT were able to provide areliability-aware routing service that avoids passing throughthose unstable nodes.
The above relationship between node lifetime and residuallifetime is particularly stressed in [7], where it is foundthat random selection of replacement from all existing nodesafter some node fails produces surprisingly lower churn thanchoosing replacement from a fixed set of nodes. It should alsobe noted that Lemma 4 achieves the result that has been usedby Leonard et al. [14]. However, the results are obtained indistinct approaches and under different modeling assumptions.In [14], the node arrival/departure is modeled as a renewalprocess and the residual lifetime distribution is immediatelyobtained from existing results of renewal theory. Their modelrelies on an important assumption: that the probability that anewly arriving node finds a neighbor at any point within thatneighbor’s lifespan is equally likely. Unfortunately, it is notclear under what circumstances this assumption holds, or how
it can be interpreted in a more plausible way. Also in theirsimulations, they assume that a leaving node is immediatelyreplaced by a fresh node with the same lifetime distribution,an arrival pattern yet to be justified. In contrast, we assumea Poisson arrival pattern which has been verified by previousempirical studies [28] [3]. More importantly, our modelingreveals more details of the stochastic properties of a node’sage and residual lifetime, which facilitate the analysis of datadelivery reliability in more complicated contexts, as will beshown later in the paper.
B. Delivery Ratio
Consider a data delivery path from the source node � P tosome receiving node �����N I;J � � . We assume that the twonodes have a maximum path length of N . Between the twonodes is a sequence of forwarding nodes
� � 2 ��� 9 �/j j/j�� � ��2 � .When a forwarding node � F on the path fails (departs), itschild node � F � 2 will try to find a substitute for � F and re-establish a new path
� � P ��� ` 2 � j/j/j ��� `F ��� F � 2 �/j j/j[��� � � , where, inmost instances, � `� � � � $%�/j j/j��"D �;� � is a node succeeding� � on the Chord ring. (It is possible that the length of thenew path becomes less than N . We again assume the worstcase where the path length remains unchanged.) We make aminor modification to the Scribe protocol so that the nodes� 2 ��� 9 � j/j j/��� F ��2 need not to be changed: when �1F fails, itschild � F � 2 finds a substitute � `F and requests that � `F directlyconnects to its original grandparent �1F �i2 , the original pathfrom � F ��2 to � P being re-used. Now, the path
� � P ��� 2 �/j j/j/����� �will have only one forwarding node replaced when � F fails.An important consequence of this is that the replacement offorwarding nodes on the path becomes independent of eachother, and so the modeling analysis can be greatly simplified.From the viewpoint of the system, this change obviates theneed to destroy the previously established path and thusreduces the communication cost. In addition, this modificationis easy to implement – recall that a Chord node alreadymaintains a successor list for each of its neighbors for thepurpose of fault tolerance [27].
Now, each forwarding node on the path has two states:normal state and failure state. The normal period is equivalentto the residual lifetime V of a randomly chosen node amongthe present nodes in the network. The failure period includesfailure detection and finding a substitute node, which weassume takes a random time � , called the fixing time. We cantherefore treat a forwarding node’s evolution as an alternatingrenewal process [19], and using Smith’s theorem, we obtainthe stationary probability of a forwarding node being found ina normal state + = V A��%�� = V A � = � A3� . Since all forwardingnodes on the path are independent of each other in their ownrenewal processes, the joint probability of their status beingnormal simultaneously is simply � �i2 , which corresponds tothe probability of the receiving node not being in starvation.Thus the following result concerning the delivery ratio seenby the receiver node holds.
Theorem 1: In a DHT network, the worst-case stationarydelivery ratio between two nodes that are at most N hops apart
is given by� � � = V A = VA � = �:A�� � �i2 � (i9 � )�9
( 9 � ) 9 � $<(( = �:A�� � ��2 j(4)
Theorem 1 provides an estimate for the worst-case deliveryratio regardless of the actual node lifetime distributions (suchas Pareto, lognormal, Weibull, etc.). As an example, Table Ishows the worst-case delivery ratios between two nodes withPareto node lifetime distribution for varying maximum pathlengths and mean fixing time = �:A s. As we often do through-out the paper, the parameters ! and � are set to � and
�respectively, such that the mean lifetime ( .�1j�� hours andthe mean residual lifetime = V A- �
hour. In the table, theworst-case delivery ratio drops noticeably as the maximumpath lengths or mean fixing time increases. For example, forN���+� and = �:A� �
minute, the worst-case delivery ratio isonly slightly higher than 60%, a level far from satisfactory formany applications.
Mean fixing Maximum path length �time ��� ��� 10 20 3030 seconds 92.8% 85.4% 78.6%1 minute 86.2% 73.0% 61.9%2 minutes 74.4% 53.6% 38.6%
TABLE I
WORST-CASE DELIVERY RATIO FOR VARYING � S AND ��� ��� S.
C. Maximum Node Stress
In a DHT network, a node can be responsible for at most�@$ML �������Q� �!#" ���$�%!&"'�%!&"���� IDs with high probability (whp)(balls-in-bins model [17]); each of these IDs corresponds toa virtual node which has an in-degree of at most H , so thefollowing theorem holds.
Theorem 2: With high probability, the maximum nodestress of an � -node DHT network is 9�(�L� �Q�*) +�, �) +�,-) +�, � � .
IV. OPTIMIZING SCHEMES AND ANALYSIS
Given a fixed path length, Theorem 1 suggests two ways toimprove the delivery ratio: increase = V A and decrease = �:A .In this work, we focus on the first approach. The general ideaof increasing = VA is to give preference to nodes that havestayed alive for a relatively long period of time when selectingforwarders in the delivery path. In this section, we introducethree schemes that can assist with this. Note that we do notelaborate on low-level protocol details here; rather, we focuson the main ideas and modifications on the original multicastand DHT protocols.
A. The Senior Member Overlay (SMO) Scheme
The SMO scheme organizes nodes above a certain thresholdage into a special overlay, called a senior member overlay(SMO), which will take the responsibility of all the forwardingtasks in data delivery. Now that most young (and thus unstable)nodes are pushed to the leaf level of the tree, they will notaffect other nodes and thus the data delivery ratio can beimproved. The idea of biased task allocation among heteroge-neous nodes (in terms of processing capacity, bandwidth, up
time, etc.) is not new. Our contribution here is the applicationof this idea to the new context of DHT-based multicast and aformalized analysis with respect to data delivery reliability.
The formation of the SMO is simple. The only parameterinvolved is the threshold age � P ; when a node has stayed inthe base overlay for � P , it joins the SMO with the help ofsome bootstrap node, which can be easily obtained throughthe propagation and exchange of node information in the baseoverlay.
In the SMO scheme, every publisher is required to join theSMO. For a non-publisher node, if it is not in the SMO, itneeds to identify on the Chord ring the nearest successor nodethat belongs to the SMO, called the SMO successor. When anode joins a multicast group, if it is already a member ofthe SMO, it simply performs the joining routine of multicastprotocol within the SMO; otherwise it asks its SMO successorto perform the joining routine within the SMO, and then asksthe parent of the SMO successor to add itself as a child. Afterthis, the SMO successor is dropped from the parent’s childlist.
To help understand the tradeoff between delivery ratio andload balancing, we define another metric that is related to� P . The SMO fraction, denoted � , is the proportion of allnodes that are selected to join the SMO. Clearly, � P isequal to the � �b� �O� -quantile, denoted � 2���� , of a node’sage distribution � � . To calculate this quantile, a node firstestimates the nodes’ lifetime distribution � 6 by monitoringthe arrival and departure times of its neighbors and exchangingthis information with others; it then obtains ��� according toLemma 3 and finally calculates � 2���� .
Let random variable V������ denote the residual lifetime of arandomly chosen SMO node. (We are interested in only SMOnodes because they are the forwarding nodes for data delivery.)The following lemma concerning the density function of V�����holds.
Lemma 5: In a DHT network with the SMO scheme, asT+�5� ,
* 6� �� �871�h �( ` =
� � �,� ��7%� A � (5)
where ( ` ( � ��������P = � � � � ��Z��BA4�%Z�� �@�Q^���\ � ��� moreover,
= V������[A� (i9 � )�9$<( ` j (6)
Now, the worst-case data delivery ratio can be obtainedusing the asymptotic results of renewal theory, as done inTheorem 1.
Theorem 3: In a DHT network with the SMO scheme, theworst-case stationary delivery ratio of two nodes that are atmost N hops apart is given by� �������- � (i9 � )�9
( 9 � ) 9 � $<( ` = �:A�� � ��2 � (7)
where ( ` ( � 4�������P = � � ���h�3Z��BA��%Z�� �@�Q^���\ � ��jWhen node lifetime follows a Pareto distribution, the worst-
case delivery ratio has an elegant expression. The followingcorollary can be obtained after simple integration.
Corollary 1: For Pareto node lifetime distribution, theworst-case stationary delivery ratio of two nodes that are atmost N hops apart in a DHT network with the SMO scheme is� � ����� � (i9 � )�9
( 9 � ) 9 � $���(( = �:A�� � ��2 j (8)
Eq.(8) differs from Eq.(4) only in the factor � in thedenominator, which clearly shows the impact of � on deliveryratio. This is further demonstrated in Figure 2, which showsthe worst-case delivery ratios under the SMO scheme forvarying values of � and = �:A . The node lifetime model isthe same as that used in Table I. It can be seen that the useof SMO can effectively improve the worst-case delivery ratio;moreover, the smaller the SMO, the higher this ratio. However,a small SMO may result in more load imbalance between theSMO nodes and non-SMO nodes. This tradeoff is quantifiedby the following theorem.
Theorem 4: With high probability, the maximum nodestress of an � -node network with the SMO scheme is� L � 2��39�(� � � � ) +�, � �) +�,-) +�, � �"! jB. The Longer-Lived Neighbor Selection (LNS) Scheme
LNS makes use of the flexibility of neighbor selectionprovided by some DHT algorithms such as Chord, Pastry, andKademlia. In Chord, for example, it is possible for a node �to choose its D th neighbor from a subset of nodes, called acandidate subset, in the range = � � $ F ��� � $ F � 2/� [13]. Thisenables a node to choose reliable neighbors by selecting theoldest node from each candidate set, so that the data deliverypath can be more reliable. Considering that the candidate setmay grow too large as D increases, we let LNS sample at most#
consecutive nodes starting from � � $ F for the D th neighborof node � .
We now consider the residual lifetime of a forwarding nodeon a delivery path. The following lemma characterizes such arandom variable and further provides its mean for the specialcase of Pareto lifetime distribution.
Lemma 6: Let V F� � � �@�b^ D:^ N � be the residual lifetime ofa node’s D th neighbor node in a DHT network with the LNSscheme, then the density of V F� � � is given by
* 6"$% & ��71� (' F() $* 3P
* � ��� � 71� ! * �P� � � � � ��Z�� ! �%Z ' ) $ �i2 �%���
(9)where ' F ,+.-0/
� $ F �i2 � # � . Specially, if node lifetime followsa Pareto distribution,
= V F� � � A� �! �?�21 �
� � 22 ��� ! ' F�31 � ' F
� � � 22���� !j (10)
To see the trend of = V F� � � A as ' F increases, setting 4 2�%�i2and 5 6�1��2 1 �
� � 22���� ! , we can expand Eq. (10) as
= V F� � � A8795;: �8< ! � � 4' F� 4 ' ) $
� 2>=a9 � ' F � 4M� < � (11)
which indicates that = V F� � � A grows with ' F (thus#
) and tendsto infinity. Also notice that in the special case of
# � �"!G �and �W �
, Eq. (10) reduces to = V F� � � A� �,�%�@! � $+� + = VA ,
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0 W
orst
-cas
e de
liver
y R
atio
SMO Fraction (delta)
E[R] = 30 sec E[R] = 60 sec E[R] = 90 sec E[R] = 120 sec
Fig. 2. Impact of SMO fraction � on worst-casedelivery ratio.
1 2 5 10 20 30 60 100 150 0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Wor
st-c
ase
deliv
ery
ratio
Maximum candidate set size (K)
E[R] = 30 sec E[R] = 60 sec E[R] = 90 sec E[R] = 120 sec
Fig. 3. Impact of maximum candidate size � onworst-case delivery ratio.
10 15 20 25 30 35 0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Wor
st-c
ase
deliv
ery
ratio
Maximum path length
Plain DHT, E[R] = 60 sec Plain DHT, E[R] = 90 sec RRS DHT, E[R] = 60 sec RRS DHT, E[R] = 90 sec
Fig. 4. Worst-case delivery ratio as a function ofmaximum path length � .
which corresponds to the node residual lifetime in a plainDHT.
The following theorem gives the delivery ratio result forDHT network under the LNS scheme.
Theorem 5: In a DHT network where nodes’ lifetimesfollow a Pareto distribution and the LNS scheme is applied,the worst-case stationary delivery ratio of two nodes that areat most N hops apart is given by� � � � � � �i2�
F,$ 2� 1 �
� � 22���� ! ' F 3� 1 �� � 22���� ! ' F 3
� ��! �?� � 1 � ' F�;� � 22 ��� ! = �:A �(12)
where ' F,,+.-0/� $ F � # � .
Figure 3 shows the worst-case delivery ratios under the LNSscheme for varying candidate set sizes and mean fixing times.The node lifetime model is the same as that used in Table I. Asexpected, the worst-case delivery ratio improves substantiallyas#
varies from 1 to 100. A large#
, however, implies moresignificant load imbalance. The following theorem shows alinear relationship between
#and the maximum node stress.
Theorem 6: With high probability, the maximum nodestress for an � -node DHT network with the LNS scheme is� L 9�(� �Q� ) +�, �) +�,-) +�, � � .C. The Reliable Route Selection (RRS) Scheme
Although in its original proposal, Chord greedily routes amessage towards a destination in decreasing ID distances, theorder of the distances is in no way essential to the correctnessand efficiency of routing. In other words, in terms of total hopcount, a route that spans a sequence of distances is equivalentto any route that spans a permutation of that sequence, ifthat permutated sequence could be achieved. For example,the route traversing a node sequence
� ��� � � � (with distancesequence � ��� ) is equivalent to the route traversing nodes� ���1� � � (with distance sequence � �� ). This flexibility providessome room for choosing stable nodes for a data delivery path.Specifically, at each node the RRS scheme chooses the oldestnode from a set of neighbors, called the candidate set, as thenext hop. The candidate set is selected in such a way thatthe total hop count will not be increased. Here we analyze anapproach to achieving this: if the distance of a node to some
destination is expressed as a binary number, then neighbor Dis selected to the set if that binary number has a 1 in the D thposition. This can be thought of as a binary string having its 1bits cleared one at a time as a message routes from the sourceto the destination, and hence we call it bit-clearing. Thisheuristic is originally proposed in [13] for achieving networkproximity; it, however, can guarantee a fixed hop count onlyin a fully populated DHT. Some other heuristics are possibleand we will discuss these in Section V.
In the following, we assume two nodes that are initially $ � ��distance apart on the ring, then starting from the receiving
node, the first node can choose any of its N � �neighbors as its
next hop, the next node has N � $ possible next hops, and soon to generate a route. Following the same line of Lemma 6and Theorem 5, we can obtain the following results.
Lemma 7: Let V F� � ��� ^ D ^ N@� be the residual lifetime ofthe D th forwarding node on the path
� � P ��� 2 �/j j/j[����� � , where� P is the source node and ��� the receiving node, in a DHTnetwork with the RRS scheme. Then the density of V F � isgiven by
* 6 $��� ��7%� D( F* 3P
* � ��� � 71� ! * �P� � � � � �3Z�� ! �1Z ' F �i2 �%��j (13)
Specially, if node lifetime follows a Pareto distribution,
= V F� � A� �! �]� 1 �
� � 22���� ! D 31 � D
� � � 22���� !j (14)
Theorem 7: In a DHT network where nodes’ lifetimesfollow a Pareto distribution and the RRS scheme is applied,the worst-case stationary delivery ratio of two nodes that areat most N hops apart is given by� � � �0
� �i2�
F,$ 2� 1 �
� � 22���� ! D 3� 1 �� � 22���� ! D 3
� ��! �]� � 1 ��D�;� � 22 ��� ! = �:A j(15)
Theorem 8: With high probability, the maximum nodestress of an � -node DHT network with the RRS scheme isL� �9�(� �Q�*) +�, �) +�, ) +�, � � .Figure 4 shows the delivery ratios under the RRS scheme forvarying path lengths and mean fixing times. The node lifetimemodel is the same as that used in Table I. As can be seen, the
RRS scheme leads to a considerable improvement in the worst-case delivery ratio. Moreover, the curves for the RRS schemehave smaller slopes than those for a plain DHT, indicatingthat as the path length increases, the worst-case delivery ratiounder the RRS scheme drops at a smaller speed than in a plainDHT. This is because a longer path provides larger candidatesets, which in turn increases = V F � A . This partly compensatesfor the loss of delivery ratio due to the increase in path length.
D. Comparison of the Schemes
Besides the differences in delivery ratio and maximum nodestress, the three schemes differ in a number of other respects,including applicability to different DHT algorithms, controlflexibility, implementation cost, and communication overhead.
The SMO scheme does not rely on any particular underlyingDHT geometries, so it can be applied to all types of DHTnetwork. It provides the parameter SMO fraction to balancebetween data delivery ratio and load balancing, and thereforehas good control flexibility. To implement this scheme, theoriginal multicast protocol needs to be strengthened to main-tain the age information of nodes and be aware of the existenceof two overlays, whereas the underlying DHT algorithm neednot be changed. The major drawback of this scheme is thatit requires a fraction of nodes to stay in two DHT overlayssimultaneously, which means a higher node overhead andmessage traffic for those nodes.
The LNS scheme makes use of the flexibility of neighborselection, which is unavailable for some DHT algorithms suchas CAN and de Bruijn. Like SMO, the scheme also provides aparameter
#to balance between reliability and load balancing.
This scheme only requires minor modification to existingDHT algorithms, which is transparent to upper-layer multicastprotocols. The extra overhead imposed on the nodes is smallbecause the nodes only need to sample a limited number ofnodes to choose the oldest ones as its neighbors.
The RRS scheme relies on the flexibility of route selection,which can only be provided by Chord and CAN in our cases.Therefore, this scheme has the least applicability in terms ofDHT choice. Like LNS, it has minimal implementation costand running overhead.
V. SIMULATION STUDY
A. Methodology
An event-driven simulator is developed with the Chord andScribe protocols implemented. Nodes enter the DHT networkin a homogeneous Poisson process such that the averagenumber of DHT nodes remains at approximately 800,000.Upon joining the DHT, some nodes also participate in one of30 multicast groups with equal probability; the total number ofmulticast members remains at approximately 300,000. Sincethe reliability of data delivery is our focus, the simulatordoes not model network latency. Two lifetime distributionsare tested: Pareto distribution and Lognormal distribution asobserved by [26] and [30]. The Pareto distribution has thesame parameters as those used in Table I and the Lognormaldistribution has the scale and shape parameters set to 5.5 and
2, respectively, such that both distributions have a mean ofaround 30 minutes. We wish to see how the variations inlifetime distribution affect multicast reliability and how theoptimizing schemes adapt to them.
In the simulator, the data loss is only caused by nodedepartures (without notification). The total failure detectionand recovery time is assumed to be uniformly random between= �1���M�<A seconds. We call the accumulative time a node spendsin failure detection and recovery its failure time, and definethe ratio of the failure time to its session time (current timeminus arrival time) as the data loss ratio (or loss ratio), whichis equal to
� �the delivery ratio.
Other performance metrics include node stress and pathlength. For loss ratio and path length, we only report thestatistical results of all the leaf nodes, as they reflect the worst-case performance of the multicast. All the following resultsare taken from a typical snapshot of the network after it hasevolved for 3.6 hours.
B. Simulation Results
1) Data loss ratio: Figures 5(a) and 5(b) show the cumula-tive distribution functions (CDFs) of loss ratio under the SMOscheme. It can be seen that the SMO scheme considerablyreduces the loss ratios. In the Pareto case (Figure 5(a)), forexample, an SMO fraction of 0.1 reduces the mean loss ratioby nearly a half (from 18.0% to 9.3%); and the fractionof nodes whose loss ratios are below 10% improves from22.6% to 68.9%. Similar observations can be made in theLognormal case (Figure 5(b)). Compared with the Pareto case,the Lognormal case has consistently lower mean loss ratios.Lemma 4 helps us understand the case for SMO fraction �(plain DHT): the mean residual lifetime = V A is 1 hour for thePareto lifetime distribution, whereas with Lognormal lifetimedistribution, = V A 7 � � j�� hours, which means a higherdelivery ratio according to Theorem 1. (The path lengths underthe two cases are indeed very close to each other. The resultsare reported in [29].) Lemma 5 and Theorem 3 can explainthe difference in curves for the other SMO fractions.
Figures 6(a) and 6(b) show the CDFs of loss ratios underthe LNS scheme, which demonstrate a noticeable improvementby LNS scheme. The difference between the Pareto andLognormal cases lies in the sensitivity of loss ratio to thechanging value of
#: in the Lognormal case, the loss ratio
improves more significantly with small values of#
. Forexample, the loss ratio drops by 49% as
#grows from 1
to 4, whereas a reduction of only 16% can be observed as#
grows from 5 to 16.The loss ratios under the RRS scheme with Pareto lifetime
distribution are shown in Figure 7. The bit-clearing heuristicresults in a substantially higher mean loss ratio than in theplain DHT. This is because in a non-fully-populated DHT,the distance span for each hop is not necessarily a power of2, thus the binary string of the distance may often find new1 bits appear when messages route from the source to thedestination, which results in a larger number of hop countsand hence an increased loss ratio than in the plain DHT.
1.5625 3.125 6.25 12.5 25 50 100
0
20
40
60
80
100
SMO fraction = 0.1 Mean = 9.3%
SMO fraction = 1 (plain DHT) Mean = 18.0%
SMO fraction = 0.5 Mean = 14.6%
Cum
ulat
ive
% o
f nod
es
Data loss ratio (%)
SMO fraction = 1 SMO fraction = 0.5 SMO fraction = 0.1
(a)
1.5625 3.125 6.25 12.5 25 50 100
0
20
40
60
80
100
SMO fraction = 0.1 Mean = 6.1%
SMO fraction = 1 (plain DHT) Mean = 14.9%
SMO fraction = 0.5 Mean = 11.7%
Cum
ulat
ive
% o
f nod
es
Data loss ratio (%)
SMO fraction = 1 SMO fraction = 0.5 SMO fraction = 0.1
(b)
Fig. 5. CDF of Data loss ratio. (a) SMO + Pareto lifetime distribution; (b)SMO + Lognormal lifetime distribution.
1.5625 3.125 6.25 12.5 25 50 100
0
20
40
60
80
100
L = 16 Mean = 10.4%
K = 1 (plain DHT) Mean = 18.0%
L = 4 Mean = 14.6%
Cum
ulat
ive
% o
f nod
es
Data loss ratio (%)
Candidate set size K=1 Candidate set size K=4 Candidate set size K=16
(a)
1.5625 3.125 6.25 12.5 25 50 100
0
20
40
60
80
100
K = 16 Mean = 6.4%
K = 1 (plain DHT) Mean = 14.9%
K = 4 Mean = 7.5%
Cum
ulat
ive
% o
f nod
es
Data loss ratio (%)
Candidate set size K=1 Candidate set size K=4 Candidate set size K=16
(b)
Fig. 6. CDF of Data loss ratio. (a) LNS + Pareto lifetime distribution; (b)LNS + Lognormal lifetime distribution.
In view of this, we consider another heuristic for theselection of next hop nodes. Assuming the distance of aneighbor is
�and the age of that neighbor is � , then the
neighbor with the maximum value of��� � � �@!S� �M� is
selected as the next hop node. The heuristic trades off totalhop count against the choices of stable nodes, and the powerof age ! determines the effect of age. When !Gc� , the routingscheme reduces to that of a plain DHT. Figure 7 shows theCDFs of loss ratio for three values of ! . For ! �
, theloss ratio is still higher than that of the plain DHT, whichimplies the negative effect of increased hop count exceeds thepositive effect of stable node selection. Lowering ! from 1to 0.5 remedies this situation slightly, making the loss ratiovery close to that of the plain DHT. Further reduction of ! ,however, have very little effect on the loss ratio under thePareto distribution. (The results are not shown to preserve theclarity of the figures.)
The stable node selection appears to be more effective fora Lognormal lifetime distribution (Figure 7(b)), although theimprovement is only moderate: for !]e� j � , the loss ratio isreduced by 31%; the bit-clearing heuristic still performs worst.
2) Node stress: Figure 8 shows the distribution of nodestress under the SMO scheme. When � ^ �
, a fraction�-� �
of nodes are outside the SMO and have a node stress of zero,so the total number of zero-stress nodes are larger than that ofa plain DHT. For example, the number of zero-stress nodes is105,695 for a plain DHT, whereas this figure grows to 150,010for the SMO scheme with �Q �1j � . On the other hand, sinceall the forwarding tasks are assigned to only a fraction of thenodes, there are likely to be more nodes with high stress valuesthan in a plain DHT. This is shown by the bars in Figure 8
1.5625 3.125 6.25 12.5 25 50 100
0
20
40
60
80
100
Power of age = 1 Mean = 19.1%
Power of age = 0.5 Mean = 17.9%
Bit-clearing Mean = 34.6%
Power of age = 0 (plain DHT) Mean = 18.0%
Cum
ulat
ive
% o
f nod
es
Data loss ratio (%)
Plain DHT Power of age = 1 Power of age = 0.5 Bit-fixing
(a)
1.5625 3.125 6.25 12.5 25 50 100
0
20
40
60
80
100
Power of age = 0.5 Mean = 11.4%
Power of age = 0 (plain DHT) Mean = 14.9%
Bit-clearing Mean = 15.4%
Power of age = 1 Mean = 10.4%
Cum
ulat
ive
% o
f nod
es
Data loss ratio (%)
Power of age = 0 Power of age = 1 Power of age = 0.5 Bit fixing
(b)
Fig. 7. CDF of Data loss ratio. (a) RRS + Pareto lifetime distribution; (b)RRS + Lognormal lifetime distribution.
for stress values ranging from 12 to 36. Generally, the smallerthe SMO fraction, the more nodes are distributed near bothends of the range of stress values. Similar observations can bemade from Figures 9 and 10, which depict the distributions ofnode stress under the LNS and the RRS schemes, respectively,although the latter figure shows a smaller deviation of stressdistribution from that of the plain DHT.
3) Path length: Although it is an important factor for datadelivery reliability, path length is critical for many applicationsin its own right. Figure 11 shows the mean path length of thevarious schemes. It can be seen that the SMO and the LNSschemes can slightly shorten the multicast paths, especiallywith a smaller SMO fraction or a large candidate set size.This is because a smaller set of forwarding nodes reduces thenumber of necessary intermediate nodes on the path. Considerin the extreme case where only one forwarder is present(corresponding to an SMO with a single node or
# �� ), theoverlay graph assumes to a star-like structure, and the pathlength between an pair of nodes would be only 2.
The RRS scheme with the bit-clearing heuristic yields verylarge path lengths, which verifies the qualitative discussion inSection V-B.1. For the heuristic using the product of distanceand age, the path length is still longer than that of the plainDHT – a consequence of sacrificing small hop counts forchoices of high reliability routes.
VI. RELATED WORK
From the perspective of stochastic modeling, perhaps theclosest to our work is by Leonard et al. [14]. Based on a similarnode lifetime model to ours (see the discussion of differencesin Section III-A), they analyze the resilience of general peer-to-peer networks and derive the expected delay before a user isisolated from the network and the probability of this occurringwithin his/her lifetime. They model the evolution of a node’sneighbors as a superposition of renewal processes, and thenobtain the limiting probability of at least one neighbor beingalive using renewal theory. We also use renewal theory toanalyze the normal probability of an intermediate node on adelivery path, but our focus is on the probability of all nodesbeing at the normal state simultaneously.
For the resilience of DHT networks, Gummadi et al. [13]identify several representative routing geometries and analyzetheir degrees of flexibility which benefits static resilience and
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36
2 8
2 9
2 10
2 11
1x2 12
1x2 13
2 14
2 15
2 16
2 17
Num
ber
of n
odes
Node stress value
SMO fraction = 1 (Plain DHT) SMO fraction = 0.5 SMO fraction = 0.1
Fig. 8. Node stress under theSMO scheme.
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36
2 9
2 10
2 11
1x2 12
1x2 13
2 14
2 15
2 16
2 17
Num
ber
of n
odes
Node stress value
Candidate set size = 1 (plain DHT) Candidate set size = 4 Candidate set size = 8
Fig. 9. Node stress under the LNSscheme.
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36
2 9
2 10
2 11
1x2 12
1x2 13
2 14
2 15
2 16
2 17
Num
ber
of n
odes
Node stress value
Power of age = 0 (plain DHT) Power of age = 0.5 Power of age = 1
Fig. 10. Node stress under theRRS scheme.
plain DHT SMO,0.5 SMO,0.1 LNS,4 LNS,16 RRS,1 Bit-clearing 0 5
10 15 20 25 30 35 40 45 50 55 60 65 70 75
Pat
h le
ngth
(ho
p co
unt)
Scheme
5th and 95th percentiles
Fig. 11. Path lengths under dif-ferent schemes.
proximity. Loguinov et al. [15] examine graph-theoretic prop-erties of several DHTs and analyze their routing performanceand fault resilience. Stutzbach et al. [28] characterize the churnof P2P networks through empirical experiments.
In the field of application-layer multicast, reliability hasbeen a topic of enduring interest. The early work of Chu etal. [12] and Padmanabhan et al. [20] proposes some simpleand effective techniques for improving the multicast reliability.Castro et al. [9] propose to use multiple trees to improve theresilience of streaming to interruptions. In [25], a number ofsingle-tree construction algorithms are proposed and evaluatedusing traces from large-scale commercial systems. Due to thelack of a generic supporting overlay topology, these optimizingschemes are somewhat ad hoc, and are therefore difficult tomodel and evaluate using an integrated framework.
VII. CONCLUSIONS
This paper investigates the reliability of DHT-based mul-ticast. The contributions include: (1) A node residual timemodel, which is fundamental to our analysis and we believewill be a useful tool in other contexts of overlay-based applica-tions; (2) A renewal-theory-based model for stationary deliveryratio, which provides a worst-case estimate for the reliabilityof data delivery between two nodes in the DHT network; (3)Three optimization schemes and analysis of their reliability;and (4) simulation experiments which provide insight into theperformance of DHT-based multicast from a number of majorrespects. In the future, we will consider how the model and theoptimization schemes can be applied to bandwidth-demandingapplications such as video streaming.
REFERENCES
[1] RSSDHT. http://sourceforge.net/projects/rssdht/[2] Bamboo DHT project. http://bamboo-dht.org/
[3] K. C. Almeroth and M. H. Ammar. Collecting and Modeling theJoin/Leave Behavior of Multicast Group Members in the MBone. Proc.of the High Performance Distributed Computing (HPDC), 1996.
[4] S. Banerjee, S. Lee, B. Bhattacharjee, and A. Srinivasan. Resilientmulticast using overlays. ACM SIGMETRICS 2003.
[5] A. R. Bharambe, S. G. Rao, V. N. Padmanabhan, S. Seshan and H.Zhang. The Impact of Heterogeneous Bandwidth Constraints on DHT-Based Multicast Protocols. IPTPS, 2005.
[6] M. Bishop, S. Rao, and K. Sripanidkulchai. Considering Priority inOverlay Multicast Protocols under Heterogeneous Environments. InProc. of INFOCOM 2006.
[7] P. B. Godfrey, S. Shenker, and I. Stoica. Minimizing Churn in DistributedSystems. Proc. of SIGCOMM 2006.
[8] F. E. Bustemante and Y. Qiao. Friendships that last: peer lifespan andits role in P2P protocols. WCW workshop 2003.
[9] M. Castro, P. Druschel, A-M. Kermarrec, A. Nandi, A. Rowstronand A. Singh. SplitStream: High-bandwidth multicast in a cooperativeenvironment. Proc. of SOSP 2003.
[10] M. Castro, P. Druschel, A-M. Kermarrec and A. Rowstron. Scribe: Alarge-scale and decentralised application-level multicast infrastructure.IEEE Journal on Selected Areas in Communications (JSAC). Oct., 2002.
[11] Y. Chu, A. Ganjam, T. S. E. Ng, S. G. Rao, K. Sripanidkulchai,J. Zhan and H. Zhang. Early Experience with an Internet BroadcastSystem Based on Overlay Multicast. USENIX 2004 Annual TechnicalConference.
[12] Y. Chu, S. Rao, and H. Zhang. A Case for End System Multicast. Proc.of ACM SIGMETRICS, June 2000.
[13] P. K. Gummadi, R. Gummadi, S. D. Gribble, S. Ratnasamy, S. Shenkerand I. Stoica. The impact of DHT routing geometry on resilience andproximity. Proc. of ACM SIGCOMM 2003.
[14] D. Leonard, V. Rai, and D. Loguinov. On lifetime-based node failureand stochastic resilience of decentralized peer-to-peer networks. Proc.of ACM SIGMETRICS 2005.
[15] D. Loguinov, A. Kumar, V. Rai, and S. Ganesh. Graph-theoretic analysisof structured peer-to-peer systems: routing distances and fault resilience.Proc. of ACM SIGCOMM 2003.
[16] P. Maymounkov and D. Mazieres. Kademlia: A Peer-to-peer InformationSystem Based on the XOR Metric. IPTPS, 2002.
[17] M. Mitzenmacher and E. Upfal. Probability and Computing . CambridgeUniversity Press, 2005.
[18] F. Kaashoek and D. R. Karger. Koorde: A simple degree-optimaldistributed hash table. IPTPS 2003.
[19] V. G. Kulkarni. Modeling and Analysis of Stochastic Systems. Chapman& Hall Ltd. ISBN: 0-41204-991-0, 1996.
[20] V. N. Padmanabhan, H. J. Wang and P. A. Chou. Resilient Peer-to-PeerStreaming. Proc. of ICNP 2003.
[21] S. Ratnasamy, M. Handley, R. Karp, S. Shenker. Application-levelMulticast using Content-Addressable Networks. Proc. of InternationalWorkshop on Networked Group Communication (NGC) 2001.
[22] S. Ratnasamy, P. Francis, M. Handley, R. Karp and S. Shenker. AScalable Content-Addressable Network. Proc. of SIGCOMM 2001.
[23] S. Rhea, B. Godfrey, B. Karp, et al. OpenDHT: A Public DHT Serviceand Its Uses. Proc. of SIGCOMM 2005.
[24] A. Rowstron and P. Druschel. Pastry: Scalable, distributed object loca-tion and routing for large-scale peer-to-peer systems. IFIP/ACM Intl.Conference on Distributed Systems Platforms 2001.
[25] K. Sripanidkulchai, A. Ganjam, B. Maggs and H. Zhang. The feasibilityof supporting large-scale live streaming applications with dynamicapplication end-points. Proc. of ACM SIGCOMM, 2004.
[26] K. Sripanidkulchai, B. Maggs and H. Zhang An analysis of livestreaming workloads on the Internet. SIGCOMM IMC 2004.
[27] I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, H. Balakrishnan. Chord:A Scalable Peer-to-Peer Lookup Service for Internet Applications. Proc.of ACM SIGCOMM 2001.
[28] D. Stutzbach, and R. Rejaie. Understanding Churn in Peer-to-PeerNetworks. SIGCOMM IMC 2006.
[29] G. Tan and Stephen A. Jarvis. On the reliability of DHT-based multicast.Technical Report CS-TR-06, University of Warwick, 2006.
[30] E. Veloso, V. Almeida, W. Meira, A. Bestavros, and S. Jin. A Hierarchi-cal Characterization of A Live Streaming Media Workload. IEEE/ACMTrans. on Networking, 14(1), 2006.
[31] S. Q. Zhuang, B. Y. Zhao, A. D. Joseph, R. H. Katz, J. D. Kubiatowicz.Bayeux: An Architecture for Scalable and Fault-tolerant Wide-area DataDissemination. Proc. of NOSSDAV 2001.
