Gossip Protocols
Mrk Jelasity
Hungarian Academy of Sciences andUniversity of Szeged, Hungary
2
Introduction A few key ideas from previous class
Overlay networks are the key abstraction Function is always implemented as a distributed
algorithm that communicates over this network Overlay networks can emerge or can be created
Main points in this class Gossip protocols over networks
Dissemination, computation, overlay construction Applications for distributed data mining
EM algorithm, clustering, collaborative filtering
3
Introduction Gossip-like phenomena are commonplace
human gossip epidemics (virus spreading, etc) computer epidemics (malicious agents: worms, viruses) phenomena such as forest fires, branching processes
and diffusion are all similar mathematically extremely simple locally, powerful and robust globally In computer science, epidemics are relevant
for security (against worms and viruses) for designing useful protocols (we look at this here)
4
Problem Xerox corporate Internet, replicated databases Each database has a set of keys that have values (along with a
time stamp) Goal: all databases are the same, in the face of key updates,
removals and additions Updates are made locally and have to be replicated at all sites
(300 sites) Solution in 1986: emailing updates
problems with detecting and correcting errors (done by hand!) bottleneck with the originating (updated) site not scalable (slow if very large number of nodes) (message complexity quite good though!)
Epidemic Database Updates
5
Gossip to the rescue Main components are replaced by gossip
update spreading: rumor mongering (no bottleneck) error correction: anti-entropy gossip (reliable)
anti-entropy uses simple epidemics with two states: infective and
susceptible (a.k.a. SI model) guarantees perfect dissemination
rumor mongering uses complex epidemics with an additional state:
removed (a.k.a. SIR model) certain (quite small) probability of error
6
SI gossip
What overlay network is assumed here?
7
Some Properties of the SI model the push model
N nodes communicate in rounds (cycles) in each cycle, a node that has the update (infected)
sends it to a random other node, that becomes infected too
In anti-entropy nodes send the (hash of) the entire database (not only
a single update) as a side effect, all new updates are spread
according to the SI model receiving nodes update their own database via merging
the unseen updates
8
Mean-field model of push SI Let pi be the proportion of not infected nodes
in cycle i 1-p
0=1/N
Pittel (1987) shows that the model below is quite accurate for predicting p
i
E p i1=pi 1 1N N 1p i
p i e1p i
9
Speed and cost of push SI
Let SN be the first cycle where p
i=0
Pittel (1987) shows that in probability
SN=log N ln N O 1 This is quite fast... But the number of overall messages sent is
O N logN
10
Pull and push-pull SI With pull, we have
This is very fast when pi is small (end phase)...
Karp et al (2000) show that the number of overall messages sent with push-pull is
O N loglogN
E p i1=pi2
But termination is trickier when no updates are available (for anti-entropy does not matter)
11
SIR for spreading single updates For anti-entropy, use a pull or push-pull SI modell For the spreading of updates, the termination problem
needs to be addressed: rumor mongering with SIR model
Push approach when a rumor (update) becomes cold, stop
pushing Pull approach
same as push, only stop offering update when pulled when it becomes cold
12
SIR gossip
13
Stop spreading info with probability 1/k if unsuccessful infection attempt (become removed)
s: susceptible, i: infective, r: removed
Egifk=1,20%missthegossip,ifk=2,6%missit Ingeneral,withpush,theprob
tomisstheupdateisapproxem(mistheoverallmessages)
dsdt
=si
didt
=si1k
1s i
s=ek11s
Rumor mongering with push
14
Some other rumor mongering algorithms
Removal algorithms Counter: removed after exactly k
unsuccessful attempts Random: removed with pr. 1/k after each
unsuccessful attempt Blind: removal algorithm is run in each cycle
irrespective of contacted node Feedback: removal algorithm runs if contacted
node was not susceptible
15
Random networks Note that gossiping nodes pick another node in
each cycle: they do not need to know all the nodes
The actual communication pattern defines a random graph by looking at these graphs, we can understand the
properties of the communication better we can design better gossip protocols if we
understand the implications of our design decisions
16
The ER model is often used to reason about gossip protocol design. This is problematic for a number of reasons In the ER model nodes can get stuck without
neighbors. This is the main reason for disconnectivity. In push or push-pull this is impossible
If we guarantee that all nodes communicate to at least 4 other nodes after receiving the update, we have a radically different model
Message and node failure pushes the underlying network toward the ER model, but not completely
Comments on theoretical models
17
Aggregation
Calculate a global function over distributed data eg average, but more complex examples include
variance, network size, model fitting, etc usual structured/unstructured approaches exist
structured: create an overlay (eg a tree) and use that to calculate the function hierarchically
unstructured: design a stochastic iteration algorithm that converges to what you want (gossip)
18
push-pull averaging
19
Illustration of averaging
12
8
7
2
6
3
20
Illustration of averaging
12
8
7
2
6
3
(12+6)/2=9
21
Illustration of averaging
9
8
7
2
9
3
22
Initialstate Cycle1 Cycle2
Cycle3 Cycle4 Cycle5
Illustration of averaging
23
Main intuition Mass conservation: the sum of approximations
in the network is constant Variance reduction: in all steps, the variance of
the approximations decreases (if the participating two values are different)
24
Some theoretical properties
If yi is the original value, and x
i is the current
estimate at node i, and y is the correct average then let us define
Then it can be shown that
25
Some applications We can calculate max (and min) if we replace
(m.x+x)/2 with max(m.x, x) (or min(m.x, x)) Apart from the average, in general we can
calculate the f-mean
f(x)=ln x gives geometric mean, f(x)=1/x gives harmonic mean, f(x)=xm gives the power mean
26
Some applications Using the averages of powers we can
approximate the variance and higher moments, eg
Network size can be approximated if one node sets its value to 1 and the other nodes set it to 0: then the average is 1/N
Sum can be calculated, since it is the average multiplied by the network size
27
Improvements Tolerates asymmetric message loss (only push
or pull) badly Tolerates overlaps in pairwise exchanges badly [Kempe et al 2003] propose a slightly different
version: push averaging all nodes maintain s (sum estimate) and w (weight) estimate is s/w only push: send (s/2,w/2), and keep s=s/2, w=w/2
several other variations exist
28
Push averaging
29
Main intuition Mass conservation: the sum of x values in the
network is constant, as well as the sum of w values (which is N).
Variance reduction: the is no similarly simple clear intuition; however, a variance-like potential function
t can be shown to decrease
in expectation
30
Some applications Min and max can be calculated without w The f-mean and moments can be calculated Sum can be calculated by setting w=1 at
exactly one node, and w=0 at all the other nodes
Network size can also be calculated (set x=1 at all nodes and calculate sum)
31
EM Gaussian mixture k-component
Gaussian mixture
We want to estimate the parameters to maximize log likelihood
32
EM Gaussian mixture
Each data point xi has a distribution q
i(s) that
describes the level of responsibility of each Gaussian for generating x
i
The EM algorithm iterates between estimating qi (E-step) and (M-step)
Similar to k-means clustering
33
EM Gaussian mixture
E-step: qi(s)=p(s|x
i,)
M-step
34
Gossip-based implementation
Assume that each node i stores a vector xi, and
a global k is known. Let's run a distributed EM! Initialize the qi values arbitrarily Perform M-step by calculating the three averages
and the use these values to calculate Perform E-step (completely local operation)
35
A Gossip Skeleton Originally for information dissemination in a
very simple but efficient and reliable way Later the gossip approach has been
generalized resulting in many local probabilistic and periodic protocols
we will introduce a simple common skeleton and look at information dissemination topology construction aggregation
36
A Push-Pull Gossip Skeleton
37
Rumor mongering as an instance state: set of active updates selectPeer: a random peer from the network
very important component, we get back to this soon update: add the received updates to the local
set of updates propagation of one given update can be limited
(max k times or with some probability, as we have seen, etc)
38
Peer Sampling A key method is selectPeer in all gossip
protocols (influences performance and reliability)
In earliest works all nodes had a global view to select a random peer from scalability and dynamism problems
Scalable solutions are available to deal with this random walks on fixed overlay networks dynamic random networks
39
Random walks on networks if we are given any fixed network, we can
sample the nodes with any arbitrary distribution with the Metropolis algorithm:
This Markov chain has stationary distribution where d
i is the degree of node i (undirected
graph)
40
Gossip based peer sampling basic idea: random peer samples are provided by a
gossip algorithm: the peer sampling service The peer sampling service uses itself as peer
sampling service (bootstrapping) no need for fixed (external) network
state: a set of random overlay links to peers selectPeer: select a peer from the known set of
random peers update: for example, keep a random subset of the
union of the received and the old link set
41
Gossip protocols for topology management
ADE
SX
W
A E
42
Gossip protocols for topology management
ADE
SX
W
A E
SelectPeer
43
Gossip protocols for topology management
A E
Exchange of views
44
Gossip protocols for topology management
A EBoth sides apply updatethereby redefining topology
45
Gossip based peer sampling in reality a huge number of variations exist
timestamps on the overlay links can be taken into account: we can select peers with newer links, or in update we can prefer links that are newer
these variations represent important differences w.r.t. fault tolerance and the quality of samples the links at all nodes define a random-like overlay
that can have different properties (degree distribution, clustering, diameter, etc)
turns out actually not really random, but still good for gossip
46
Gossip based topology management
We saw we can build random networks. Can we build any network with gossip?
Yes, many examples proximity networks DHT-s (Bamboo DHT: maintains Pastry
structure with gossip inspired protocols) semantic proximity networks etc
47
T-Man T-MAN is a protocol that captures many of
these in a common framework, with the help of the ranking method: ranking is able to order any set of nodes according
to their desirability to be a neighbor of some given node
for example, based on hop count in a target structure (ring, tree, etc)
or based on more complicated criteria not expressible by any distance measure
48
Gossip based topology management
basic idea: random peer samples are provided by a gossip algorithm: the peer sampling service
state: a set of overlay links to peers selectPeer: select the peer from the known set of
peers that ranks highest according to the ranking method
update: keep those links that point to nodes that rank highest
49
Initialstate Cycle3 Cycle5
Cycle15Cycle12Cycle8
50
51
Some other examples firefly-inspired synchronization partitioning (slicing) and sorting in P2P
networks asynchronous implementation of matrix
iterations ranking (PageRank) reputation systems
emergent cooperation
52
Modular design We have seen that all gossip protocols use the
peer sampling service that is itself a gossip protocol
Can be generalized: gossip protocols can be stacked or arbitrarily combined actual local communication is the same (all
protocols can often piggyback the same message) conceptual structure is modular
53
Example modular architecture
54
Outlook Gossip is similar to many other fields of
research that also have some of the following features: periodic, local, probabilistic, symmetric
examples include swarm systems, cellular automata, parallel
asynchronous numeric iterations, self-stabilizing protocols, etc
55
A slide on viruses and worms We focused on good epidemics but malicious
applications are known viruses and worms replicate themselves via similar
algorithms using some underlying network such as email contacts or the Internet itself
The dynamics is described by SIS model Underlying networks are typically scale free
(power law degree distribution) can be proven: no threshold: it is nearly impossible
to completely eliminate a disease
56
Some open problems gossip in mobile contact networks and its
potential applications (also malware...) security
gossip is robust to benign failure but very sensitive to malicious attacks
current secure gossip protocols sacrifice simplicity and light-weight
interdisciplinary connections: toward a deeper understanding of self-organization and gossip protocols as a special case