Split and Subsume∗Subscription Normalization for Effective Content-based Messaging

K. R. Jayaram and Patrick EugsterDepartment of Computer Science, Purdue University

{jayaram, peugster}@cs.purdue.edu

Abstract—Content-based publish/subscribe networks (CPSNs)scale to large numbers of publishers and subscribers by havingbrokers summarize subscriptions from subscribers and down-stream brokers based on coverage relationships (“subsumption”)between subscriptions. A broker forwards the summary tobrokers which are upstream on the routes to the publishers.Current summarization and event processing mechanisms induceheavy event processing load on brokers, leading to low eventthroughput and high latency and further sharp performancedegradation under high rates of churn, i.e., addition, deletion,or modification of subscriptions.

This paper describes Beretta, a novel CPSN that leverages asimple model of typed events, enabling a succinct and uniformnormalized representation of subscriptions. This in turn supportshighly effective subsumption and attribute-wise split filtering withmatching complexity logarithmic in the number of subscriptions,and enables the systematic introduction of parameters intosubscriptions to support both parametric and structural updates.We empirically demonstrate that our techniques significantlyimprove throughput and latency of event propagation and reduceresponse times to subscription updates.

Keywords—subscription; summarization; subsumption; nor-malization; content-based; messaging;

I. INTRODUCTION

Designing distributed applications following a publish/sub-scribe model, i.e., as components interacting anonymouslyby publishing events and consuming events subscribed to,is a popular approach. Such a design limits coupling ofcomponents (supporting scalability in terms of participatingcomponents), and allows for communication to be streamlinedthrough a dedicated middleware system thus avoiding redun-dant traffic (supporting scalability in terms of communicatedevents).

A. Content-based Publish/Subscribe (CPS)

In content-based publish/subscribe (CPS), the most expres-sive publish/subscribe variant, a subscription is a predicate onthe data attributes in an event. CPS middleware systems typi-cally construct content-based publish/subscribe networks (CP-SNs) to connect publishers with the appropriate subscribers,performing content-based routing and filtering in a decentral-ized manner over intermediate broker nodes. Examples of CPSmiddleware are Siena [10][9], HERMES [28], REBECA [19],Gryphon [3], PADRES [23][24],GREEN [32], XSiena [36]and JEDI [15]. The current standard approach views eventcontent as sets of attribute-value pairs, and subscriptions as

∗Research partially funded by US NSF (grants 0644013 and 0834529).

logical combinations of elementary predicates of the formattr op value where op is an operator for event attribute attr.

To avoid storing and evaluating all subscriptions at eachbroker, CPSNs “summarize” subscriptions based on simi-larities among them. Subscription subsumption [9][23] is apopular technique used to summarize subscriptions. When abroker receives a subscription predicate Φ from a subscriberor a downstream broker, it forwards the subscription to anupstream broker unless Φ is subsumed (or covered) by aprevious subscription Φ′ (Φ � Φ′). We say that Φ � Φ′

iff ∀ e ,Φ(e) ⇒ Φ′(e). To facilitate addition and removal ofpredicates, these are stored as a partially ordered set (poset),ordered through �.

Another popular, divide and conquer-style, summariza-tion technique was introduced by Triantafillou and Econo-mides [26]. Instead of handling whole subscriptions, thistechnique splits subscriptions into attributes, and computesper-attribute summaries efficiently using arrays.

B. Current Bottlenecks in CPSNs

Subscription summarization significantly reduces the eventprocessing load at each broker, when compared to approacheslike static broadcast groups. But, if a broker receives nsubscriptions for a given type of event with k attributes, severaloperations performed at the broker for subscription storage,management and event forwarding exhibit O(k n)1 complex-ity. If posets, implemented as d − ary max heaps are usedinstead of arrays to summarize subscriptions, the complexityof subscription addition can be brought to O(k logd n), but thecomplexities of unsubscription and event forwarding remainO(k n). Several other used data-structures such as binarydecision diagrams (BDDs) have not improved this bound (seeboth [23] and [2] for BDDs).

C. Dynamic Content-based Publish/Subscribe

Another bottleneck in current CPSNs arises from new ap-plication demands. Emerging applications like high frequencyalgorithmic trading (HFT) which accounts for 73% of all UStrading volume (2009 figures [33], [4]) or location-sensitiveapplications with mobile clients (e.g., mobile social network-ing, vehicular networks) inherently require subscription up-dates, as stock values (and thus thresholds) as well as locations

1Some authors ignore k, because k is usually small; it is hard to imagineevents with more than 100 attributes.

of mobile devices (and thus zones of interest) change. Re-subscriptions consisting in issuing a new subscription and sub-sequently canceling the outdated one were perfectly suitablea decade ago but tend to bog down brokers under high churn(subscription addition, removal, and update) rates of modern-day applications by affecting the very data-structures that areused for event forwarding. Parametric subscriptions [35] [21],which allow for variables in subscriptions, have proven tobe an effective technique for subscription updates but theseonly address the simpler part of the problem – structuralsubscription updates such as addition or removal of elementarypredicates are still not possible through parametric subscrip-tions.

D. ContributionsThis paper presents Beretta, a novel CPSN which sustains

high throughput and low event-propagation latencies underhigh churn, whilst scaling to thousands of event types, pub-lishers, and subscribers. More precisely, our contributions are:• We propose a succinct and uniform model of events and

subscriptions. Our model leverages strong event typingand represents all subscriptions in a normalized formas combinations of value intervals and set inclusionswithout compromising on expressiveness. Our use ofevent types in lieu of a structural approach allows foraggressive performance optimizations without hamperinginteroperability or the addition of new event types.

• We describe a novel highly efficient broker algorithm,which reduces the number of elementary predicates eval-uated for matching an event from O(k n) to O(k log n),with n the number of subscriptions for the event type andk the number of its attributes. In short, our algorithmrelies on a divide and conquer strategy which we call“split and subsume”. Event types and normalization areexploited to split subscriptions into predicates on individ-ual event types and attributes and to efficiently regroupthese in interval trees and hash maps. Normalizationalso supports the systematic introduction of variables intopredicates and subscription summarization approximationthus accelerating updates in the set of subscriptions (addi-tion, removal, parametric and structural updates) handledby a broker.

• We empirically evaluate Beretta, first through two casestudies based on algorithmic trading and traffic manage-ment respectively, and then through scalability studies byworkload generation. Our evaluations demonstrate verypromising performance.

E. RoadmapSection II overviews the state of the art in CPSNs. Sec-

tion III describes the design of Beretta. Section IV presentsour empirical evaluation. Section V presents related work.Section VI draws final conclusions.

II. BACKGROUND ON CPSNS

In this section, we recall relevant notions and pinpointbottlenecks for throughput and latency in current CPSNs.

A. Notions and Principles

We assume that CPSNs use dedicated, interconnected, bro-ker processes bi to convey events between client nodes ci,i.e., publishers and subscribers. Brokers which serve clientprocesses are called edge brokers. We use the term client torefer to publishers and subscribers.

As pointed out by [9], like IP Multicast [17], CPS algo-rithms follow the principles of (a) downstream replicationwhere an event is routed in a single copy as far as possiblefrom the publisher and only cloned downstream as close aspossible to the subscribers interested in receiving it, and (b) up-stream evaluation where unwanted events are filtered away asclose as possible to the publisher to avoid wasting bandwidth.An advertisement defines the types of events produced by apublisher. When a CPSN does not use advertisements, the pathof an event from a publisher to a subscriber is determined onlyby subscriptions. Subscriptions then have to be routed from asubscriber to every other publisher in the CPSN, essentiallyforming a spanning tree from the subscriber to every publisherattached to the CPSN. When an event is published, it is routedalong this tree from the publisher to the subscriber usingthe reverse path forwarding. When advertisements are used,a subscription for an event of type τ only has to be routed tothose publishers that generated an advertisement of type τ.

B. Events and Subscriptions

An event e can be viewed as a set of attribute-value pairs{a1:v1,. . . ,an:vn} [25], [3], [9]. A subscription is usuallyrepresented as a predicate Φ in disjunctive normal form witha grammar such as the following:

Predicate Φ ::= Φ ∨Ψ | ΨConjunction Ψ ::= Ψ ∧Θ | ΘCondition Θ ::= a op vOperator op ::= ≤ | < | = | > | ≥

To decide on the routing of an event e = {a1 : v1, . . . , an :vn}, predicate Φ is evaluated on e by substituting vi for ai.This evaluation, written as Φ(e), yields a boolean value. Asmentioned earlier, we say that predicate Φ′ covers (subsumes)Φ, denoted by Φ � Φ′, iff ∀e,Φ(e) ⇒ Φ′(e). Disjoinedconjunctions are usually handled internally just like separatesubscriptions, with duplicate elimination. Therefore, for sim-plicity, we only consider predicates consisting of a singleconjunction.

C. Subscription Summarization Through Subsumption

Figure 1a shows an example of a CPSN with four clients– three subscribers (c1, c2, c3), one publisher (c4) and threebrokers (b1, b2, b3). Consider a single event with two attributesx and y of integer type. Figure 1b shows a part of the CPSN,and how subscriptions propagate. c1 subscribes to e with thepredicate Φ1 = x>15∧y<60. b1 gets the subscription, storesit and propagates it to b2. Then c2 subscribes with predicateΦ2=x>20∧y<50. b1 gets this subscription, but does notforward it to b2, because x>15∧y<60 covers x>30∧y<40(Φ2 � Φ1). Similarly, x>30∧y<40 is received from c3, and

c1

c2c3

b1

b2

b3c4

(a) A CPSN

c1 c2 c3

x > 20 ∧ y < 50

x > 30 ∧ y < 40

x > 15 ∧ y < 60

x > 30 ∧ y < 40x > 20 ∧ y < 50

x > 1

5 ∧ y

< 6

0

x > 15 ∧ y < 60

(b) Subscription summarization on b1 (I)

c1 c2 c3

x > 20 ∧ y < 70 x > 55 ∧ y < 95x > 15 ∧ y < 60

x > 15 ∧ y < 60 x > 20 ∧ y < 70

( x > 15 ∧ y < 60 ) || (x > 20 ∧ y < 70) || (x > 55 ∧ y < 95)

( x > 15 ∧ y < 60 ) || (x > 20 ∧ y < 70) || (x < 55 ∧ y < 95)

<x : 16, y : 80> is unwantedand not transmitted from b3 to b1

x > 55 ∧ y < 95

(c) Subscription summarization on b2 (II)

Fig. 1: Classic CPSNs and subscription summarization

not forwarded to b2. In the scenario of Figure 1b, Φ1 =x>15∧y<60 is the summary of Φ1, Φ2 and Φ3.

To compute the summary of subscriptions on a given eventtype, a broker uses the covering relation � to construct apartially ordered set (poset) P of predicates. Figure 1b showssuch a poset. The least upper bound (lub) of P is the predicatethat covers all other predicates. If no lub inherently existsamong the predicates, this predicate — dubbed LUB(P) —is computed as the disjunction of all predicates that are notalready covered by any other predicate. This poset can bestored as a tree and evaluated for an event e by letting e“trickle down” node-wise. Events that do not satisfy LUB(P)are immediately discarded and events that satisfy individualsubscriptions are forwarded to the corresponding subscribers.

D. Current Bottlenecks

Below we summarize the bottlenecks in current CPSNs:1) Linear time operations: If an event type τ has k

attributes, verifying Φ′ � Φ is O(k). Insertion of a newsubscription into a poset P involves searching the posetto add the subscription while respecting the subsumptionrelation. When P is implemented as a d − ary max-heap,this incurs a complexity of O(k logd|P|)=O(k logd n) wheren = |P| is the number of existing subscriptions. But,processing an unsubscription is O(k |P|) = O(kn), becausedeleting a subscription may involve recreating the whole poset,e.g., when the subscription being removed is the root of theposet. Evaluation of P to forward an event to all interestedsubscribers is O(k |P|) = O(k n).

Many alternatives to posets have been explored includingthe Rete algorithm [8], used for instance in PADRES [24].Rete yields a very effective event matching, because it usesa memory-intensive event flow graph. However, it does notinherently perform subscription summarization, and the com-plexities of subscription, unsubscription and event match-ing remain O(k n). Another popular approach is based onbinary decision diagrams (BDDs), but as shown by Cam-pailla et al. [2] linear complexity is still observed (in fact 3knoperations [2] for many).

2) Opaque subscriptions: Many, especially early, systemsadvocated for structural conformance between events andsubscriptions [25]. This leads to collapsing all subscriptionsfor all types of events into one single data-structure (e.g.,poset) thus exacerbating bottlenecks. Most systems, even suchdescribed without event types, use types in practice, but onlymuch later have the benefits of strong typing of events startedto be fully exploited [7]. Similarly, only little work exists onthe taking splitting a step further by systematically handlingsubscriptions attribute-wise (e.g., [10], [26]).

3) Large subscription summaries: The nodes of a posetcan get arbitrarily complex if subscriptions do not subsumeeach other. Consider the scenario in Figure 1c. None of thesubscriptions from the clients c1, c2, or c3 subsume each other.Hence, the lub is the disjunction of Φ1,Φ2 and Φ3. In the worstcase, if n clients connect to a broker, the lub is the disjunctionof n subscriptions, and these propagate throughout the networktowards publishers (e.g., b1 forwards its lub to b2).

III. SPLIT AND SUBSUME IN BERETTA

This section presents the principles underlying Beretta, andthe corresponding broker data-structures and algorithms.

A. Principles

Beretta involves a broker overlay network implemented ontop of TCP/IP. While the algorithms are not bound to TCP/IP,we assume in the following pairwise FIFO reliable communi-cation channels offering primitives SEND (non-blocking) andRECEIVE for presentation simplicity. For the same reason,we assume the absence of cycles in the broker network anda single broker or client per network node. This translatesdirectly to FIFO guarantees on events and subscription up-dates. There are ways to operate in the absence of theseassumptions but this makes the algorithms more complicatedwithout invalidating our contributions. By the same token,we elide node failures; broker fault-tolerance is achieved byvarious means which are orthogonal to the contributions ofthis paper.

The scalability and efficiency of Beretta are based on sixkey ideas:

1. strong typing of events,2. normalized subscriptions,3. splitting of subscriptions by event types and attributes,4. use of interval trees and hash maps,5. systematic introduction of variables into subscriptions,6. summarization approximation.In short, (1) supports (2); (1) and (2) make (3) possible; (2)

and (3) enable (4), (5) and (6).

B. Strongly Typed Events

In Beretta events are strongly typed. An event type τ denotesa set {τ1 a1,. . . ,τk ak} of named attributes ai which aretypically of primitive types τ i. An event e can thus be viewedas a set of attribute-value pairs {a1:v1,. . . ,ak:vk} as before,but e carries an identifier for its type. This information can beused by a broker to efficiently dispatch an event e and to onlyevaluate subscriptions of the respective type on e. New eventtypes can be added at runtime with only minimal propagationtime through the network. For brevity, but unambiguously, wewrite ai ∈ τ ⇔ τ

.= {. . . , τi, ai, . . .}

C. Subscription Normalization

Subscriptions in Beretta are normalized, i.e., they are repre-sented uniformly through a grammar modified as follows withrespect to that of Section II-B:

Condition′ Θ ::= a ∈ [v, v] | a ∈ {v, . . . , v}

Conditions are thus either interval queries or set inclusions.The former apply to totally ordered attribute types such asinteger or floating point, and the latter apply to enumerabletypes such as strings. Our grammar improves the efficiencyof CPS as we shall explain in Section III-E, yet is expres-sive enough to represent the subscriptions of the grammarpresented in Section II-B. For example, a predicate x>1000where x is an integer attribute, can be expressed as x∈ [1001,MAXINT], and the predicate z="IBM" can be expressed asz∈ {"IBM"}. In fact, Beretta automatically normalizes sub-scriptions of the grammar of Section II-B. Assuming an eventtype τ .= {int x, int y, string z} with three attributes, asubscription x∈ [45, 78] can be imagined as being transformedto x∈ [45, 78]∧y∈ [MIN_INT, MAX_INT]∧z∈ {*}, thusadding wildcard conditions for attributes without constraints.A key insight also for dealing with updates is that normal-ization ensures that a subscription to event type τ has exactlyone condition for every attribute ar of τ – be it a wildcardcondition. Implementation-wise, such a condition does notactually have to exist at runtime, but if the application addsa corresponding condition to the subscription then the brokerknows exactly where and how to create it based on the “new”bounds provided by the application.

D. Subscription Splitting

Each broker in the CPSN knows the set of neighboringbrokers and the set of clients (publishers and subscribers) con-nected to it. Figure 3 presents the client algorithm of Beretta.Advertisements are omitted for brevity. A client subscribes to

its edge broker by using event types (line 8). For presentationsimplicity, the figure assumes a single subscription per clientand type, and subscriptions to be already normalized. Thus asubscription is a conjunction of conditions (

∧r Θr). Thanks to

normalization the shape of any condition is known by the typeof the respective attribute (totally ordered ⇒ interval query;otherwise enumerable ⇒ set inclusion), and thus the actualconditions do not have to be transmitted to the broker at all.As every node has a single subscription for a given event type,the algorithm does not need to actually denote any variables –the node identity together with the identifier r of the attributear will allow to uniquely identify a variable based on the typeof the attribute. More precisely, vr represents the vector ofvalues for a condition Θr on attribute ar. In the case of aninterval query vr’s size is 2 as it contains both bounds (lowerbound first), while in the case of set inclusion the value set istransmitted.

The broker (see Figure 4) stores subscribers and publishersper types as well. (Advertisements and initial setup are againelided for brevity.) Subscriptions are stored in a hash mapS indexed not only by type τ but also by attribute ar (of τ ).S[τ][r] refers to a data-structure, detailed below, that representsall conditions from all subscriptions to type τ gathered fromthe broker’s neighbors. S being a hash map enables constanttime O(1) lookup. Since published events are all taggedwith their respective types this splitting allows to scale toany number of types, and does not impose restrictions oncomplexity of types.

E. Interval Trees and Hash Maps

Instead of maintaining subscriptions in a single poset asshown in Figure 1c, Beretta deals these up as mentioned.This finer granularity has benefits for all operations on andwith subscriptions – addition, removal, updates, and of courseevaluation. The modified subscription grammar and result-ing normalization enable to further reduce complexity. Moreprecisely, we organize conditions for totally ordered attributetypes (interval queries) in interval trees [14]; for enumeratedtypes (set inclusion) hash maps are used just like for sub-scription splitting. Consider the CPSN shown in Figure 1awhere a broker b1 is connected to five subscribers c1, ..., c5,and the type τ .= {int x, int y, string z} introducedearlier. Figure 2a shows the subscriptions issued by each clientattached to b1, and Figure 2b and Figure 2c show the intervaltrees associated with attributes x and y respectively. b1 receivesfive subscriptions on z. As shown in Figure 2d, Beretta uses ahash map whose key is z – each bucket stores a string v andthe set of clients/brokers subscribed to z= v.

When an event {a1:v1,. . . ,ak:vk} of type τ arrives, search-ing an interval tree for all intervals into which the valueof an attribute ar of totally ordered type falls is O(log n)in the worst case if there are n intervals in the tree, i.e.,n subscriptions for τ currently stored. For an attribute ofenumerable type, retrieving the set of matching nodes caneven be accomplished in constant time O(1) due to the useof a hash map. For an event with k attributes, the worst-case

c1

c2

c5

b1

c3c4

x ∈ [8,9] ∧y ∈ [25, 30] ∧z ∈ {IBM}

x ∈ [5

, 8] ∧

y ∈ [1

7,

19] ∧

z ∈ {M

SFT}

x ∈ [0,3] ∧y ∈ [19,

20] ∧z ∈ {YAHOO}

x ∈ [6,10] ∧y ∈

[0,

5] ∧z ∈ {AMZN}

x ∈ [15,23] ∧y ∈ [26, 26] ∧z ∈ {IBM}

(a) Subscriptions

[ 8,9 ]

[ 5, 8 ]

[ 0, 3 ] [ 6, 10 ]

[ 15, 23 ]

LUB = [ 0, 23 ]

(b) Interval tree for x

[ 25, 30 ]

[ 17, 19 ] [ 26, 26 ]

[ 0, 5 ] [ 19, 20 ]

LUB = [ 0, 26 ]

(c) Interval tree for y

hash(z)

<"IBM", {c1}><"MSFT", {c2, c3}><"YAHOO", {c4}>

<"AMZN", {c5}>...

(d) Hash map for z

Fig. 2: Examples of interval trees and hashing

complexity of event evaluation becomes O(k log n). On thecontrary, using a poset P as explained in Section II-D resultsin an event evaluation complexity of O(k |P|) = O(k n).

Figure 4 abstracts the handling of interval trees and hashmaps for attribute conditions, for brevity. Individual conditionsof an attribute ar are stored as c-nodes (condition nodes)of the form 〈vr, nj〉, where vr are the current values fornj’s condition on attribute ar. For the ordering within aninterval tree, the two bounds v are obviously used. For agiven data-structure and event, function MATCH() returns theset of clients and brokers with c-nodes in the data-structurewhose values, plugged into the corresponding condition, matchthat of the event. At line 22, the intersection of these sets istaken across all attributes of the given type to determine theforwarding set. This would further increase the complexity ofevent evaluation. In Beretta the individual c-nodes are thuslinked to each other which allows for this intersection to becreated implicitly in a way similar to the fast forwarding in[10]. Thereby we also avoid output-sensitivity, i.e., that thecomplexity of an interval tree query is conditioned by the sizeof the output set. The LUB() function simply returns the lubof a given interval tree or hash map. Functions INSERT(),DELETE(), and UPDATE() abstract the insertion, removal,and update of a given c-node respectively; these functionsimplicitly trigger a reorganization of the data-structure and anupdate of the lub if necessary. Requests for new subscriptions(SUB), unsubscriptions (UNSUB), and variable updates (UPD)are all handled similarly: if the request alters the lub of thecorresponding data-structure then this can trigger an updaterequest (line 32) to parent brokers.

F. Systematic Variables for Updates

The normalization of subscriptions in Beretta lends itselfwell to the systematic use of variables for efficient updates.For an example of variable use, consider the condition x>varwhere x is the integer attribute of the type τ introduced earlierand var is a program variable (or identified by a handle as partof an API); the condition is normalized to x∈ [var, MAXINT].An update to var can be directly propagated through thesystem. Systematically using variables involves viewing anycondition on attribute x as x∈ [var1, var2], and trackingthe current values of var1 and var2. Since we know the

1: init2: b {edge broker}

3: to PUBLISH(e) of τ do4: SEND(PUB, τ, e) to b

5: to SUBSCRIBE(∧r=1..n

Θr) to τ do6: for all ar ∈ τ do7: vr ← values in Θr

8: SEND(SUB, τ,⋃rvr) to b

9: to UNSUBSCRIBE from τ do10: SEND(UNSUB, τ ) to b

11: upon RECEIVE(PUB, τ , e) do12: if Φ(e) | Φ is subscription to τ then {maybe changed}13: DELIVER(e)

14: upon change of value in Θr for subscription to τ do15: SEND(UPD, τ, r, newvr) to b

Fig. 3: Beretta client algorithm

exact shape of conditions on a given attribute, a variable isuniquely identified in our algorithms by the node identifierni (and an index in the case of multiple subscriptions), eventtype τ , and attribute ar. In fact, all the client algorithm ofFigure 3 does is sending the edge broker “abstract” conditions,consisting in the current values of implicitly defined variables(for bounds in interval queries; sets in set inclusion) whenissuing a subscription. Corresponding c-nodes are added tointerval trees or hash maps.

Upon an update request (UPD), the corresponding c-node isfirst updated, and then the respective interval tree or hash map.If the lub c-node changes, function PROPAGATE() is invokedwhich issues an update request to any upstream brokers if anyof the bounds changed. Note that such a recursive upstreamupdate request — just like an upstream update request engen-dered by a subscription addition or removal — contains theidentity of the broker and not of the node associated with therespective lub c-node, to avoid propagating references and thuscreating dependencies. Our model implies that any addition,removal, or update of a subscription results in a set of variableupdates, recursively through brokers. In practice, the differentupdate requests for a same type sent at line 32 are bundled.

Observe that our model does not only support variable-

1: init2: subs[] {hashmap, indexed by event types τ }3: pubs[] {hashmap, indexed by event types τ }4: S[][] {hashmap, indexed by event types τ and attribute index}

5: upon RECEIVE(SUB, τ,⋃rvr) from nj do

6: for all ar ∈ τ do {insert condition for attribute r}7: subs[τ]← subs[τ] ∪ {nj} {add node to subscribers of τ }8: c-node← 〈vr, nj〉9: c-node0 = 〈v0

r, nk〉 ← LUB(S[τ][r]) {old lub}10: INSERT(S[τ][r], c-node)11: c-nodeν = 〈vνr, nl〉 ← LUB(S[τ][r]) {new lub}12: PROPAGATE(c-node0, c-nodeν , τ, r)

13: upon RECEIVE(UNSUB,τ ) from nj do14: subs[τ]← subs[τ]\{nj}15: for all ar ∈ τ do16: c-node← 〈v, nj〉 ∈ S[τ][r] {get vals for nj ’s condition}17: c-node0 = 〈x0, nk〉 ← LUB(S[τ][r]) {old LUB}18: DELETE(S[τ][r], c-node)19: c-nodeν = 〈xν , nl〉 ← LUB(S[τ][r]) {new LUB}20: PROPAGATE(c-node0, c-nodeν , τ, r)

21: upon RECEIVE(PUB, τ, e) from nj do22: for all nk ∈

⋂ar∈τ

MATCH(S[τ][r], e) do23: SEND(PUB, τ, e) to nk

24: upon RECEIVE(UPD, τ, r, (v)) from nj do25: c-node0 = 〈v0, ns〉 ← LUB(S[τ][r]) {old LUB}26: c-nodeupd ← 〈vupd, nj〉 ∈ Sτ [r]27: UPDATE(S[τ][r], c-nodeupd, x) {update the node in S[τ ][r]}28: c-nodeν = 〈vν , nq〉 ← LUB(S[τ][r]) {new LUB}29: PROPAGATE(c-node0, c-nodeν , τ, r)

30: procedure PROPAGATE(〈v0〉, 〈vν〉, τ, r)31: if ∃vνs 6= v0

s then {LUB change⇒update upstream}32: SEND(UPD, τ, vν) to all bk ∈ pubs[τ]

Fig. 4: Beretta broker algorithm as executed by bi. Commonprocessing of interval tree modifications (new subscriptions,unsubscriptions, updates) are regrouped in PROPAGATE

based updates (e.g., changes of bounds on intervals). Sincesubscriptions are normalized by including wildcard subscrip-tions for all attributes without actual conditions, this holdsequally for structural updates of subscriptions – it is as ifbehind the scenes a subscription to a type τ had a conditionfor every attribute of τ already. Condition addition (as wellas removal) just boils down to an update of the correspondingvariable(s). As mentioned earlier though, a wildcard conditiondoes not actually have to exist in memory on a broker; if avariable update is received for such a condition, the brokerknows exactly how to interpret the “update”.

G. Summarization Approximation

When sending a summary to a parent, a broker approxi-mates the summarization. Remember that a broker managesa separate data-structure S[τ][r] for each attribute ar of agiven event type τ . The lub of such a data-structure coversall conditions on the respective attribute for all subscriptionsknown to the broker. A broker summary for a given event typesimply consists in the conjunction of these lubs. In Figure 4this conjunction arises implicitly, as an update request (line 32)

contains the lub for the respective type and attribute. It is easyto see that this is an approximation. Consider the conditionsof an attribute ar of type τ for a number n of subscriptionsequalling the number k of attributes in τ (n = k). Subscriptionr’s condition on ar can be the lub for ar, while the conditionfor every other aq , q 6= r, of subscription r can be covered bythat of subscription q.

This approximation can be disabled in Beretta, e.g., forsets of summarized subscriptions below a threshold size, anda disjunction (created like in traditional approaches) sent toparent brokers instead. In this scenario, a subscription sentby a broker bi to its parent broker bj consists in fact ina set of (logically disjoined) normalized subscriptions, sayof size u, necessary to cover all subscriptions of bj . Upona change at bi to its data-structures, e.g., induced by theaddition, removal, or update of a subscription, there are twoscenarios: (a) the set of disjoined subscriptions in this rootsubscription remains the same, or (b) it changes. In the formercase, if the values of certain variables have changed onlythe updates need to be transmitted to bj . In the latter casewe can further distinguish three cases, based on whether thenumber of disjoined subscriptions in the root (b.1) grows tov, (b.2) remains u (some subscriptions in the disjunction mayhave changed though), or (b.3) shrinks to v. In all cases, weidentify min(u, v) subscriptions — preferably such that werein the previous set already — with the previous ones, and sendcorresponding variable updates where necessary. If we havemore subscriptions now (v > u) or fewer ones (v < u) thenbi additionally informs bj of the new ones to be added or ofthose to be removed. The latter actions are handled differentlyfrom regular new subscriptions on unsubscriptions, as brokerstake note of multiple subscriptions that are logically linked toa same peer or client to avoid multiple transmissions of thesame event.

IV. EVALUATION

We first evaluate Beretta on two benchmarks modeled ontwo different real-life applications – highway traffic manage-ment and algorithmic trading. Then we evaluate the scalabilityof Beretta through a statistically generated workload.

A. Metrics

Similar to [21], our evaluation uses four metrics.1) Throughput: The throughput of a CPSN is the average

number of events delivered by a subscriber per s. Through-put thus depends on: (a) the number of publishers, (b) theproduction rate of events at each publisher, (c) the updaterate for subscriptions of individual subscribers (updates canbe structural or parametric) and, (d) the selectivity of thesubscription of a subscriber. Selectivity is the probability thata event published by a publisher satisfies the subscription. Aselectivity of 1.0 implies that every published event satisfiesthe considered subscription and 0.0 implies that none do.

2) Latency: Latency refers to event dissemination latency:if an event e is produced at time t1 and is received by asubscriber at time t2, then the dissemination latency of e is

1500 20 40 60 80 100 120

30,000

0

5000

10,000

15,000

20,000

25,000

Number of edge brokers

Thro

ughp

ut (e

vent

s/s)

Siena

EV-Resub and EV-Upd

Beretta

(a) Throughput (0 updates/s) (HTM benchmark)

1500 20 40 60 80 100 120

35,000

0

5000

10,000

15,000

20,000

25,000

30,000

Number of subscribers

Thro

ughp

ut (e

vent

s/s)

Beretta

SienaEV-Upd and EV-Resub

(b) Throughput (0 updates/s) (AT benchmark)

1500 20 40 60 80 100 120

600

0

100

200

300

400

500

Number of edge brokers

Late

ncy

(ms)

Siena

EV-Resub &EV-Upd

Beretta

(c) Latency (0 updates/s) (HTM benchmark)

1600 20 40 60 80 100 120 140

400

0

50

100

150

200

250

300

350

Number of subscribers

Late

ncy

(ms)

Siena

EV-Upd and EV-Resub

Beretta

(d) Latency (0 updates/s) (AT benchmark)

50000 1000 2000 3000 4000

100,000

70,000

75,000

80,000

85,000

90,000

95,000

Number of event types

Thro

ughp

ut (e

vent

s/s) 8 conditions/subscription

6 conditions/subscription

4 conditions/subscription

2 conditions/subscription

(e) Broker throughput vs subscription complexity

10 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

130,000

60,000

70,000

80,000

90,000

100,000

110,000

120,000

Selectivity

Thro

ughp

ut (e

vent

s/s)

1000 event types2000 event types

3000 event types

4000 event types

(f) Broker throughput vs subscription selectivity

Fig. 5: Comparing Beretta’s performance with EV-Resub, EV-Upd and Siena. Figs. (a) to (d) show improvements in throughputand latency when the update frequency is 0 and the number of subscribers in AT and edge brokers in HTM is varied in theHTM and AT benchmarks. Figs. (e) and (f) show per-broker throughput on statistically generated workloads

t2−t1. Latency of e measures the timeliness of the subscriptionsubsumption, update, and event forwarding algorithms, and isa function of: (i) the time taken by a broker to decide whethere has to be discarded or sent to a downstream broker (ii)the bandwidth of the network links used to connect brokersand (iii) network traffic and congestion on broker-broker links.Although no algorithms described in this paper require ordirectly benefit from synchronized clocks, we ensured thatclocks of publishers and subscribers were synchronized whileconducting experiments that measured latency.

3) Update delay: Update delay is the time it takes for aCPSN to react to subscription updates (either structural orparametric). If a subscriber changes its subscription Φi to Φ′i attime t1 and the first event matching Φ′i but not Φi is deliveredat time t2, then t2-t1 yields an approximation of the updatedelay at the subscriber.

4) Spurious events : The effect of inefficient updates mightbe absorbed if brokers are powerful dedicated servers orindividual clients are only interested in few events . Other-wise, increased stress might manifest especially on resource-constrained clients. To gauge this, we measure the amount ofspurious events delivered by clients. If a subscriber changesits subscription Φi to Φ′i at time t1, then spurious events arethose matching Φi but not Φ′i and received by the client aftert1 and filtered out locally to it (see line 12 in Figure 3). These

capture the overhead imposed on clients.

B. Systems for Comparison

To characterize performance gains due to our novel routingmechanisms, we compare Beretta against three CPSNs:

1) Siena [10], [9], a seminal, open-source, CPSN.2) EV-Resub, i.e., EV (no subscription updates): EV is a

publish/subscribe middleware based on the Rete algo-rithm [24], [18]. The Rete algorithm is highly effectiveand highly touted for matching events to subscriptions,and has been used also, e.g., in PADRES [24] andJBoss messaging middleware [29]. It has been shown tosustain high throughput also in complex event detection(patterns of events) [18]. EV uses posets for subscriptionmanagement and the Rete algorithm for event matching.

3) EV-Upd, i.e., EV with parametric updates: To sepa-rate the performance benefits of dynamic updates fromthe performance benefits of replacing posets for eventmatching, we evaluate a version of EV supporting para-metric subscription updates.

For systems 1) and 2) without support for subscriptionupdates we emulate updates by resorting to the standardsolution of re-subscriptions: issuing a new subscription beforecanceling the outdated one.

1000 10 20 30 40 50 60 70 80 90

30,000

0

5000

10,000

15,000

20,000

25,000

Updates/s/edge broker

Thro

ughp

ut (e

vent

s/s)

Beretta

EV-Upd

EV-Resub Siena

(a) Throughput vs. update frequency

1000 10 20 30 40 50 60 70 80 90

400

0

50

100

150

200

250

300

350

Updates/s/edge broker

Late

ncy

(ms)

Siena

EV-Upd

EV-Resub

Beretta

(b) Latency vs. update frequency

1000 10 20 30 40 50 60 70 80 90

800

0

100

200

300

400

500

600

700

Updates/s/edge broker

Del

ay (m

s)

Siena

EV-ResubEV-Upd

Beretta

(c) Update delay vs. update frequency

1500 25 50 75 100 125

35,000

0

5000

10,000

15,000

20,000

25,000

30,000

Number of edge brokers

Thro

ughp

ut (e

vent

s/s)

BerettaEV-UpdEV-Resub

Siena

(d) Throughput vs. number of edge brokers

1500 25 50 75 100 125

350

0

50

100

150

200

250

300

Number of edge brokers

Late

ncy

(ms)

Siena

Beretta

EV-Resub EV-Upd

(e) Latency vs. number of edge brokers

1500 25 50 75 100 125

900

0100200300400500600700800

Number of edge brokers

Del

ay (m

s)

Siena

EV-ResubEV-Upd Beretta

(f) Update delay vs. number of edge brokers

Fig. 6: Comparing Beretta’s performance with EV-Resub, EV-Upd and Siena for the HTM benchmark by varying the updatefrequency (keeping number of edge brokers fixed at 150) in Figures (a)–(c) and the number of edge brokers (keeping updatefrequency fixed at 50 updates/s/edge broker) in Figures (d)–(f).

C. Infrastructure

All brokers were executed on commodity dual core IntelXeon 3.2Ghz machines with 4GB RAM running Linux, witheach machine executing exactly one broker. Subscribers weredeployed on a cluster, where each node is an eight core IntelXeon 1.8Ghz machine with 8GB RAM running Linux, with 8subscribers deployed on each node (one subscriber per core).Publishers were deployed on dual core Intel Pentium 3Ghzmachines with 4GB RAM, with no more than 2 publishersper machine (one publisher per core). Deploying publishers,subscribers and brokers on different nodes ensured that allrelevant communication (publisher-broker, broker-broker andsubscriber-broker) was over a network, and across LANs.10 ms delays were added to each network link to simulatewide area network characteristics as is done in EmuLab [1].

D. Real-life Benchmarks

We now evaluate Beretta by comparison against three CP-SNs based on two real-life benchmarks.

1) Highway Traffic Management (HTM): As in [32], toevaluate our algorithms, we used a traffic management systembased on [31], [16] which controls an area, with ten peripheralhighways and four highways intersecting at the middle. Thesystem we used consists of a CPSN where the publishers are

several wireless sensors and cameras located along the high-way, monitoring road conditions, traffic density, speeds, tem-perature, rainfall, snow etc. Each sensor connects wirelesslyto a base station which serves as the broker – the base stationsare connected to each other by a wired network. Subscribersare navigation systems and other location-based applications(e.g., daemons which serve location-based ads) in vehicles,computers and mobile phones. Subscribers subscribe to eventsin a certain geographical range around their current location(GPS coordinates), based on a driver (or passenger’s) inputand hence the system issues both structural and parametricupdates.

The system uses a non-hierarchical CPSN (highways in anurban area are not hierarchical) with 200 brokers (150 edgebrokers and 50 “non-edge” brokers), and 5 publishers per edgebroker, resulting in a total of 750 publishers. The distributionof operators op in subscriptions was 37% ≥, 41% ≤, and 22%=. On this benchmark, we evaluate our algorithms with updatefrequencies ranging from 10 updates/s to 100 updates/s/edgebroker. Update frequencies of 83.33 updates/s/edge broker arepretty common in traffic management – consider a trafficdensity of 500 cars/mile (250 cars in either direction) on a 10mile stretch of six lane highway controlled by an edge broker.Assuming an update frequency of 1 update/min/subscriber, up-

1000 10 20 30 40 50 60 70 80 90

35,000

0

5000

10,000

15,000

20,000

25,000

30,000

Updates/s/subscriber

Thro

ughp

ut (e

vent

s/s)

Beretta

EV-Upd

EV-ResubSiena

(a) Throughput vs. update frequency

1000 10 20 30 40 50 60 70 80 90

700

0

100

200

300

400

500

600

Updates/s/subscriber

Late

ncy

(ms)

Beretta

EV-ResubEV-Upd

Siena

(b) Latency vs. update frequency

1000 10 20 30 40 50 60 70 80 90

900

0100200300400500600700800

Updates/s/subscriber

Del

ay (m

s)

Siena

EV-ResubEV-Upd

Beretta

(c) Update delay vs. update frequency

1500 25 50 75 100 125

35,000

0

5000

10,000

15,000

20,000

25,000

30,000

Number of subscribers

Thro

ughp

ut (e

vent

s/s)

Beretta

EV-Resub

EV-Upd

Siena

(d) Throughput vs. number of subscribers

1500 25 50 75 100 125

400

0

50

100

150

200

250

300

350

Number of subscribers

Late

ncy

(ms)

Siena

EV-ResubEV-Upd

Beretta

(e) Latency vs. number of subscribers

1500 25 50 75 100 125

900

0100200300400500600700800

Number of subscribers

Del

ay (m

s) Siena EV-Resub

EV-Upd

Beretta

(f) Update delay vs. number of subscribers

Fig. 7: Comparing Beretta’s performance with EV-Resub, EV-Upd and Siena for the AT benchmark by varying the updatefrequency (keeping number of subscribers fixed at 150) in Figures (a)–(c) and the number of subscribers (keeping updatefrequency fixed at 50 updates/s/subscriber) in (d)–(f)

1000 10 20 30 40 50 60 70 80 90

1200

0

200

400

600

800

1000

Updates/s/edge broker

Spur

ious

eve

nts/s

Siena

EV-UpdEV-ResubBeretta

(a) Spurious events – HTM

1500 25 50 75 100 125

800

0

100

200

300

400

500

600

700

Number of edge brokers

Spur

ious

eve

nts/s

Siena

EV-Resub

EV-Upd

Beretta

(b) Spurious events – HTM

1000 10 20 30 40 50 60 70 80 90

1100

0100200300400500600700800900

1000

Updates/s/subscriber

Spur

ious

eve

nts/s

Siena

EV-Resub EV-UpdBeretta

(c) Spurious events – HTM

1500 25 50 75 100 125

700

0

100

200

300

400

500

600

Number of subscribers

Spur

ious

eve

nts/s Siena

EV-ResubEV-Upd

Beretta

(d) Spurious events – HTM

Fig. 8: Comparing the performance of Beretta with EV-Upd, EV-Resub and Siena with respect to the number of spuriousevents

date frequency at the corresponding edge broker is 500*10/60= 83.33.

Each publisher generates events (traffic conditions, weatherconditions, commercial advertisements, etc) at the rate of 150events/s. Each subscriber (vehicle) subscribes to less than 40%of events generated, and updates its location to the edge broker.

2) Algorithmic Trading (AT) : We consider the monitoringcomponent of an algorithmic stock trading system. We usea CPSN disseminating commodity prices with 20 brokers,10 publishers and 150 subscribers. In AT, the number ofpublishers is small – stock quotes are published by stockexchanges only. We assume that a subscriber is a computer

at an AT firm. Our benchmark had 300 event types, whichincludes the quotes of 153 stocks, analyst predictions, etc.In the experimental setup, we employed a hierarchical brokeroverlay network, which is typical in stock quote dissemination– newswires and market data systems at the top, large clearinghouses at the second level, large brokerages and trading firmsat the next level to which small brokerages and consumersconnect.

Each publisher generates events (stock quotes, analyst re-ports, insider trading information etc) at the rate of 6000events/s. Each subscriber subscribes to less than 20% of eventsgenerated – which is typical in the case of stock brokers that

Metric Increase in throughput Decrease in latency Decrease in delay Decr. in spurious eventsBenchmark HTM AT HTM AT HTM AT HTM ATBeretta Fig. 6a Fig. 7a Fig. 6b Fig. 7b Fig. 6c Fig. 7c Fig. 8a Fig. 8cvs. EV-Upd up to 1.85× up to 1.65× up to 2.36× up to 1.74× up to 2.44× up to 1.80× up to 2.36× up to 1.90×Beretta Fig. 6a Fig. 7a Fig. 6b Fig. 7b Fig. 6c Fig. 7c Fig. 8a Fig. 8cvs. EV-Resub up to 4.29× up to 2.03× up to 3.41× up to 2.40× up to 4.8× up to 3.79× up to 5.21× up to 6.42×Beretta Fig. 6a Fig. 7a Fig. 6b Fig. 7b Fig. 6c Fig. 7c Fig. 8a Fig. 8cvs. Siena more than 10× more than 10× up to 4.61× up to 3.78× up to 11.11× up to 8.45× up to 9.85× up to 11.23×

TABLE I: Comparing Beretta’s performance with EV-Resub, EV-Upd and Siena – this table shows improvements in throughput,latency, delay, and spurious events when the update frequency is varied yet number of subscribers in AT and edge brokers inHTM is fixed at 150.

Metric Increase in throughput Decrease in latency Decrease in delay Decr. in spurious eventsBenchmark HTM AT HTM AT HTM AT HTM ATBeretta Fig. 6d Fig. 7d Fig. 6e Fig. 7e Fig. 6f Fig. 7f Fig. 8b Fig. 8dvs. EV-Upd up to 2.26× up to 1.59× up to 2.32× up to 2.24× up to 1.5× up to 2.19× up to 2.34× up to 1.83×Beretta Fig. 6d Fig. 7d Fig. 6e Fig. 7e Fig. 6f Fig. 7f Fig. 8b Fig. 8dvs. EV-Resub up to 4.98× up to 1.8× up to 3.20× up to 2.81× up to 2.9× up to 4.40× up to 9.71× up to 4.09×Beretta Fig. 6d Fig. 7d Fig. 6e Fig. 7e Fig. 6f Fig. 7f Fig. 8b Fig. 8dvs. Siena more than 10× more than 10× up to 4.45× up to 4.60× up to 15.34× up to 10.13× up to 12.5× up to 10×

TABLE II: Comparing Beretta’s performance with EV-Resub, EV-Upd and Siena – this table shows improvements in throughput,latency, delay, and spurious events when the update frequency is fixed and the number of subscribers in AT and edge brokersin HTM is varied. Update frequency is 50 updates/s/subscriber for AT and 50 updates/s/edge broker for HTM

consume a fraction of the information.

E. Results and Analysis – 0 Updates/sFigures 5a,5b,5c and 5d and Table III describe the per-

formance of Beretta vis-a-vis EV-Resub, EV-Upd and Sienawhen there are no subscription updates, either structural orparametric.

Increase in throughput Decrease in latencyBenchmark HTM AT HTM ATBeretta vs. 1.48× 1.69× 2.39× 2.48×EV-Resub/EV-Upd Fig. 5a Fig. 5b Fig. 5c Fig. 5dBeretta vs. 117× 132× 6.57× 7.34×Siena Fig. 5a Fig. 5b Fig. 5c Fig. 5d

TABLE III: Performance of Beretta vs. EV-Resub and EV-Updwhen there are no subscription updates

The throughput of Beretta is more than 100× the throughputof Siena, which demonstrates the effect of reducing eventmatching from O(k n) to O(k log n). This also decreases eventdissemination latency by more than 5× in both benchmarks.Figures 5a and 5b show that the throughput of Beretta is upto 1.48× the throughput of EV-Resub or EV-Upd for the ATbenchmark and 1.69× the throughput for the HTM benchmark.When there are no updates, the throughput of EV-Resub andEV-Upd are almost equal, as expected. This proves that Berettaimproves on the highly effective Rete algorithm by more than48%, despite Rete’s construction of a separate event flowgraph. We also note that Rete is memory intensive but Berettais not.

F. Results and Analysis (Broker-level)Figures 5e and 5f show that Beretta retains a high per-

broker throughput irrespective of the selectivity of the sub-

scriptions and the complexity (number of conditions) of thesubscription. Figures 5e and 5f measure per-broker throughputunder various statistically generated workloads, and they donot correspond to either the HTM or AT benchmarks. InFigure 5e, we measured the throughput of a Beretta brokerwhile simultaneously increasing the number of event typesand the number of conditions per subscription (predicate).The total number of subscriptions was 100,000, which wereuniformly distributed over all event types i.e. when there are100,000 subscriptions and 5000 event types, there were 20,000subscriptions per event type. Though the throughput of Berettadecreases by ∼25% when the number of event types involvedis increased from 100-1000, the throughput stabilizes anddecreases only by ∼6% when the number of event types isincreased from 1000-5000. Similarly, Figure 5f shows thatthroughput decreases as the selectivity of the subscriptiondecreases (from 0 to 1). Recall that selectivity of a subscriptionrefers to the probability that a subscription matches an event.When a subscription is highly selective, more events arefiltered out by Beretta’s algorithms, and consequently moreevents can be “pushed through” a broker, resulting in higherthroughput. When selectivity decreases and reaches 1, everyevent is matched to some subscription, which leads to moreconditions being evaluated on each event, thereby decreasingthroughput.

G. Results and Analysis (With Updates)

To gauge the performance of Beretta under high rates ofchurn, i.e., the modification and removal of subscriptions,we evaluated it against the three other CPSNs, under twoscenarios. First, we varied the update frequency at eachsubscriber (in AT) and edge broker (in HTM) from 0 updates/s

to 100 updates/s, while keeping the number of subscribers(in AT) and the number of edge brokers (in HTM) constantat 150. The updates were 40% parametric updates and 60%structural updates. Table I summarizes the improvements inperformance of Beretta vis-a-vis EV-Upd, EV-Resub and Sienaunder increasing update frequencies. Then, we measured theperformance of Beretta with an increase in the number ofsubscribers (in AT) or edge brokers (in HTM) while keepingupdate frequency constant at 50 updates/s, and the perfor-mance improvements under this scenario are summarized inII. From Figures 6, 7 and 8, we observe the following:

1) Increased throughput: From Figures 6 and 7 we observethat update support significantly increases event processingthroughput of a CPSN with an increasing frequency of sub-scription updates at a subscriber or edge broker respectively.Resubscriptions lead to thrashing – where a CPSN expendsmore time and resources tearing down and rebuilding posetsand other data-structures used in routing subscriptions therebyseverely limiting resources available at brokers for filteringand forwarding events. Thrashing is especially evident in thedrastic drop of Siena’s throughput (Figures 6a, 6d, 7a, and7d). The O(k n) processing time for subscription managementdoes not help either. We also observe that the choice of theforwarding algorithm does affect throughput, and the extent towhich it decreases.

The throughput of Siena is low because it uses the posetfor forwarding events. But the throughput of EV-Resub andEV-Upd are higher because they use the Rete algorithm. Thedecrease in the throughput of EV-Resub, inspite of usingRete can be explained by thrashing. It can also be observedthat throughput decreases with an increase in the number ofsubscribers (and consequently, an increase in the number ofsubscriptions).

2) Reduced latency: Event dissemination latency increaseswith an increase in update frequencies in all systems (Fig-ures 6b, 7b, 6e and 7e). This is due to the sharing of resources,between routing events and processing subscription updates,of each broker on the path of an event from publisher toa subscriber at increased update frequencies. But, in bothbenchmarks, the latency of a CPSN that uses resubscriptionsto update subscriptions increases at a much faster rate thanthat of a CPSN with inherent support for updates.

3) Reduced update delay: Figures 6 and 7 illustrate thestrong benefits in terms of decreased delay in responding tosubscription updates with inherent update support. Beyond theraw figures on the decrease in delay (up to 2.44× vs. EV-Upd,up to 4.8× vs. EV-Resub and up to 15.34× vs. Siena), weobserve that this decrease is more pronounced as the updatefrequency increases or as the number of subscribers increases.This is because resubscriptions can potentially involve twooperations on the poset – INSERT and DELETE, whereas sys-tematically introducing variables into intervals and updatingthem only involves one operation. Since both INSERT andDELETE have to acquire a lock on P[τ ] (in both versions ofEV) or interval tree (in Beretta), replacing two operations byone reduces the contention on either of these data structures

by half, thereby significantly decreasing delay. The decreasein delay is greater than 2× because update messages not onlyreduce contention at edge brokers but also at upstream brokerson the path to the publisher.

4) Reduced number of spurious events: Figure 8 shows thatBeretta significantly decreases the number of spurious eventsreceived by a subscriber compared to Siena, EV-Resub or EV-Upd in both benchmarks. In fact, Beretta delivers up to 2.34×fewer spurious events than EV-Upd, which supports parametricupdates. This demonstrates the agility of Beretta’s subscriptionmanagement and event forwarding algorithms.

V. RELATED WORK

To the best of our knowledge, Beretta is the first CPSN withprovable logarithmic matching complexity and demonstratingthat it can sustain high throughput under a high frequency ofparametric as well as structural subscription updates. Severalseminal CPS systems have already been discussed earlier inthis paper and thus we refrain from repeating those here.In general, many CPSN systems have been proposed andenumerating all of them is impossible in the allocated space.2

Gryphon [3] constructs an overlay graph such that individualattribute matches only occur on one node. This requires knowl-edge of all subscriptions and updates if any of those change,and achieves O(nλ) complexity on brokers where λ dependson actual subscriptions, coming close to 1/2 at times [3].These support only equality comparisons. Li et al [23], [24]propose subscription covering, merging, and content match-ing algorithms based on a modified BDD representation inPADRES. XSiena [36] uses Bloom filters for subscriptionmanagement and event forwarding, but as proven in [36],the complexity of subscription addition/removal is O(2m),where m is the number of bits required to store a conditionin a subscription (i.e., filter in XSiena terminology). Bloomfilters also require O(2m) space [36]. XSiena became openlyavailable very recently and we plan to compare it empiricallyto Beretta. Meghdoot [20] uses a distributed hashtable (DHT)to determine the location of subscriptions and to route eventsto the subscribers. The partitioning of the DHT across peersallows Meghdoot to eliminate the need of brokers, however,the design is inflexible when the schema is dynamic as itrequires the complete cartesian space to be reconstructed.HERMES [28] provides type- and attribute-based routing.GREEN [32] is a highly configurable and re-configurable pub-lish/subscribe middleware for pervasive computing applica-tions. Adaptation to varying underlying infrastructures, whichthe authors focus on, is very important in such environments.These efforts are complementary to Beretta, which focuses onadapting to application changes, i.e., subscriptions.

Topic-based publish/subscribe (TPS) provides limited ex-pressiveness compared to CPS, as a subscriber receives allmessages in a topic. Beretta is able to encode even hierar-chical topics. Examples of TPS systems are SCRIBE [30],

2http://event-based.org/link-collection/ provides an exhaustive view of litera-ture on the topic.

Bayeux [37], and Spidercast [12]. [11] is a recent divide andconquer approach for TPS.

Astrolabe [34] can be instantiated as a CPSN. With an em-phasis on fault tolerance, nodes periodically exchange mem-bership information. This information includes interests, whichare aggregated based on topological or logical constraints.Nodes are selected to represent others based on the samecriteria, leading to an overlay hierarchy. Interests are expressedlike SQL queries, which can be applied to multicast events aswell as to data stored on the overlay network. In that sense itis more like a content distribution network (CDN) (e.g. CoralCDN [13], Akamai [5] and Amazon CloudFront [6]). A CDNis a system of computers containing copies of data, placedat various points in a network so as to maximize bandwidthfor access to the data from clients throughout the network.A client accesses a copy of the data near to the client, asopposed to all clients accessing the same central server, soas to avoid bottlenecks. CDNs consist of web caches (thatstore the most frequently accessed objects), software loadbalancers, hardware load balancers (layer 4-7 switches) andrequest routing infrastructures. CDNs are orthogonal to CPSNs– they solve different problems.

Recently, there have been proposals for a clean-slate content-centric redesign of the Internet such asProject CCNx [27] and Data-Oriented Network Architecture(DONA) [22]. Both CCNx and DONA combine the bestaspects of CDNs and CPSNs, along with inherent support forconfidentiality, data- persistence, availability and authenticity.

VI. CONCLUSIONS

We have presented Beretta, a highly scalable and efficientCPSN with inherent support for subscription updates (para-metric as well as structural). The benefits of Beretta hinge ona combination of several key ideas, rooting in strong messagetyping and a novel model of normalized subscriptions. Wehave illustrated the benefits of our techniques through real-lifeand statistical workloads. We are currently extending Berettawith support for message pattern detection.

REFERENCES

[1] Emulab – Total Network Testbed. http://www.emulab.net.[2] A. Campailla and S. Chaki and E. Clarke and S. Jha and H. Veith.

Efficient Filtering in Publish-Subscribe Systems Using Binary DecisionDiagrams. In ICSE ’01.

[3] M. Aguilera, R. Strom, D. Sturman, M. Astley, and T. Chandra.Matching Events in a Content-Based Subscription System. In PODC’99.

[4] Aite Group. Algorithmic Trading: Hype or Reality? http://www.aitegroup.com/reports/20050328.php, 2005.

[5] Akamai. Akamai HD Network. http://www.akamai.com/html/solutions/hdnetwork.html.

[6] Amazon. Amazon CloudFront. http://aws.amazon.com/cloudfront/.[7] J. Bacon, K. Moody, J. Bates, R. Hayton, C. Ma, A. McNeil, O. Seidel,

and M. Spiteri. Generic Support for Distributed Applications. IEEEComputer, 33(3).

[8] C. .L. Forgy. On the Efficient Implementation of Production Systems.[9] A. Carzaniga, M. J. Rutherford, and A. L. Wolf. A Routing Scheme for

Content-based Networking. In INFOCOM ’04.[10] A. Carzaniga and A. Wolf. Forwarding in a Content-based Network. In

SIGCOMM ’03.[11] C. Chen, H.-A. Jacobsen, and R. Vitenberg. Divide and Conquer

Algorithms for Publish/Subscribe Overlay Design. In ICDCS 2010.

[12] G. Chockler, R. Melamed, Y. Tock, and R. Vitenberg. SpiderCast: aScalable Interest-aware Overlay for Topic-based Pub/Sub Communica-tion. In DEBS ’07.

[13] Coral. The Coral Content Distribution Network. http://www.coralcdn.org.[14] T. H. Cormen, R. L. Rivest, C. Leiserson, and C. H. Stein. Introduction

to Algorithms, 2010.[15] G. Cugola, E. Di Nitto, and A. Fuggetta. The JEDI Event-Based

Infrastructure and Its Application to the Development of the OPSSWFMS. IEEE TSE, 27(9), 2001.

[16] D. Barnett. Publish-Subscribe Model connects Tokyo Highways. http://www.industrial-embedded.com/articles/barnett/.

[17] S. Deering and D. Cheriton. Multicast Routing in Datagram Internet-works and Extended LANs. ACM TOCS, 8(2).

[18] P. Eugster and K. R. Jayaram. EventJava: An Extension of Java forEvent Correlation. In ECOOP’09.

[19] L. Fiege, F. Gartner, O. Kasten, and A. Zeidler. Supporting Mobility inContent-Based Publish/Subscribe Middleware. In Middleware ’03.

[20] A. Gupta, O. Sahin, D. Agrawal, and A. Abbadi. Meghdoot: Content-based Publish/Subscribe over P2P Networks. In Middleware ’04.

[21] K. R. Jayaram and C. Jayalath and P. Eugster. Parametric Subscriptionsfor Content-based Publish/Subscribe Networks. In MIDDLEWARE 2010.

[22] T. Koponen, M. Chawla, B.-G. Chun, A. Ermolinskiy, K. Kim,S. Shenker, and I. Stoica. A Data-oriented (and Beyond) NetworkArchitecture. In SIGCOMM 2007.

[23] G. Li, S. Hou, and H.-A. Jacobsen. A Unified Approach to Routing,Covering and Merging in Publish/Subscribe Systems Based on ModifiedBinary Decision Diagrams. In ICDCS ’05.

[24] G. Li, V. Muthusamy, and H.-A. Jacobsen. Adaptive Content-basedRouting in General Overlay Topologies. In Middleware ’08.

[25] B. Oki, M. Pfluegl, A. Siegel, and D. Skeen. The Information Bus: anArchitecture for Extensible Distributed Systems. In SOSP ’93.

[26] P. Triantafillou and A. A. Economides. Subscription Summarization: ANew Paradigm for Efficient Publish/Subscribe Systems. In ICDCS’04.

[27] Palo Alto Research Center. Project CCNx. http://www.ccnx.org/.[28] P. Pietzuch, B. Shand, and J. Bacon. A Framework for Event Compo-

sition in Distributed Systems. In Middleware ’03.[29] Red Hat Inc. JBoss Messaging Middleware. 2010.[30] A. Rowstron, A.-M. Kermarrec, M. Castro, and P. Druschel. SCRIBE:

The Design of a Large-Scale Event Notification Infrastructure. InNetworked Group Communication 2001.

[31] S. Schneider. DDS and Distributed Data-centric Embedded Systems.http://www.drdobbs.com/embedded-systems/196601852.

[32] T. Sivaharan and G. S. Blair and G. Coulson. GREEN: A Configurableand Re-configurable Publish-Subscribe Middleware for Pervasive Com-puting. In OTM Conferences (1), 2005.

[33] The Economist. Moving Markets: Shifts in Trading Patterns areMaking Technology Ever More Important. http://www.economist.com/business-finance/displaystory.cfm?story id=E1 VQSVPRT, 2006.

[34] R. van Renesse, K. Birman, and W. Vogels. Astrolabe: A Robust andScalable Technology for Distributed System Monitoring, Management,and Data Mining. ACM TOCS, 21(2), 2003.

[35] Y. Huang and H. Garcia-Molina. Parameterized Subscriptions inPublish/Subscribe Systems. Data and Knowledge Eng., 60:435–450,2007.

[36] Z. Jerzak and C. Fetzer. Bloom Filter Based Routing for Content-basedPublish/Subscribe. In DEBS ’08.

[37] S. Zhuang, B. Zhao, A. Joseph, R. Katz, and J. Kubiatowicz. Bayeux:an Architecture for Scalable and Fault-tolerant Wide-Area Data Dissem-ination. In NOSSDAV ’01.