Stream Processing Blocks

Stream Processing Blocks

Dario BoninoPolitecnico di Torino, Dipartimento di Automatica ed Informatica

Corso Duca degli Abruzzi 24, 10129 - Torino, [email protected]

February 2, 2012

1 Abstract

This report describes the foundations of the event delivery system implementedby the Jeerp Collector module and provides an up-to-date reference on howto design, configure and run real-time processing for event delivery based oncomputation and detection blocks. Event streams are managed by exploitingmodular processing blocks able to perform complex computations on real-timedata streams coming from on-the-field measurement instruments.

The following paragraphs define the stream processing block structure andterminology, and introduce the specific blocks adopted in the Jeerp project.

2 Terminology

2.1 Event Stream

An event stream is a possibly infinite temporal sequence of discrete events, e.g.,measures coming from a real meter, data flowing from sensors, stock quotes,etc., or data sequences computed starting from such sources. Each event carriesa simple payload, either a real number (see Appendix A.4 for more details onunit of measure specification) or a Boolean (0/1) value, and is marked with atime stamp. Event streams are usually generated with a possibly high rate, andthey usually cannot be re-played or buffered. These peculiar features imposestrict requirements on stream processing applications, such as:

• High throughput – applications often need to process large volumes ofevents (between 1,000 to 100k events per second)

• Low latency – applications might need to react in real-time to conditionsthat occur (from a few milliseconds to a few seconds)

• Support to complex computations – applications might need to detectpatterns among events (event correlation), filter events, aggregate time orlength windows of events, join event streams, trigger based on the absenceof events etc.

1

2.2 Stream Processing Block

A stream processing block is a (software) component taking one or more eventstreams in input and generating one or more event streams as output. Theoutput and input streams are correlated by means of a processing function,which, in general, is not linear (e.g., threshold) and/or with memory (e.g., amoving average). A block is:

• transparent if the block only filters out a subset of entering events, depend-ing on some block parameter, while it is called opaque if it computes/gen-erates new event values;

• linear if the processing function only applies linear operators to the inputstream, while it is not linear if not linear operators are adopted;

• with memory if the computations made for generating the output streamdepends on past events, while it’s without memory if the processing func-tion does not depend on past events.

Stream Processing

Block

Input Output

Parameters

port n

port 1port 1

port n

Figure 1: Generic Stream Processing Block

Figure 1 shows the general structure of a stream processing block. A streamprocessing block has a set of input ports, and a set of output ports, identifiedby unique port identifiers (mnemonic strings). Every port can only handle aspecific type of event, i.e., it has an associated data-type, either real-valuedor Boolean, that shall match the type of events received (generated) in input(output).

A set of constant parameters can be defined to affect/tune the inner blockfunctionality, e.g., values, window lengths, operating modes.

2.2.1 Chains

Stream processing blocks can be interconnected to form more complex elabo-ration chains (cycles are not allowed). Chains can be seen as user-definableprocessing blocks whose inner functionality is defined through block intercon-nection, see Figure 2.

2

B1 B2 B3

B5 B6 B7

in out

out2

Chain 1

Figure 2: A generic stream processing chain seen as a block.

Figure 3: Moving window example.

2.3 Time windows

2.3.1 Moving Window

A moving window is a time window extending to the specified time interval intothe past based on the system time. Moving windows enable to limit the numberof events considered by a processing block. As a practical example, considerthe need to determine all accounts where the average withdrawal amount peraccount for the last 4 seconds of withdrawals is greater then 1000. The diagramin Figure 3 illustrates the behavior of a moving window. The diagram starts ata given time t and displays the contents of the time window at t + 4 and t + 5seconds and so on.

The activity as illustrated by the diagram is:

1. At time t + 4 seconds an event W1 arrives and enters the moving window.

2. At time t + 5 seconds an event W2 arrives and enters the moving window.

3. At time t + 6.5 seconds an event W3 arrives and enters the moving win-dow.

3

Figure 4: Batch window example.

4. At time t + 8 seconds event W1 leaves the time window.

2.3.2 Batch Window

The batch window buffers events and releases them every specified time intervalin one update. The diagram in Figure 4 serves to illustrate the functioning of abatch window. The diagram starts at a given time t and displays the contentsof the time window at t + 4 and t + 5 seconds and onwards.

The activity as illustrated by the diagram is:

1. At time t + 1 seconds an event W1 arrives and enters the batch.

2. At time t + 3 seconds an event W2 arrives and enters the batch.

3. At time t + 4 seconds the batch window ends and a new batch is started.

4. At time t + 6.5 seconds an event W3 arrives and enters the batch.

5. At time t + 8 seconds the batch window ends and a new batch is started.

3 Jeerp Block Types

The Jeerp Collector exploits an extensible core set of processing blocks, whichcan be combined for obtaining complex event elaboration chains.

Eleven different blocks are currently provided (see Table 1), and their func-tionalities and specific settings are reported in the following sub-sections. Theycan be combined by connecting one output port of a given block port with aninput port of another one, according to the data type consistency constraintssummarized in the connection matrix defined in Table 2.

The Jeerp block set mainly includes elements with only one or two inputports. While for Jeerp-specific tasks this core set satisfies most event processingrequirements, and multiple inputs are handled by composing core blocks in

4

Table 1: Jeerp Core Blocks

Block name Type #IN #OUT #P

Last transparent, linear, with memory 1 2 1Average opaque, non-linear, with memory 1 2 2Sum opaque, linear, with memory 1 2 2Threshold opaque, non-linear, with memory 1 1 2Hysteresis Thr. opaque, non-linear, with memory 1 1 3Time Guard opaque, non-linear 1 1 2Range opaque, non-linear, with memory 1 1 4Time Filter transparent, linear 1 1 1And opaque, non-linear, without memory n 1 1Join transparent, linear, without memory n 1 0Delta opaque, linear, without memory 1 1 0Scale opaque, linear, without memory 1 1 1

#IN = number of input ports, #OUT = number of output ports, #P = number ofparameters.

multiple input chains, other domains might require the development of blocksbeing inherently multi-port. The latter case can be successfully addressed byextending the set of proposed stream processing blocks.

Table 2: Block connection matrix.

PPPPPPFromto

Last

Average

Sum

Abs

Threshold

Hysteresis

Thr.

Time

Guard

Range

Time

Filter

And

Join

Delta

Scale

Last 3 3 3 3 3 3 3 3 3 3 3 3 3

Average 3 3 3 3 3 3 3 3 3 3 3 3 3

Sum 3 3 3 3 3 3 3 3 3 3 3 3 3

Abs 3 3 3 3 3 3 3 3 3 3 3 3 3

Threshold 3 7 3 7 7 7 3 7 3 3 3 7 7

Hysteresis Thr. 3 7 3 7 7 7 3 7 3 3 3 7 7

Time Guard 3 7 3 7 7 7 3 7 3 3 3 7 7

Range 3 7 3 7 7 7 3 7 3 3 3 7 7

Time Filter 3 3 3 3 3 3 3 3 3 3 3 3 3

And 3 3 3 3 3 3 3 3 3 3 3 3 3

Join 3 3 3 3 3 3 3 3 3 3 3 3 3

Delta 3 3 3 3 3 3 3 3 3 3 3 3 3

Scale 3 3 3 3 3 3 3 3 3 3 3 3 3

3: ports have compatible datatypes, therefore blocks can be connected; 7: port datatypesare incompatible;

3.1 Last

The Last block operates a transparent batch-window filtering of a given inputstream. Transparent filtering means that no modification is introduced into thestream events which are simply discarded on the basis of a temporal window w.In particular, given the time window w, whenever w expires this block providesas output the last received event within the window. Every time a filteringwindow expires, events in the window are dropped and a new batch filtering is

5

started (see Figure 5).The Last block additionally provides and a delayed output stream (prev)

shifted one time window back with respect to the main output stream (Figure 5).

t0 w 2w

filtering window

filtering window

e1 e2 e3 e4 e5 e6 e7 e8 e9 e10 e11

t0 w 2w

e6 e11

in

out

t0 w 2w

e6 prev

Figure 5: The full Last block temporal behavior.

From the modeling viewpoint, the Last block can be defined as shown inFigure 6. It features only one input port, identified by the port id in, andtwo output ports, respectively identified by ids out and prev. The out portprovides the main output stream, while prev conveys delayed information toother connecting blocks. The type of the output ports (out and prev) is areal number, like the type of the input port in. One set-up parameter allowsspecifying the time window w (see A.2 for the corresponding grammar) withwhich specific block instances shall operate.

Lastin

out

prev

w

Figure 6: The Last block model.

3.2 Sum

The sum block is an opaque block computing the sum of all events1 received ina given time window w. It can either operate in batch or moving mode.

1in case of Boolean events, they are mapped to the {0,1} set and summed as integers.

6

t

filtering window

0 w 2w

filtering window

filtering window

e1 e2 e3 e4 e5 e6 e7 e8 e9 e10 e11

t0 w 2w

in

out

t0 w 2w

sum(e1,...,e6) sum(e7,...,e11)

out

sum(e1,...,e6)

sum(e2,...,e7) sum(e7,...,e11)

t0 w 2w

sum(e1,...,e6)

t0 w 2w

sum(e1,...,e6)

sum(e2,...,e7) sum(e6,...,e10)

prev

prev

BA

TC

HM

OV

ING

Figure 7: The full Sum temporal behavior

Sumin

out

prev

w mode:{batch,moving}

Figure 8: The Sum block model

7

3.3 Average

The Average block is an opaque block computing the average of all (real-valued)events received in a given time window w. It can either operate in batch ormoving mode. The former computes the average of all events in the windowat repeating intervals (see Section 2.3.2). The latter, instead, computes themoving average of incoming events, over a window of lenght w and for everynew received event, a new updated value of the average is output. Figure 9shows the block temporal behavior in both modes.

t

filtering window

0 w 2w

filtering window

filtering window

e1 e2 e3 e4 e5 e6 e7 e8 e9 e10 e11

t0 w 2w

in

out

t0 w 2w

avg(e1,...,e6) avg(e7,...,e11)

out

avg(e1,...,e6)

avg(e2,...,e7) avg(e7,...,e11)

t0 w 2w

avg(e1,...,e6)

t0 w 2w

avg(e1,...,e6)

avg(e2,...,e7) avg(e6,...,e10)

prev

prev

BA

TC

HM

OV

ING

Figure 9: The full Average temporal behavior

Similarly to the Last block, also the average block provides an auxiliaryoutput channel (prev), with delayed stream events. This block requires 2 con-figuration parameters: the time window w and the operating mode, either batchor moving. Figure 10 reports the corresponding block diagram.

3.4 Abs

The Abs block is an opaque block, without memory, that simply provides asoutput (real-valued, positive) the absolute value of incoming real-valued events,see Figure 11 for the block model.

8

Averagein

out

prev

w mode:{batch,moving}

Figure 10: The Average block model

xin out

Figure 11: The Abs block model

3.5 Threshold

The Threshold block is a non linear, with memory, opaque, block that gener-ates a stream of Boolean events by monitoring one real-valued event stream forspecific threshold passing. Whenever the real-valued stream in input reachesthe parameter-set threshold value, a threshold traversal event (Boolean) is gen-erated, depending on the threshold detection mode set by means of the modeparameter. The Threshold block can work in 3 different threshold checkingmodes:

rising (>). In the rising mode a new threshold traversal event is generated ifan event value exceeds (>) the set threshold, given that previous eventvalue was under the threshold (rising-edge activation).

falling (<). In the falling mode, instead, a threshold traversal event is gener-ated if event values become lower (<) than the block threshold parameter,given that previous one was greater (falling-edge activation).

both. Finally, in the both mode, traversal events are triggered at every thresh-old crossing, whether falling or rising.

Figure 12 reports the block temporal behavior for all operating modes.From the block diagram viewpoint, the Threshold block has only one input

port (in) and one output port (out) and it accepts 2 parameters: the thresholdvalue, depending on the type of events being analyzed (real-valued with a specificunit of measure), and the mode parameter, assuming only the rising, falling andboth string values (see Figure 13 for the corresponding diagram).

3.6 Hysteresis Threshold

The Hysteresis Threshold block acts almost like the threshold block except forthe threshold traversal detection, which exploits a tunable hysteresis parameterto smooth frequent near-threshold triggers that might make the output signaltoo unstable. In a sense, the Threshold block can be seen as a particular case ofthis hysteresis block, with the hysteresis parameter set at 0. Figure 14 shows theblock temporal behavior, in rising, falling and both modes, whereas Figure 15

9

t0

e1

e2 e3e4 e5 e6

e7e8

e9 e10

e11

t0

in

out

RISIN

G

1

Threshold

value

t0

out

FALLIN

G

1

t0

out

BOTH

1

Figure 12: The full Threshold block temporal behavior.

Thresholdin out

threshold mode:{rising,falling, both}

Figure 13: The Threshold block model.

10

reports the corresponding block diagram. The hysteresis parameter defines theamplitude of the whole hysteresis band, centered around the threshold value.

t0

e1

e2 e3e4 e5 e6

e7e8

e9 e10

e11

t0

in

out

RISIN

G

1

Threshold

value

t0

out

FALLIN

G

1

t0

out

BOTH

1

hysteresis

Figure 14: The full Hysteresis Threshold block temporal behavior

Hysteresis

Thresholdin out

threshold mode:{rising,falling, both}

hysteresis

Figure 15: The Hysteresis Threshold block model.

3.7 Time Guard

The time guard is a non-linear, opaque block generating a Boolean event streamby monitoring a stream of Boolean or real-valued events for time limit/fre-quency compliance. The time guard block has two operating modes: missingand present. In the former it checks if no event has been received in the given

11

guard time tg, i.e., if an expected event is missing. Whereas in the latter itchecks if at least one event has been received before the given tg, i.e. if anexpected/unexpected event is present. It can either work by using a batch (seeFigure 16) or moving (Figure 17) guard window.

t0

e1

e2 e3

e4 e5 e6

e7e8

e9 e10

e11

t0

in

out

1

t0

out

1

tg

tg tg tg tg

tg tg tg

MISSING

PRESENT

Figure 16: The batch Time Guard temporal behavior

From the block model viewpoint (Figure 18) the Time Guard block hasonly one input (port id in) and one output (port id out) and it requires 3configuration parameters: the working mode, either present or missing, thewindowing mode, either batch or moving, and the guard time tg.

3.8 And

The And block is a non linear and opaque block which acts as a multi-porttime guard. Given a time window tw, usually short (around few seconds orless) the block provides one output event (Boolean) iff all input ports havereceived at least one event in tw. Although the And block typically works withBoolean events as input, it may also be exploited on real-valued events, in thatcase it can be used to enforce short delays between different channels (pseudo-synchronization). Figure 19 shows the corresponding temporal behavior whereasFigure 20 shows the block diagram;2

3.9 Range

The Range block is an opaque, non-linear block that checks real-valued inputevents against a range of accepted values (either in-range or out-range checkingis supported). Whenever an event value enters/leaves the range a new Boolean

2It is important to notice that in the first release And blocks have only 2 input ports.

12

t0

e1

e2 e3

e4 e5 e6

e7e8

e9 e10

e11

t0

in

out

1

t0

out

1

MISSING

PRESENT

tg

tg tg tgtg

tgtg

tgtg

tg

Figure 17: The moving window Time Guard temporal behavior

Time

Guardin out

windowing:

{batch,moving}

mode:

{missing,present}

Time_guard (tg)

Figure 18: The Time Guard block model

13

t0

t0

in1

t0

t0

in2

in3

out

tw

1

Figure 19: The And block temporal behavior.

And

1

out

time_window

2

n

Figure 20: The And block model.

14

event is generated on the block output stream. Figure 21 shows the blockbehavior in the in-range operating mode while the out-range operating mode isshown in Figure 22.

t0

e1

e2 e3e4 e5 e6

e7e8

e9 e10

e11

t0

in

out

ENTERING

1

HThreshold

value

t0

out

LEAVING

1

t0

out

BOTH

1

LThreshold

value

range

IN IN INOUT OUT OUT

Figure 21: The Range block temporal behavior (in-range mode).

It can be easily noticed that out-range and in-range operations are dual, i.e.,the Entering event of the in-range checker is equivalent to the Leaving event ofthe out-range checker. The Range block diagram is reported in Figure 23 andincludes one input (port id in), one output (port id out) and 4 configurationparameters:

1. HThreshold, defining the range high boundary,

2. LThreshold, defining the range low boundary,

3. range mode, either in-range or out-range,

4. mode, either entering, leaving or both.

3.10 Time filter

The time filter is a linear, transparent block which basically acts as a time-basedswitch, allowing incoming events to pass through depending on the current time.

15

t0

e1

e2 e3e4 e5 e6

e7e8

e9 e10

e11

t0

in

out

ENTERING

1

HThreshold

value

t0

out

LEAVING

1

t0

out

BOTH

1

LThreshold

value

range

IN IN INOUT OUT OUT

Figure 22: The Range block temporal behavior (out-range mode).

Rangein out

range_mode:

{in-range,

out-range}

mode:

{entering,

leaving, both}

HThreshold LThreshold

Figure 23: The Range block model

16

If the current time is included in the provided validity interval, events are simplypropagated from the input to the output channel, otherwise they are dropped,without any computation involving event values. As a consequence, the TimeFilter block can seamlessly handle different types of event streams, e.g., Booleanand real-valued. Figure 24 shows the temporal behavior of this block whileFigure 25 reports the block diagram.

t0

e1

e2 e3

e4 e5 e6

e7e8

e9 e10

e11

t0

in

out

e4 e5 e6

e7e8

e9

Figure 24: The Time Filter temporal behavior.

Interval specification can be achieved by adopting several standards. In thefirst release, a string encoding a cron-based interval specification is adopted forthe sake of simplicity (see A.3).

Time

Filterin out

valid interval

Figure 25: The Time Filter block model.

3.11 Join

The Join block is a linear, transparent block that multiplexes input events ondifferent channels into a single output channel. It works with an event basedparadigm: whenever an event arrives on any input port, it is automaticallyforwarded to the output port. The order with which events are received ininput is maintained in the output channel, thus allowing to perform order-basedoperations in subsequent blocks. Figure 26 shows the block temporal behavior,while Figure 27 shows the corresponding block diagram.

Although the join block might allow multiplexing of events having a differentnature, i.e., real or Boolean events, it is advisable to avoid mixing different eventtypes, especially in cases where subsequent stream processing blocks are not ableto correctly deal with such a mixed stream. For example, averaging the streamresulting from the Join of a real-valued event stream and of a Boolean valuedone is clearly a non-sense and might lead to unpredictable results.

17

t0

t0

Join port1

input stream

t0

t0

Join port2

input stream

Join port3

input stream

Join

output stream

Figure 26: The Join block temporal behavior.

Join

1

out2

n

Figure 27: The Join block model.

18

3.12 Delta

The Delta block is a linear, opaque block that computes the difference betweenpairs of consecutive events arriving on the block real-valued input port. Eventsparticipating in the difference are discarded one at time, i.e., the block workswith a sampling window having a width of 2 samples. Whenever a new eventarrives on the input, the oldest window event is discarded and difference iscomputed between the two events in the sampling window, subtracting thevalue of the old event from the value of the new event (in the window). Theblock temporal behavior is shown in Figure 28, while the corresponding diagramis depicted in Figure 29.

t0

e1 e2 e3 e4 e5 e6 e7 e8 e9 e10 e11 Delta

input stream

t0

Delta

output stream

e2-e1e3-e2

e4-e3

e11-e10

Figure 28: The Delta block temporal behavior.

Din out

Figure 29: The Delta block model.

3.13 Scale

The scale block is a linear, opaque block that scales the value of input events bya given multiplying factor s defined as block parameter. Figure 30 reports thecorresponding temporal behavior while the block diagram is shown in Figure 31.

4 XML Syntax

To support abstract and application-independent definition of complex streamprocessing chains, a suitable XML syntax has been defined allowing to modelblock instances and connections. Block and connection modeling adopted in thespXML (stream processing XML) format is inspired by well-known modelingand synthesis languages such as VHDL , etc. It is organized into 3 main parts(see Listing 1):

19

t0

e1 e2 e3 e4 e5 e6 e7 e8 e9 e10 e11 Scale

input stream

t0

Scale

output stream

e1*s

s = scale factor

e11*s

Figure 30: The Scale block temporal behavior.

in out

s

Figure 31: The Scale block model.

block instantiation, where the stream processing blocks belonging to a givenelaboration chain are instantiated;

block connection, where the stream processing blocks composing a chain arecascaded by connecting their inputs and outputs;

chain definition, grouping block instantiation and block connection and iden-tifying chain inputs and outputs.

Listing 1: The basic structure of a spXML definition

<?xml ve r s i on =”1.0” encoding=”UTF−8”?><spXML: cha ins xmlns : spXML=”http :// e l i t e . p o l i t o . i t / b locks ”

xmlns : x s i=”http ://www.w3 . org /2001/XMLSchema−i n s t ance ”x s i : schemaLocation=”http :// e l i t e . p o l i t o . i t / b locks b locks . xsd”>

<!−− chain d e f i n i t i o n −−>

<spXML: chain id=”Example chain 1”>

<!−− block i n s t a n t i a t i o n −−>

<spXML: blocks><spXML: block func t i on=”LAST” id=” l a s t 1”>

<spXML: param name=”timeWindow”value =”10.0”unitOfMeasure=”s ” />

</spXML: block>

20

<spXML: block func t i on=”DELTA” id=”d i f f 1 ”></spXML: block><spXML: block func t i on=”THRESHOLD” id=”thr1”>

<spXML: param name=”thre sho ld ”value=”10”unitOfMeasure=”kW” />

<spXML: param name=”th r e sho ldD i r e c t i on ”value=” r i s i n g ” />

</spXML: block>. . .

</spXML: blocks>

<!−− block connect ion −−>

<spXML: connect ions><spXML: connect ion>

<spXML: from blockId=” l a s t 1 ”port=”out” />

<spXML: to b lockId=”d i f f 1 ”port=”in ” />

</spXML: connect ion>. . .

</spXML: connect ions>

<!−− chain input /output d e f i n i t i o n −−>

<spXML: input b lockId=” l a s t 1 ” port=”in ” id=”cha in1 in”/><spXML: output b lockId=”thr1 ” port=”out” id=”cha in1 out”/>

</spXML: chain></spXML: chains>

Chain definitions can furthermore be reused as blocks inside new chain defini-tions, in such a case the actual chain implementation will be a duplicate of thereused chain, built with different instances of the same blocks.

Example 1 To better clarify the spXML syntax, whose schema is reportedin Appendix A.1, consider the following example. A stream processing chainis defined for checking whether the difference between the average consumptionvalue of an electric appliance in the last hour and in the previous one is higherthan a given attention threshold τ (0,67). Graphically, the stream processingchain can be represented as in Figure 32.

Averagein

out

prev

1h batch

Din out Thresholdin out

t rising

Figure 32: Threshold on average delta in the last hour.

Such a processing chain can be defined in spXML by building the structurereported in Listing 2. As first, a “Delta1hAlert”chain must be defined, repre-senting the whole elaboration process. The chain includes 3 processing blockrespectively named “Avg1h”,“Delta1” and “Th1”. Connections between blocksare established through suitable spXML:Connection elements. In particular,the out output port of the “Avg1h” block is connected to the in port of the“Delta1” block, and the out port of the latter to the in port of the “Th1” block.Finally the chain input and output channels are defined as the in port of the“Avg1h”block and the out port of the “Th1” block, respectively.

Listing 2: Example 1 in spXML.

21

<?xml ve r s i on =”1.0” encoding=”UTF−8”?><spXML: cha ins xmlns : spXML=”http :// e l i t e . p o l i t o . i t / b locks ”

xmlns : x s i=”http ://www.w3 . org /2001/XMLSchema−i n s t ance ”x s i : schemaLocation=”http :// e l i t e . p o l i t o . i t / b locks b locks . xsd ”><spXML: chain id=”Delta1hAlert”><spXML: blocks><spXML: block id=”Avg1h” func t i on=”AVERAGE”><spXML: param name=”window” value=”1h” /><spXML: param name=”mode” value=”batch” />

</spXML: block><spXML: block id=”Delta1 ” func t i on=”DELTA”/><spXML: block id=”Th1” func t i on=”THRESHOLD”><spXML: param name=”thre sho ld ” value =”0.67” /><spXML: param name=”mode” value=” r i s i n g ” />

</spXML: block></spXML: blocks><spXML: connect ions><spXML: connect ion><spXML: from blockId=”Avg1h” port=”out” /><spXML: to b lockId=”Delta1 ” port=”in ” />

</spXML: connect ion><spXML: connect ion><spXML: from blockId=”Delta1 ” port=”out”/><spXML: to b lockId=”Th1” port=”in”/>

</spXML: connect ion></spXML: connect ions><spXML: input b lockId=”Avg1h” port=”in ” id=”in”/><spXML: output b lockId=”Th1” port=”out” id=”out”/>

</spXML: chain></spXML: chains>

5 Use cases

Jeerp defines a set of alert types that can be generated through stream process-ing and, for each of them, provides a sample use case. They encompass:

Threshold passing, e.g., if measurements coming from sensor S1 exceed apredefined value (e.g., 5kW) raise an alarm;

Threshold on hourly/temporal average, e.g., if the hourly average of datacoming from sensor S1 exceeds the value of 37.2 C◦ raise an alarm;

Exit from an operative range, e.g., given the temperature data measuredby sensor S3, raise an alarm if the temperature exits from the operativerange 0− 50 C◦;

Sudden changes detection, e.g., given the flow data measured by sensor S4

detect if in the last 10s the measured value is changed by more than 0.1m3

s ;

No measures, e.g., if sensor S5, which is configured to send new measuresevery 2s does not provide any measure for more than 5s, raise an alarm.

Many other use cases can be considered, however, at the current stage ofdevelopment, Jeerp mainly concentrates on these 5 use cases, which are furtherdetailed in the following paragraphs.

It must be noted that provided use cases exploit an extension of the spXMLlanguage, called spConfig, which allows specifying event stream sources anddrains, and to connect them to chains.

22

5.1 Threshold passing

In the threshold passing use case, a stream of real-valued events is monitored tocheck that event values are always under a given threshold value. Whenever thethreshold is passed, an alarm event must be generated, in a rising-edge fashion.If stream event values remain over the threshold, no more alarm events aregenerated until the event values fall below the given threshold. Such a behaviormight be useful to detect anomalies without generating an overwhelming amountof alarm events.

We consider a typical instance of the threshold passing use case: a sensor S1measures the power consumed by a drying apparatus; whenever the apparatusconsumption exceeds its maximum operative value, set at 5kW, an alarm shallbe generated to prevent any damage to the dryer. In terms of block modeling,this use case can be addressed by just connecting a Threshold block to theoutput of the sensor and by setting the threshold value at 5kW (see Figure 33).The corresponding spXML definition is reported in Listing 3.

ThresholdS1 Alarm

5kW rising

Figure 33: Threshold passing use case.

Listing 3: Threshold passing use case in spConfig.

<?xml ve r s i on =”1.0” encoding=”UTF−8”?><spConf ig : s t r eamProces s ingConf igurat ionxmlns : spConf ig=”http :// e l i t e . p o l i t o . i t / sp con f i g ”xmlns : spXML=”http :// e l i t e . p o l i t o . i t / b locks ”xmlns : x s i=”http ://www.w3 . org /2001/XMLSchema−i n s t ance ”x s i : schemaLocation=”http :// e l i t e . p o l i t o . i t / sp con f i g sp con f i g . xsdhttp :// e l i t e . p o l i t o . i t / b locks b locks . xsd ”>

<spConf ig : chains><spXML: chain id=”usecase1 ” name=”th r e sho ld check ing”><spXML: blocks><spXML: block func t i on=”THRESHOLD” id=”Th1”><spXML: param name=”thre sho ld ” value=”5”unitOfMeasure=”kW”/>

</spXML: block></spXML: blocks><spXML: input b lockId=”Th1” port=”in ” id=”in ” /><spXML: output b lockId=”Th1” port=”out” id=”out”/>

</spXML: chain></spConf ig : chains>

<spConf ig : eventSources><spConf ig : eventStream id=”S1” type=”REAL”/>

</spConf ig : eventSources>

<spConf ig : eventDrains><spConf ig : eventStream id=”Alarm” type=”BOOLEAN”/>

</spConf ig : eventDrains>

<spConf ig : connect ions><spConf ig : connect ion><spConf ig : fromSource cha inId=”usecase1 ” inputId=”in ”source=”S1” />

23

<spConf ig : toDrain cha inId=”usecase1 ” dra in=”Alarm”outputId=”out” />

</spConf ig : connect ion></spConf ig : connect ions>

</spConf ig : s t reamProcess ingConf igurat ion>

5.2 Threshold on hourly average

In this use case, the average value of a real-valued event stream is monitored, in agiven time frame. Whenever the value exceeds a given threshold an alarm eventmust be generated. A typical instantiation of such a use case is on detectionof anomalies in energy consumptions. Independently from the activation cycle,and the instantaneous absorption profile, a given appliance consumes on theaverage 0,5kWh (i.e., the power consumption average in 1 hour is 0,5kW). Ifthe appliance exceeds the 1kWh threshold,then an alarm event shall be raised.

Assuming that the appliance power consumption is sensed through the powermeter M1, which provides new measures at regular (fixed) intervals, then, interms of block processing, the scenario can be fulfilled by concatenating anaverage block, with a 1 hour long batch window and a threshold block, witha threshold value of 1kW (see Figure 34). In spXML this means defining a2-block elaboration chain named “energy Consumption Excess”, as depicted inListing 4.

Threshold

M1

Alarm

1kW rising

Average

out

prev

1h batch

Figure 34: Threshold on hourly average.

It must be noticed that if an energy meter E1 is available, the same alertingscenario needs just a threshold block (i.e., it falls under the previous use case).

Listing 4: Threshold on hourly average use case in spXML.

<?xml ve r s i on =”1.0” encoding=”UTF−8”?><spConf ig : s t r eamProces s ingConf igurat ion

xmlns : spConf ig=”http :// e l i t e . p o l i t o . i t / spcon f i g ”xmlns : spXML=”http :// e l i t e . p o l i t o . i t / b locks ”xmlns : x s i=”http ://www.w3 . org /2001/XMLSchema−i n s t ance ”x s i : schemaLocation=”http :// e l i t e . p o l i t o . i t / sp con f i g sp con f i g . xsdhttp :// e l i t e . p o l i t o . i t / b locks b locks . xsd ”>

<spConf ig : chains><spXML: chain id=”usecase1”><spXML: blocks><spXML: block id=”Avg1” func t i on=”AVERAGE”><spXML: param name=”window” value=”1” unitOfMeasure=”h” /><spXML: param name=”mode” value=”batch” />

</spXML: block><spXML: block id=”Th1” func t i on=”THRESHOLD”><spXML: param name=”thre sho ld ” value=”1” unitOfMeasure=”kW” />

</spXML: block></spXML: blocks>

24

<spXML: connect ions><spXML: connect ion><spXML: from blockId=”Avg1” port=”out” /><spXML: to b lockId=”Th1” port=”in ” />

</spXML: connect ion></spXML: connect ions><spXML: input b lockId=”Avg1” port=”in ” id=”in ” /><spXML: output b lockId=”Th1” port=”out” id=”out” />


<spConf ig : eventSources><spConf ig : eventStream id=”M1” type=”REAL”/>




<spConf ig : connect ions><spConf ig : connect ion><spConf ig : fromSource cha inId=”usecase1 ” inputId=”in ”source=”M1” />




5.3 Exit from an operative range

This use case applies to real-valued event streams that must include eventshaving values inside a pre-defined operative range; if the values are out of sucha range, an alarm shall be raised. We consider a medium complexity use casewhere the hourly amount of operations (e.g., pen molding) shall be in a givenrange, e.g., between 1000 and 5000. If the operation amount is out of the rangeeither the monitored machine is over-used, requiring higher maintenance costsor under-used, thus implying an higher impact of the machine operation on theretail cost of produced pens. Assuming that the machine operation is monitoredthrough the sensor S5, which provides a 1-valued event whenever a new pen ismolded, then the alert chain is depicted in Figure 35 and the correspondingspXML specification is reported in Listing 5.

SumS5

out

prev

1h batch

Rangein

Alarm

in-range leaving

5000 1000

Figure 35: Exit from an operative range use case.

Listing 5: Exit from an operative range use case in spXML.

<?xml ve r s i on =”1.0” encoding=”UTF−8”?><spConf ig : s t r eamProces s ingConf igurat ionxmlns : spConf ig=”http :// e l i t e . p o l i t o . i t / sp con f i g ”xmlns : spXML=”http :// e l i t e . p o l i t o . i t / b locks ”

25

xmlns : x s i=”http ://www.w3 . org /2001/XMLSchema−i n s t ance ”x s i : schemaLocation=”http :// e l i t e . p o l i t o . i t / sp con f i g sp con f i g . xsdhttp :// e l i t e . p o l i t o . i t / b locks b locks . xsd ”>

<spConf ig : chains><spXML: chain id=”usecase3”><spXML: blocks><spXML: block id=”Sum1” func t i on=”SUM”><spXML: param name=”window” value=”1” unitOfMeasure=”h” /><spXML: param name=”mode” value=”batch” />

</spXML: block><spXML: block id=”Range1” func t i on=”RANGE”><spXML: param name=”HThreshold” value=”5000” /><spXML: param name=”LThreshold” value=”1000” /><spXML: param name=”range mode” value=”in−range ” /><spXML: param name=”mode” value=”l eav ing ” />

</spXML: block></spXML: blocks><spXML: connect ions><spXML: connect ion><spXML: from blockId=”Sum1” port=”out” /><spXML: to b lockId=”Range1” port=”in ” />

</spXML: connect ion></spXML: connect ions><spXML: input b lockId=”Sum1” port=”in ” id=”in ” /><spXML: output b lockId=”Range1” port=”out” id=”out” />


<spConf ig : eventSources><spConf ig : eventStream id=”S5” type=”REAL”/>








5.4 Sudden changes detection

In this scenario, a real-valued event stream is monitored for detecting suddenchanges in the event values. Depending on the application, the monitored valuescan either be the plain event values or the average/sum of event values in agiven time frame. For example, in a factory process temperature shall variateslowly to preserve certain qualities of the product being manufactured. If thetemperature is sensed by sensor S4, imposing a slow variation in temperaturemight mean that the difference of temperature values measured in a given timewindow, say 10s, shall be less than a given variation threshold, e.g., 0.1 C◦. Interm of processing blocks, this is equivalent to the processing chain shown inFigure 36 and encoded in Listing 6.

Listing 6: The Sudden changes detection use case in spXML.

<?xml ve r s i on =”1.0” encoding=”UTF−8”?><spConf ig : s t r eamProces s ingConf igurat ionxmlns : spConf ig=”http :// e l i t e . p o l i t o . i t / sp con f i g ”xmlns : spXML=”http :// e l i t e . p o l i t o . i t / b locks ”

26

Lastprev

10s

Threshold Alarm

0.1C rising

S4 x

Figure 36: A possible solution of the Sudden changes detection use case.

xmlns : x s i=”http ://www.w3 . org /2001/XMLSchema−i n s t ance ”x s i : schemaLocation=”http :// e l i t e . p o l i t o . i t / sp con f i g sp con f i g . xsdhttp :// e l i t e . p o l i t o . i t / b locks b locks . xsd ”>

<spConf ig : chains><spXML: chain id=”usecase4”><spXML: blocks><spXML: block id=” l a s t 1 ” func t i on=”LAST”><spXML: param name=”window” value=”10”unitOfMeasure=”s ” />

</spXML: block><spXML: block id=”de l ta1 ” func t i on=”DELTA” /><spXML: block id=”thre sho ld1 ” func t i on=”THRESHOLD”><spXML: param name=”thre sho ld ” value =”0.1”unitOfMeasure=”◦C” />

</spXML: block></spXML: blocks><spXML: connect ions><spXML: connect ion><spXML: from blockId=” l a s t 1 ” port=”out” /><spXML: to b lockId=”de l ta1 ” port=”in ” />

</spXML: connect ion><spXML: connect ion><spXML: from blockId=”de l ta1 ” port=”out” /><spXML: to b lockId=”thre sho ld1 ” port=”in ” />

</spXML: connect ion></spXML: connect ions><spXML: input b lockId=” l a s t 1 ” port=”in ” id=”in ” /><spXML: output b lockId=”thre sho ld1 ” port=”out” id=”out” />


<spConf ig : eventSources><spConf ig : eventStream id=”S4” type=”REAL” />


<spConf ig : eventDrains><spConf ig : eventStream id=”Alarm” type=”BOOLEAN” />






5.5 No Measures

The no measure scenario requires to monitor an event stream for detectingmissing events. For example, if a sensor is expected to send a notification eventevery 2s and after 5s nothing has been observed, than the sensor might be failingand an alert shall be raised to trigger maintenance intervention. If we consider

27

the sensor S5 as the sensor which fails if no measure is received after 5s, then thecorresponding block diagram is reported in Figure 37, and only involves a timeguard processing entity. Listing 7 reports the corresponding spXML definition.

Time

GuardS5 Alert

moving after

5s

Figure 37: A possible No Measure use case solution.

Listing 7: The no measure use case solution in spXML.

<?xml ve r s i on =”1.0” encoding=”UTF−8”?><spConf ig : s t r eamProces s ingConf igurat ionxmlns : spConf ig=”http :// e l i t e . p o l i t o . i t / sp con f i g ”xmlns : spXML=”http :// e l i t e . p o l i t o . i t / b locks ”xmlns : x s i=”http ://www.w3 . org /2001/XMLSchema−i n s t ance ”x s i : schemaLocation=”http :// e l i t e . p o l i t o . i t / sp con f i g sp con f i g . xsdhttp :// e l i t e . p o l i t o . i t / b locks b locks . xsd ”>

<spConf ig : chains><spXML: chain id=”usecase5”><spXML: blocks><spXML: block id=”time guard1 ” func t i on=”TIME GUARD”><spXML: param name=”time guard ” value=”5”unitOfMeasure=”s ” />

<spXML: param name=”windowing” value=”moving” /><spXML: param name=”mode” value=”miss ing ” />

</spXML: block></spXML: blocks><spXML: input b lockId=”time guard1 ” port=”in ” id=”in”/><spXML: output b lockId=”time guard1 ” port=”out” id=”out”/>


<spConf ig : eventSources><spConf ig : eventStream id=”S5” type=”REAL” />


<spConf ig : eventDrains><spConf ig : eventStream id=”Alarm” type=”BOOLEAN” />






28

A Appendix

A.1 spXMLSchema

Listing 8: spXML Schema

<?xml version=” 1 .0 ” encoding=”UTF−8”?><schema targetNamespace=” ht tp : // e l i t e . p o l i t o . i t / b locks ”elementFormDefault=” q u a l i f i e d ” xmlns=” ht tp : //www.w3 . org /2001/XMLSchema”xmlns:spXML=” ht tp : // e l i t e . p o l i t o . i t / b locks ”><complexType name=”blockType”><sequence><element name=”param” type=”spXML:paramType” maxOccurs=”unbounded”minOccurs=”0”>

</ element></ sequence><a t t r i bu t e name=” id ” type=”ID” use=” requ i r ed ”></ a t t r i bu t e><a t t r i bu t e name=”name” type=” s t r i n g ” use=” opt i ona l ”></ a t t r i bu t e><a t t r i bu t e name=” func t i on ” use=” opt i ona l ”><simpleType><r e s t r i c t i o n base=” s t r i n g ”><enumeration value=”LAST”></ enumeration><enumeration value=”AVERAGE”></ enumeration><enumeration value=”SUM”></ enumeration><enumeration value=”ABS”></ enumeration><enumeration value=”THRESHOLD”></ enumeration><enumeration value=”HYSTERESIS THRESHOLD”></ enumeration><enumeration value=”TIME GUARD”></ enumeration><enumeration value=”RANGE”></ enumeration><enumeration value=”TIME FILTER”></ enumeration><enumeration value=”AND”></ enumeration><enumeration value=”JOIN”></ enumeration><enumeration value=”DELTA”></ enumeration><enumeration value=”SCALE”></ enumeration>

</ r e s t r i c t i o n></ simpleType>

</ a t t r i bu t e><a t t r i bu t e name=” chain ” type=”IDREF” use=” opt i ona l ”></ a t t r i bu t e>

</complexType>

<complexType name=”blockListType ”><sequence><element name=”block ” type=”spXML:blockType” maxOccurs=”unbounded”minOccurs=”1”></ element>

</ sequence></complexType>

<complexType name=”chainType”><sequence><element name=” blocks ” type=”spXML:blockListType”maxOccurs=”1” minOccurs=”1”>

</ element><element name=” connect ions ” type=”spXML:connectionListType”maxOccurs=”1” minOccurs=”0”>

</ element><element name=” input ” type=”spXML:IOType” maxOccurs=”1”minOccurs=”1”>

</ element><element name=”output” type=”spXML:IOType” maxOccurs=”1”minOccurs=”1”>

</ element></ sequence><a t t r i bu t e name=”name” type=” s t r i n g ”></ a t t r i bu t e><a t t r i bu t e name=” id ” type=”ID” use=” requ i r ed ”></ a t t r i bu t e>

</complexType>

<complexType name=” connect ionListType ”><sequence><element name=” connect ion ” type=”spXML:connectionType”maxOccurs=”unbounded” minOccurs=”1”></ element>

</ sequence>

29

</complexType>

<complexType name=”connectionType”><sequence><element name=”from” type=”spXML:endpointType” maxOccurs=”1”minOccurs=”1”>

</ element><element name=” to ” type=”spXML:endpointType” maxOccurs=”1”minOccurs=”1”></ element>

</ sequence>

</complexType>

<complexType name=”endpointType”><a t t r i bu t e name=” blockId ” type=”IDREF”></ a t t r i bu t e><a t t r i bu t e name=”port ” type=” s t r i n g ”></ a t t r i bu t e>

</complexType>

<element name=” cha ins ” type=”spXML:chainListType”></ element>

<complexType name=” chainListType ”><sequence><element name=” chain ” type=”spXML:chainType” maxOccurs=”unbounded”minOccurs=”1”></ element>

</ sequence></complexType>

<complexType name=”paramType”><a t t r i bu t e name=”name” type=” s t r i n g ” use=” requ i r ed ”></ a t t r i bu t e><a t t r i bu t e name=”value ” type=”anySimpleType” use=” requ i r ed ”></ a t t r i bu t e><a t t r i bu t e name=”unitOfMeasure ” type=” s t r i n g ” use=” opt i ona l ”></ a t t r i bu t e>

</complexType>

<complexType name=”IOType”><a t t r i bu t e name=” blockId ” type=”IDREF” use=” requ i r ed ”></ a t t r i bu t e><a t t r i bu t e name=”port ” type=” s t r i n g ” use=” requ i r ed ”></ a t t r i bu t e><a t t r i bu t e name=” id ” type=”ID” use=” requ i r ed ”></ a t t r i bu t e>

</complexType></schema>

A.2 Time Periods specification

Time specification in blocks is always composed of two parts: a number and aunit of measure. The number can assume any real positive value while the unitof measure can either be ms (milliseconds), s (seconds), m(minutes), h(hours),d(days).

Listing 9: Time Period grammar

time−pe r i od : ( number ) ( ”ms” , ” s ” , ”m” , ”h” , ”d” )

A.3 Cron expressions

Cron-Expressions are strings that are actually made up of seven sub-expressions,that describe individual details of the schedule. These sub-expression are sepa-rated with white-space, and represent:

1. Seconds

2. Minutes

3. Hours

30

4. Day-of-Month

5. Month

6. Day-of-Week

7. Year (optional field)

An example of a complete cron-expression is the string ”0 0 12 ? * WED” -which means ”every Wednesday at 12:00:00 pm”.

Individual sub-expressions can contain ranges and/or lists. For example,the day of week field in the previous (which reads ”WED”) example could bereplaced with ”MON-FRI”, ”MON,WED,FRI”, or even ”MON-WED,SAT”.

Wild-cards (the ” character) can be used to say ”every” possible value of thisfield. Therefore the ” character in the ”Month” field of the previous examplesimply means ”every month”. A ’*’ in the Day-Of-Week field would thereforeobviously mean ”every day of the week”.

All of the fields have a set of valid values that can be specified. These valuesshould be fairly obvious - such as the numbers 0 to 59 for seconds and minutes,and the values 0 to 23 for hours. Day-of-Month can be any value 0-31, but youneed to be careful about how many days are in a given month! Months can bespecified as values between 0 and 11, or by using the strings JAN, FEB, MAR,APR, MAY, JUN, JUL, AUG, SEP, OCT, NOV and DEC. Days-of-Week canbe specified as values between 1 and 7 (1 = Sunday) or by using the stringsSUN, MON, TUE, WED, THU, FRI and SAT.

The ’/’ character can be used to specify increments to values. For example,if you put ’0/15’ in the Minutes field, it means ’every 15th minute of the hour,starting at minute zero’. If you used ’3/20’ in the Minutes field, it would mean’every 20th minute of the hour, starting at minute three’ - or in other words itis the same as specifying ’3,23,43’ in the Minutes field. Note the subtlety that”/35” does *not mean ”every 35 minutes” - it mean ”every 35th minute of thehour, starting at minute zero” - or in other words the same as specifying ’0,35’.

The ’?’ character is allowed for the day-of-month and day-of-week fields. Itis used to specify ”no specific value”. This is useful when you need to specifysomething in one of the two fields, but not the other.

The ’L’ character is allowed for the day-of-month and day-of-week fields.This character is short-hand for ”last”, but it has different meaning in each ofthe two fields. For example, the value ”L” in the day-of-month field means ”thelast day of the month” - day 31 for January, day 28 for February on non-leapyears. If used in the day-of-week field by itself, it simply means ”7” or ”SAT”.But if used in the day-of-week field after another value, it means ”the last xxxday of the month” - for example ”6L” or ”FRIL” both mean ”the last friday ofthe month”. You can also specify an offset from the last day of the month, suchas ”L-3” which would mean the third-to-last day of the calendar month. Whenusing the ’L’ option, it is important not to specify lists, or ranges of values, asyou’ll get confusing/unexpected results.

The ’W’ is used to specify the weekday (Monday-Friday) nearest the givenday. As an example, if you were to specify ”15W” as the value for the day-of-month field, the meaning is: ”the nearest weekday to the 15th of the month”.

The ’#’ is used to specify ”the nth” XXX weekday of the month. Forexample, the value of ”6#3” or ”FRI#3” in the day-of-week field means ”thethird Friday of the month”.

31

A.4 Unit of Measure specification

Unit of measures in block parameters are always specified according to the“Unified Code for Units of Measure”[1] and handled, i.e., parsed and generatedthrough the JScience3 Java API.

References

[1] Gunther Schadow and Clement J. McDonald. The unified code for unitsof measure. Technical report, Regenstrief Institute, Inc. and the UCUMOrganization, 2009.

3http://jscience.org/

32

Documents

Stream Processing Blocks