1

CEP – complex event processingOSDC, Brisbane, November 2009Michael NealeJBoss/Red Hat R&D

2

Michael NealeR&D w. JBoss (specialise in drools, now

“cloud research”)

Open source history (user -> fulltime developer “acquired” by jboss 2005).

Thanks OSDC organisers !

me on the web:

www.michaelneale.net, twitter.com/michaelneale, michaelneale.blogspot.com

3

Outline• Define CEP?

• Where is it used

• Events and patterns

• Sliding windows

• Temporal reasoning

• Defining an event “object”

• Libraries: Drools, Esper

• Sample scenario

4

Definition• Treat “inputs” as “events” (time

based)

• Look for patterns/correlations to extract some meaning (“an inferred event/fact”)

• Act on these patterns (return a result, perform a calculation, send a message etc)

• Sometimes also called:

– Event Stream Processing

– Temporal correlation

5

eg• Event: [Hand In Air]

• Pattern: Lots of [Hand In Air] (possibly waving, as if they didn't care), happening in close (time and space) quarters...

• Infer: A party

6

Huge number of events, but only a few of real interest

Usually events are immutable

Usually queries/rules have to run in reactive mode

Strong temporal relationships between events

Individual events are usually not important

The composition and aggregation of events is important

7

Where is it used? • Logistics: Package en-route

tracking/warning

• Stock market events/trades

– Provide real time alerts to end users

• Any where you have “needle in a haystack” problem, involving time based events

8

Events and patterns• Events immutable** facts

(objects/records)

– Eg a reading from a sensor, a stock value, a “fact-oid”

• Patterns are logic constraints (predicate/higher order statements)

– Think like the “where” part of a SQL query

• Patterns search for particular events...

9

Esper example:

select * from StockTickEvent.win:time(30 sec) where price > 20select avg(price) from StockTickEvent.win:time(30 sec) ...

Drools Fusion example:

when StockTick(price > 20) over window:time( 30s )

.. and with some temporal correlation:

$eventA : StockTick( this coincides[15s, 10s] $eventB )

Event instance of StockTick

10

Patterns• Are used to detect specific events

(using constraints and higher order logic)

• Are used to detect co-incidences (over time periods)

• Allow you to be declarative (once again, think SQL, but involving time)

11

Sliding time windows• Declare what “window” of time you

care about

• Needed to narrow down from huge number of events

• Help identify “co-incidences” which might be more then just co-incidences !

• Optional: maybe you care about ALL events, maybe just close together

12

Temporal reasoning• A generalisation of “looking for co-

incidences” of interest

• Eg Hangover often follows from XXXX consumption, within certain time windows. Perhaps all of the time? (“forall”) Some of the time? (“exists”)

• Look for sets of events, related in time, to infer what is really “going on”

• Sets CEP apart from offline/batch processing

13

• Event and Time semantics:

– Point in Time

– Interval-based

• Unified semantics for event correlation over time

• Temporal Constraints:

– Set of operators to express temporal relationship between events

Event semantics

14

Event semanticsJF Allen defined a set of Time Event

semantics:

15

Temporal operatorsAfter

Before

Coincides

During

Finishes

Finished By

Includes

Meets

Met By

Overlaps

Overlapped By

Starts

Started By

16

Defining events• Generally classes defined in host

language (eg java, C#)

• Some extra metadata needed to define expiry, duration, timestamp etc (all optional).

• Expiry important for memory management

– Large amounts of data

– Why keep it in memory if not needed

– Very stateful

17

Defining events// declaring existing class

import some.package.VoiceCall

declare VoiceCall

@role( event )

@timestamp( calltime )

@duration( duration )

end

// generating an event class

declare StockTick

@role( event )

symbol : String

price : double

end

18

Drools Fusion• jboss.org/drools

• Submodule of Drools project

• Uses rule language to express CEP patterns

– Same “drl” language as normal rules

– Rules can take action directly, or be queries to return results

19

rule "Sound the alarm in case temperature rises above threshold"

when

TemperatureThreshold( $max : max )

Number( doubleValue > $max ) from accumulate( SensorReading( $temp : temperature ) over window:time( 10m ), average( $temp ) )

then

// sound the alarm

20

Esper• esper.codehaus.org

• Takes “temporal SQL” approach

• Return results for matches, a la SQL database (but its realtime events)

• SQL skill portability ?

21

select fraud.accountNumber as accntNum, fraud.warning as warn, withdraw.amount as amount,

MAX(fraud.timestamp, withdraw.timestamp) as timestamp, 'withdrawlFraud' as desc

from FraudWarningEvent.win:time(30 min) as fraud,

WithdrawalEvent.win:time(30 sec) as withdraw

where fraud.accountNumber = withdraw.accountNumber

22

Deployment• Usually a dedicated server/service

endpoint

– (but could be embedded in your app if JVM friendly)

• (eg wrapped around a REST servlet, or MQ)

• Sits there, gobbling up messages as fast as it can

23

Concurrency...Need for “realtime” performance

Need to make use of threads/processes

Use of “rulebase partitioning” to separate less coupled data to allow this

So: User doesn't have to worry

24

FedEx examplePresented at JavaOne, Adobe MAX etc..

FedEx Custom Critical:

1: En-Route Tracking Situational Awareness

2: Capacity Allocation Managementhttp://developers.sun.com/learning/javaoneonline/sessions/2009/pdf/TS-4475.pdf

25

26

27

28

29

A flex front end

30

31

Thanks, Q&A