WSO2 Complex Event Processor 4.0

Sriskandarajah SuhothayanTechnical Lead

Product Release Webinar

Lasantha FernandoSenior Software Engineer

WSO2 Analytics Platform

WSO2 Analytics Platform

Data Processing Pipeline

Realtime Analytics

What is WSO2 CEP ?

New in CEP 4.0

o Re Architecture of the CEP Servero Rewrite of Siddhi 3.0o Scalable Distributed Processing with Apache Stormo Better High Availability support o Domain Specific Execution Managero Realtime Dashboardo Improved usability

Event Flow of WSO2 CEP 4.0

CEP 3.x Input Adaptor + Event Builder = CEP 4.0 Event Receiver CEP 3.x Output Adaptor + Event Formatter = CEP 4.0 Event Publisher

Receiving & Notifying Events

*Supports custom event receivers via its pluggable architecture!

Highly Pluggable Event Receiver Architecture

Highly Pluggable Event Publisher Architecture

*Supports custom event publishers via its pluggable architecture!

Execution of Queries

What’s new in Siddhi 3.0

o New architectureo Optimized for better performanceo Improved Query API and Compiler (Antlr 4)o Integration of Disruptor

o Better Extendability o Annotation supporto Native support for distributed processing via Partition

Groupingo Event Trigger Support

What’s new in Siddhi 3.0 ...

o Inbuilt Scripting support o JavaScript, R & Scala

o Event Table with o RDBMS & Data Analytics Server Tables

o New Window supporto Cron & Sorting

o Realtime Machine learning o R, WSO2ML & PMML

o Rich set of Siddhi Extensions

define stream StockStream (symbol string, price double, volume int);

partition by (symbol of StockStream)

begin

from t1=StockStream,

t2=StockStream [(t2[last] is null and t1.price < price) or

(t2[last].price < price)]+

within 5 min

select t1.price as initialPrice, t2[last].price as finalPrice,t1.symbol

insert into IncreasingMyStockPriceStream

end;

Siddhi Query (Partition)

define table CardUserTable (name string, cardNum long) ;

@from(eventtable = 'rdbms' , datasource.name = ‘CardDataSource’ , table.name = ‘UserTable’, caching.algorithm’=‘LRU’)

define table CardUserTable (name string, cardNum long)

Cache types supportedo Basic: A size-based algorithm based on FIFO.o LRU (Least Recently Used): The least recently used event is dropped

when cache is full.o LFU (Least Frequently Used): The least frequently used event is dropped

when cache is full.

Siddhi Query (Event Table)

Supported for RDBMS, In-Memory, Analytics Table

define function concatJ [JavaScript] return string {

var str1 = data[0];

var str2 = data[1];

var str3 = data[2];

var res = str1.concat(str2,str3);

return res;

};

from cseEventStream select price , concatJ(symbol,' ',price) as concatStr

group by volume insert into mailOutput;

Siddhi Query (Scripting)

Supported for Javascript, Scala and R. Can be

extended for other languages

o Function extensiono Aggregator extensiono Window extensiono Stream Processor extension

define stream SalesStream (brand string, price double, currency string);

from SalesStream

select brand, custom:toUSD(price, currency) as priceInUSD

insert into OutputStream ;Referred with namespaces

Siddhi Query (Extension) ...

Provides support for common functionalities across multiple domains out of the box

o geo: Geographical processing o nlp: Natural language Processing (with Stanford NLP)o ml: Running machine learning models of WSO2 Machine Lerner o pmml: Running PMML models learnt by Ro timeseries: Regression and time series o math: Mathematical operationso str: String operations o regex: Regular expression o ...

New Siddhi Extensions

Siddhi Try-it ToolAllows users to o Try out different queries o Validate queries by sending events

Event Simulator

Allows users to test execution environment and simulate complex execution scenarios o Ability to send events to a stream without configuring

event receiverso Send events

o Directly via UIo Upload a file and send/replay eventso Point to a database table and play those events

o Configurable delay between events when sending via file/database

Event Simulator

Ideal for business users, Form based UI to create/edit/delete execution logico Allows to create a predefined execution plan template to a domaino Which can be managed via UI

Domain Specific Execution Manager

Event Flowo Visualizes the event flow to provide an overall context

Scaling & HA

High Availability with WSO2 CEP 4.0

Scalability: Why not use existing CEPs ...Spark Streamingo Supports distributed processingo Runs micro batcheso Does not support pattern & sequence detection

Apache Storm o Supports distributed processingo Stream processing engine

Scalability: Why not use Apache Storm ?Advantages

o Supports distributed processingo Supports Partitioningo Extendableo Opensource

Disadvantages

o Need to write Java code o Need to start from basic principles ( & data structures ) o Adoption for change is slowo No support to govern artifacts

WSO2 CEP 4.0 Scalability

Distributed Realtime = Siddhi +

Advantages over Apache Stormo No need to write Java code (Supports SQL like query language)o No need to start from basic principles (Supports high level

language)o Adoption for change is fasto Govern artifacts using Toolboxeso etc ...

How we scale ?

Scaling with Storm

Handling Stateless & Stateful Queries

Siddhi QL define stream StockStream (symbol string, volume int, price double);

@name(‘Filter Query’)from StockStream[price > 75]select *insert into HighPriceStockStream ;

@name(‘Window Query’)from HighPriceStockStream#window.time(10 min)select symbol, sum(volume) as sumVolume insert into ResultStockStream ;

Siddhi QL - with partition define stream StockStream (symbol string, volume int, price double);

@name(‘Filter Query’)from StockStream[price > 75]select *insert into HighPriceStockStream ;

@name(‘Window Query’)partition with (symbol of HighPriceStockStream)begin

from HighPriceStockStream#window.time(10 min)select symbol, sum(volume) as sumVolume insert into ResultStockStream ;

end;

Siddhi QL - distributeddefine stream StockStream (symbol string, volume int, price double);

@name(Filter Query’)@dist(parallel= ‘3')from StockStream[price > 75]select *insert into HightPriceStockStream ;

@name(‘Window Query’)@dist(parallel= ‘2')partition with (symbol of HighPriceStockStream)begin

from HighPriceStockStream#window.time(10 min)select symbol, sum(volume) as sumVolume insert into ResultStockStream ;

end;

Distributed Execution on Storm UI

Realtime Dashboard

Realtime Dashboard

Provides an overall idea on what's going ono Dashboard using

o Google Gadget o HTML5 + javascripts

o Support custom gadgets and dashboards

o Gadget generation witho D3 and Vega

Gadget Generation Tool

Customize with few clicks …Support realtime and stored data visualization

How ?○ Start with data in tabular format○ Select chart type (Bar, Line, Map, ...)○ Map columns to dimensions (X Y, color, point size, … )○ Also do drill downs

Geo Dashboard

With configurable alerting & Monitoring capabilities from UI.

http://wso2.com/library/articles/2015/01/article-geo-spatial-data-analysis-using-wso2-complex-event-processor-0/

Demo

Questions?

More Information

o WSO2 CEP 4.0.0 download - http://wso2.com/products/complex-event-processor/

o Product documentation - https://docs.wso2.com/display/CEP400/WSO2+Complex+Event+Processor+Documentation

o WSO2 Analytics Platform - http://wso2.com/analytics

o Quick Start Guide doc - https://docs.wso2.com/display/CEP400/Quick+Start+Guide

o Quick Start Guide video - https://www.youtube.com/playlist?list=PLp0TUr0bmhX5v5yBa0TINUktuF-_eDPC_

o Pre Release Webinar on CEP 4.0 - http://wso2.com/library/webinars/2015/07/real-time-analytics-from-data-to-actions-in-milliseconds/

Contact us !