View
606
Download
1
Category
Preview:
Citation preview
WSO2 Complex Event Processor 4.0
Sriskandarajah SuhothayanTechnical Lead
Product Release Webinar
Lasantha FernandoSenior Software Engineer
WSO2 Analytics Platform
WSO2 Analytics Platform
Data Processing Pipeline
Realtime Analytics
What is WSO2 CEP ?
New in CEP 4.0
o Re Architecture of the CEP Servero Rewrite of Siddhi 3.0o Scalable Distributed Processing with Apache Stormo Better High Availability support o Domain Specific Execution Managero Realtime Dashboardo Improved usability
Event Flow of WSO2 CEP 4.0
CEP 3.x Input Adaptor + Event Builder = CEP 4.0 Event Receiver CEP 3.x Output Adaptor + Event Formatter = CEP 4.0 Event Publisher
Receiving & Notifying Events
*Supports custom event receivers via its pluggable architecture!
Highly Pluggable Event Receiver Architecture
Highly Pluggable Event Publisher Architecture
*Supports custom event publishers via its pluggable architecture!
Execution of Queries
What’s new in Siddhi 3.0
o New architectureo Optimized for better performanceo Improved Query API and Compiler (Antlr 4)o Integration of Disruptor
o Better Extendability o Annotation supporto Native support for distributed processing via Partition
Groupingo Event Trigger Support
What’s new in Siddhi 3.0 ...
o Inbuilt Scripting support o JavaScript, R & Scala
o Event Table with o RDBMS & Data Analytics Server Tables
o New Window supporto Cron & Sorting
o Realtime Machine learning o R, WSO2ML & PMML
o Rich set of Siddhi Extensions
define stream StockStream (symbol string, price double, volume int);
partition by (symbol of StockStream)
begin
from t1=StockStream,
t2=StockStream [(t2[last] is null and t1.price < price) or
(t2[last].price < price)]+
within 5 min
select t1.price as initialPrice, t2[last].price as finalPrice,t1.symbol
insert into IncreasingMyStockPriceStream
end;
Siddhi Query (Partition)
define table CardUserTable (name string, cardNum long) ;
@from(eventtable = 'rdbms' , datasource.name = ‘CardDataSource’ , table.name = ‘UserTable’, caching.algorithm’=‘LRU’)
define table CardUserTable (name string, cardNum long)
Cache types supportedo Basic: A size-based algorithm based on FIFO.o LRU (Least Recently Used): The least recently used event is dropped
when cache is full.o LFU (Least Frequently Used): The least frequently used event is dropped
when cache is full.
Siddhi Query (Event Table)
Supported for RDBMS, In-Memory, Analytics Table
define function concatJ [JavaScript] return string {
var str1 = data[0];
var str2 = data[1];
var str3 = data[2];
var res = str1.concat(str2,str3);
return res;
};
from cseEventStream select price , concatJ(symbol,' ',price) as concatStr
group by volume insert into mailOutput;
Siddhi Query (Scripting)
Supported for Javascript, Scala and R. Can be
extended for other languages
o Function extensiono Aggregator extensiono Window extensiono Stream Processor extension
define stream SalesStream (brand string, price double, currency string);
from SalesStream
select brand, custom:toUSD(price, currency) as priceInUSD
insert into OutputStream ;Referred with namespaces
Siddhi Query (Extension) ...
Provides support for common functionalities across multiple domains out of the box
o geo: Geographical processing o nlp: Natural language Processing (with Stanford NLP)o ml: Running machine learning models of WSO2 Machine Lerner o pmml: Running PMML models learnt by Ro timeseries: Regression and time series o math: Mathematical operationso str: String operations o regex: Regular expression o ...
New Siddhi Extensions
Siddhi Try-it ToolAllows users to o Try out different queries o Validate queries by sending events
Event Simulator
Allows users to test execution environment and simulate complex execution scenarios o Ability to send events to a stream without configuring
event receiverso Send events
o Directly via UIo Upload a file and send/replay eventso Point to a database table and play those events
o Configurable delay between events when sending via file/database
Event Simulator
Ideal for business users, Form based UI to create/edit/delete execution logico Allows to create a predefined execution plan template to a domaino Which can be managed via UI
Domain Specific Execution Manager
Event Flowo Visualizes the event flow to provide an overall context
Scaling & HA
High Availability with WSO2 CEP 4.0
Scalability: Why not use existing CEPs ...Spark Streamingo Supports distributed processingo Runs micro batcheso Does not support pattern & sequence detection
Apache Storm o Supports distributed processingo Stream processing engine
Scalability: Why not use Apache Storm ?Advantages
o Supports distributed processingo Supports Partitioningo Extendableo Opensource
Disadvantages
o Need to write Java code o Need to start from basic principles ( & data structures ) o Adoption for change is slowo No support to govern artifacts
WSO2 CEP 4.0 Scalability
Distributed Realtime = Siddhi +
Advantages over Apache Stormo No need to write Java code (Supports SQL like query language)o No need to start from basic principles (Supports high level
language)o Adoption for change is fasto Govern artifacts using Toolboxeso etc ...
How we scale ?
Scaling with Storm
Handling Stateless & Stateful Queries
Siddhi QL define stream StockStream (symbol string, volume int, price double);
@name(‘Filter Query’)from StockStream[price > 75]select *insert into HighPriceStockStream ;
@name(‘Window Query’)from HighPriceStockStream#window.time(10 min)select symbol, sum(volume) as sumVolume insert into ResultStockStream ;
Siddhi QL - with partition define stream StockStream (symbol string, volume int, price double);
@name(‘Filter Query’)from StockStream[price > 75]select *insert into HighPriceStockStream ;
@name(‘Window Query’)partition with (symbol of HighPriceStockStream)begin
from HighPriceStockStream#window.time(10 min)select symbol, sum(volume) as sumVolume insert into ResultStockStream ;
end;
Siddhi QL - distributeddefine stream StockStream (symbol string, volume int, price double);
@name(Filter Query’)@dist(parallel= ‘3')from StockStream[price > 75]select *insert into HightPriceStockStream ;
@name(‘Window Query’)@dist(parallel= ‘2')partition with (symbol of HighPriceStockStream)begin
from HighPriceStockStream#window.time(10 min)select symbol, sum(volume) as sumVolume insert into ResultStockStream ;
end;
Distributed Execution on Storm UI
Realtime Dashboard
Realtime Dashboard
Provides an overall idea on what's going ono Dashboard using
o Google Gadget o HTML5 + javascripts
o Support custom gadgets and dashboards
o Gadget generation witho D3 and Vega
Gadget Generation Tool
Customize with few clicks …Support realtime and stored data visualization
How ?○ Start with data in tabular format○ Select chart type (Bar, Line, Map, ...)○ Map columns to dimensions (X Y, color, point size, … )○ Also do drill downs
Geo Dashboard
With configurable alerting & Monitoring capabilities from UI.
http://wso2.com/library/articles/2015/01/article-geo-spatial-data-analysis-using-wso2-complex-event-processor-0/
Demo
Questions?
More Information
o WSO2 CEP 4.0.0 download - http://wso2.com/products/complex-event-processor/
o Product documentation - https://docs.wso2.com/display/CEP400/WSO2+Complex+Event+Processor+Documentation
o WSO2 Analytics Platform - http://wso2.com/analytics
o Quick Start Guide doc - https://docs.wso2.com/display/CEP400/Quick+Start+Guide
o Quick Start Guide video - https://www.youtube.com/playlist?list=PLp0TUr0bmhX5v5yBa0TINUktuF-_eDPC_
o Pre Release Webinar on CEP 4.0 - http://wso2.com/library/webinars/2015/07/real-time-analytics-from-data-to-actions-in-milliseconds/
Recommended