VP  Developer  Evangelism    Samisa  Abeysinghe  

Big  Brother  for  Enterprises:    Log  Analysis  Use  Cases  

19  Feb  2014  

2  

About  the  Presenter  

๏  Samisa  Abeysinghe  VP  Developer  Evangelism  samisa@wso2.com    

๏  Samisa  Abeysinghe,  Vice  President  of  Developer  Evangelism  joined  the  company  in  September  2005.    Prior  to  the  current  role,  Samisa  used  to  be  VP  of  Engineering  and  managed  the  development  of  WSO2  Carbon  based  product  plaOorm.    

3  

About  WSO2  ๏  Global  enterprise,  founded  in  2005  

by  acknowledged  leaders  in  XML,  web  services    technologies,  standards    and  open  source  

๏  Provides  only  open  source  plaOorm-­‐as-­‐a-­‐service  for  private,  public  and  hybrid  cloud  deployments  

๏  All  WSO2  products  are  100%  open  source  and  released  under  the  Apache  License  Version  2.0.  

๏  Is  an  AcXve  Member  of  OASIS,  Cloud  Security  Alliance,  OSGi  Alliance,  AMQP  Working  Group,  OpenID  FoundaXon  and  W3C.  

๏  Driven  by  InnovaXon  

๏  Launched  first  open  source  API  Management  soluXon  in  2012  

๏  Launched  App  Factory  in  2Q  2013  

๏  Launched  Enterprise  Store  and  first  open  source  Mobile  soluXon  in  4Q  2013  

4  

What  WSO2  Deliver  

NSA like Monitoring for Your Enterprise ๏ Analyze volumes of data

๏ Address correlation complexities in analytics

๏ Off line vs Real time operations implications

๏ Some operations got to be in real time, else the value is lost

๏ Summarized data over time (and other) dimensions for analytics

A “Big Brother” that keeps an eye on the whole enterprise

5  

Why should I bother? ๏ Deal with high volume (terra bites) of information

๏ In order to make decisions

๏ Real time & Offline

๏ Take action

6  

WSO2 :ONLY COMPLETE & INTEGRATED Platform ๏ Complete and integrated for

๏ Data capture

๏ Analysis: both real-time and batch

๏ Visualization

๏ Action taking business process execution

Making data driven intelligence for your enterprise easy

7  

WSO2  Big  Data  Analy0cs  Pla4orm  for  Your  Enterprise  

WSO2 Big Data Analytics Platform for Your Enterprise

8  

Key  Elements  

Data  CollecXon  

Data  Analysis  

Data  VisualizaXon  

Taking  AcXon  

9  

Use  Case  Scenario  1:  Monitor  your  Java  ApplicaXon  System  Logs    

with  BAM  &  CEP

10  

11  

& WSO2 CEP

Overview  of  SoluXon    o  Send Log Events to

o  Business Activity Monitor (BAM) & o  Complex Event Processor (CEP)

o  Real time Log Event Processing o  With CEP

o  Batch Processing of Log Data o  With BAM analytics

o  Visualization of Log Data o  With Gadgets on Dashboards

12  

Log  Event Publishing  (BAM)    

13  

Event  Streams  &  AlerXng  (CEP)  

14  

WSO2 CEP

LogEvent  Stream  q  Meta Data

q  clientType {String} q  Meta Data

q  tenantID {String} q  ServerName {String} q  appName {String} q  logTime {Long} q  priority {Long} q  message {String} q  logger {String} q  ip {String} q  instance {String} q  stacktrace {String}

15  

CEP  Query  

from LogEvents [priority == "ERROR"]

select message, stacktrace, serverName insert into ExceptionStream

Email Body  Error Occurred in {{serverName}} – {{message}} {{stacktrace}}

16  

AnalyXcs  &  Batch  Processing    (BAM)  

17  

Hive Query  CREATE EXTERNAL TABLE IF NOT EXISTS LogEventInfo (key STRING, tenantID INT,serverName STRING, appName STRING, priority STRING,logTime DOUBLE,logger STRING,message STRING) STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' WITH SERDEPROPERTIES ( "cassandra.host" = "localhost", "cassandra.port" = "9160","cassandra.ks.name" = "EVENT_KS", "cassandra.ks.username" = "admin","cassandra.ks.password" = "admin", "cassandra.cf.name" = "log_0_AS_2014_01_23", "cassandra.columns.mapping" = ":key,payload_tenantID,payload_serverName,payload_appName, payload_priority,payload_logTime,payload_logger,payload_message" ); CREATE EXTERNAL TABLE IF NOT EXISTS Logs(tenantID INT,serverName STRING, appName STRING, priority STRING,logTime DOUBLE,logger STRING,message STRING) STORED BY 'org.wso2.carbon.hadoop.hive.jdbc.storage.JDBCStorageHandler' TBLPROPERTIES ( 'mapred.jdbc.driver.class' = 'com.mysql.jdbc.Driver', 'mapred.jdbc.url' = 'jdbc:mysql://localhost:3306/MYBAMDB', 'mapred.jdbc.username' = 'root','mapred.jdbc.password' = 'root', 'hive.jdbc.update.on.duplicate' = 'true', 'hive.jdbc.table.create.query' = 'CREATE TABLE LogEvent(tenantID INT,serverName VARCHAR(200), appName VARCHAR(200), priority VARCHAR(200),logTime DOUBLE,logger VARCHAR(800),message VARCHAR(3800))'); insert overwrite table Logs select tenantID, serverName, appName, priority, logTime, logger, message from LogEventInfo; select tenantID, serverName, appName, priority, logTime, logger, message from LogEventInfo;

18  

VisualizaXon -­‐  Gadgets  

19  

Demo  …..  http://wso2.com/library/demonstrations/2014/02/screencast-monitoring-system-logs-with-wso2-business-activity-monitor/

20  

Use  Case  Scenario  2:  HTTP  Log  Monitoring  With  WSO2  BAM  

21  

HTTPD  Logs  and  Use  Cases    •  Monitor every web request information

–  HTTP method –  URI –  Status code

•  Monitor request properties –  Message size –  Host / IP address –  Geo location –  Date and time

22  

HTTPD Logs:  Scenarios  •  Publish HTTP logs to BAM from a data agent •  Split log entry for each fields and store in another big

data column family •  Resolve IP addresses to geo locations •  Aggregate request for geo locations and store into a

relational database (RDB) •  Visualize from gadgets

23  

SoluXon  Architecture    

24  

Key  Performance  Indicator  (KPI)    Use  Cases  •  Analysing request count variation for each host •  Analysing request count variation for each hour •  Analysing request message size variation

25  

Final  Dashboard  

26  

Demo  …..  http://wso2.com/library/demonstrations/2014/02/screencast-http-log-monitoring-with-wso2-business-activity-monitor/

27  

28  

Business  Model  

29  

Call  to  acXon  page  ๏  Be  your  own  NSA:  hhp://wso2.com/landing/nsa/    

๏  Big  Data  Webinar  Series:  hhp://wso2.com/landing/wso2-­‐bigbrother-­‐webinar-­‐series/    

๏  WSO2  Business  AcXvity  Monitor:  hhp://wso2.com/products/business-­‐acXvity-­‐monitor/    

๏  WSO2  BAM  Docs:  hhp://docs.wso2.org/display/BAM240/WSO2+Business+AcXvity+Monitor+DocumentaXon    

๏  WSO2  Complex  Event  Processor:  hhp://wso2.com/products/complex-­‐event-­‐processor/    

๏  WSO2  CEP  Docs:  hhp://docs.wso2.org/display/CEP300/WSO2+Complex+Event+Processor+DocumentaXon    

Contact  us  !