Upload
wso2
View
687
Download
3
Embed Size (px)
Citation preview
Data to Insight: Introduction to WSO2 Business Activity Monitor
Anjana FernandoSenior Technical LeadWSO2 Inc.
Outline● Introduction to Business Activity Monitoring
● WSO2 BAM v3.0: Next Generation Analytics
● WSO2 BAM High Availability Distributed Deployment
● WSO2 BAM Solutions
Business Activity Monitoring
WSO2 BAM v3.0
● What’s new?
○ Faster analytics with Apache Spark, 10x - 100x speedups
○ Rich indexing support
○ Pluggable data stores, from light-weight embedded RDBMS to highly scalable HDFS
○ Embeddable architecture for inclusion with other Carbon servers
Data Agents● Compatible with CEP/BAM
● Get data across to BAM
− Service monitoring feature – WSO2 AS, DSS, ESB, API Manager
− Mediation monitoring feature – BAM Mediator for WSO2 ESB
− Custom data-agents
● Asynchronous & non-blocking
● Multiple data transport support with Thrift, Kafka, JMS, MQTT and more.
● BAM analytics specific REST service for index operations, data retrieval and search
Data Receiver● Receives data and stores it in the backend data store
○ Pluggable data store, ranging from light-weight databases to highly scalable big data stores
● Asynchronous & non-blocking
○ Combination of Cassandra, Thrift and the non-blocking nature results in extremely fast writes
● Shared with WSO2 CEP for real time analysis
● Supports plugging in of different receiver types
Data Model● Data is sent over using strongly typed, versioned data streams
{ 'name':'phone.retail.shop', 'version':'1.0.0', 'nickName': 'Phone_Retail_Shop', 'description': 'Phone Sales', 'metaData':[ {'name':'clientType','type':'STRING'} ], 'payloadData':[ {'name':'brand','type':'STRING'}, {'name':'quantity','type':'INT'}, {'name':'total','type':'INT'}, {'name':'user','type':'STRING'} ]}
● BAM Analytics store it in a backend table store structure, with optional indexing, for efficient data lookup, pagination and near-realtime search
Analytics Data Abstraction Layer● A well defined data source API for exposing storage for analytics record and index store
○ Analytics Record Store
■ A schema-less table record store for storing individual records, with timestamps and
pagination supported
○ Analytics File System
■ A file system structure implementation to be used by the data indexing
implementation
○ Initial set of connectors to be supported
■ RDBMS - with any RDBMS supported using a configuration based query templates
■ HDFS
■ MongoDB
The Analyzer Engine● Powered by Apache Spark with querying through Spark SQL
● Parallel, distributed processing with optimized in-memory computing
● Outperforms Hadoop in efficiency and speed, making smaller deployments feasible for typical analytics tasks
● Clustered using Spark embedded Carbon servers
The Analyzer Engine● Runs on a Spark engine embedded Carbon server cluster
○ Scalable analytics
○ Cluster can range from a couple of nodes to 1000s
● Analysis is carried out on an interactive query console and analytics scripts
● Queries are based on an easy-to-learn, SQL-like query language
INSERT INTO TABLE UserTable SELECT userName, COUNT(DISTINCT orderID), SUM(quantity) FROM PhoneSalesTable WHERE version= "1.0.0" GROUP BY userName;
The Analyzer Engine● Interactive Query Console
○ Queries are entered in an console and executed one by one in the cluster, and the results are
sent back asynchronously to the console
● Scripts can be scheduled
○ e.g.:- once a minute, every Wednesday at 4:15 p.m., every 30th at 12 midnight
● Connects to the standard analytics data store
○ Any type of backend data store is supported, to read data to be analyzed and resultant data can also be written out using the same interface
○ Single set of analytics scripts / toolboxes for any type of backing data store
Distributed Data Indexing● Data indexing support with full text indexing
● Data drill down support with facets
● Based on Apache Lucene, high performant, feature rich, indexing engine
● Distributed indexing support using sharded Lucene indexes
○ Horizontal scaling of index storage and indexing performance
○ Designed to be able to dynamically add more nodes later
● Near real-time data indexing and retrieval
○ Data is indexed immediately when the data is retrieved by the event receiver
○ Batched data indexing for higher performance
The Presentation Layer● Gadget based dashboards for visualization
○ Custom user dashboard and gadget creation using wizards
○ Data drill-down views
● Activity Dashboard for correlating activities
● Message Console - one stop shop for managing the analytics data store
○ Full support for paginated data retrieval and timestamp based filtering
○ Data table creation, index association, row insertion, updates, deletions
○ Batch data upload using files
○ Rich search functionality for indexed tables
● Log analysis dashboard - out-of-the box log analysis solution
The Presentation LayerBAM Dashboards
The Presentation LayerGadget Portal
The Presentation LayerMessage Console
Message Console (cont'd)
BAM Toolboxes● BAM Toolboxes are installable and hot deployable artifacts used for deploying functionalities to a
BAM server
○ Stream definitions○ Analytics scripts○ Dashboards
● Toolboxes for monitoring and auditing most WSO2 products are available OOTB
● Toolboxes for custom scenarios can be created easily
H/A Distributed BAM deployment● WSO2 BAM can be clustered and deployed in a distributed manner to enable high-availability, fail-
over scenarios
○ Distributed deployment
■ All components of BAM are clustered (data receiver, storage, analyzer and presentation)
■ Hazelcast in-memory data grids are used for clustering implementation
■ Optimized for a simpler deployment
Service Statistics Monitoring● Improved set of analytics and visualization for web services and web applications
● Available with upcoming WSO2 AS v6.0 release
● In-built dashboard in WSO2 AS with light-weight analytics module
○ Configurable to point to an external BAM server
Service Statistics Monitoring
Service Statistics Monitoring
Service Statistics Monitoring
Service Statistics Monitoring
Service Statistics Monitoring
Service Statistics Monitoring
Service Statistics Monitoring
Service Statistics Monitoring
Activity Monitoring● Activity monitoring is for tracking events from multiple nodes in a flow to understand a specific
activity
○ e.g.:-
■ A client initiating a web services request which travels through multiple ESBs,
application servers and returns back. This flow will be uniquely identified and
visualized in BAM
○ Used for tracing messages, finding performance hotspots in the flow
○ Implemented based on a correlation id based mechanism and indexing
○ Upcoming: Mediator level tracing and profiling in WSO2 ESB 5.0
Activity Monitoring
Activity Monitoring
Log Analysis● OOTB log analysis solution in BAM v3.0
● Log event indexing
○ Uses the new BAM v3.0 indexing support
○ Event attributes can be indexed to be search by server, cluster, log type and also log
messages itself for full text search
● Custom search queries using Lucene queries and regular expressions
● Log agent client for reading and publishing log events from log files
○ No need to install a special agent inside the target servers itself
○ Extensible mechanism for specifying log formats in parsing
Log Analysis
Log Analysis
Questions?
Thank you!