Upload
anjana-fernando
View
151
Download
0
Tags:
Embed Size (px)
Citation preview
Scalable Log Analysis with WSO2 BAM
Anjana Fernando
Senior Technical LeadWSO2 Inc.
Agenda
• Logging Requirement• Thinking Big• Log Publishing• Log Storage• Log Analysis• Log Search• Custom Dashboards / Alerts• Demo• Future Improvements
Logging Requirement
• Record interested events of a system• Analyse the log events• Take appropriate actions with the analysis
Thinking Big
• Not your typical logger• WSO2’s logging solution is made
from ground up for large deployments
• Utilizes a big data architecture for logging
• WSO2 BAM as the framework
Source: fishingforsoul.wordpress.com
Log Publishing
• Asynchronous• Thrift protocol based event streams• Custom Log4J pattern layout for capturing:
• Tenant information• Server information• Application information
• Not only for Java applications, you can write your own log publisher using any language,
Log Storage
• Multi-level storage• Most recent logs stored in Cassandra• Archived logs store in HDFS
Source: www.carolinasit.com
Log Analysis
• Hive/Hadoop based log summarisation and archiving
• Log archiving daily stored as a hierarchy of tenants and applications
• Custom analytics possible via custom Hive scripts and CEP integration
Log Search
• Search the most recent logs• System Logs
• View logs from all applications, search by log level, keywords• System Logs
• View logs from a specific application, search by log level, keywords
Custom Dashboards / Alerts
• Google Gadgets / Jaggery application based dashboards• Alerting support using in-built CEP features in BAM
• Supports output adaptors such as Email, SMS, HTTP, JMS with message types text, XML, JSON etc..
WSO2 Distributed Logging Architecture
Demo
Future Improvements
• Out of the box support for many other log formats• A mechanism to define custom formats
• Improved searching capabilities• Better indexing for archived logs
• Support for additional data stores, i.e. not only Cassandra, but MongoDB, RDBMS etc.., required for better embeddability with other products
Questions?
Thank You