Event Driven Solution to monitor Datacenters through continuous queries and machine learning

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based SystemCambridge, United Kingdom

HOLMES: An event-driven solution to monitor data centers through continuous queries and

machine learning

Pedro Henriques dos Santos TeixeiraRicardo Gomes Clemente

Ronald Andreu KaiserDenis Almeida Vieira Jr

Topics

• Motivation• Use Case• The Solution

• Overview• System architecture• CEP• Machine learning• CEP & Machine learning integration• Visualization and User Interface

• Conclusion

Motivation

• Non-stop growing environment, dynamic• Understand our environment• Too many dependencies• Can't afford downtime

Motivation

• Monitoring can be tricky• Precede the inevitable and try to avoid chaos• 1.2K servers• 14K+ monitored items• Correlation

Use Case

• Big Brother Brazil• New world record• 151 million votes in 2 days• Peaks of 13500 votes per minute (~220 v/s)• DDoS atack detected

Overview

The System Architecture

HOLMES

System architecture – modules and its purposes

• CEP module: known problems• Machine learning module: unknown problems• Visualization module: situational awareness• Storage: events history/log

• Reaction to incidents in real-time is a requirement for data center monitoring

• Expression of abstract rules related to the business is desirable

• Correlation of events through user-defined queries

CEP - Esper

• Open source CEP Implementation

• Supports an EPL

• High throughput, requirement in our context

• Ease of embed in our application

CEP – simple example

SELECT avg(response_time) FROM HTTP.win:time(5 min)

E1E5 E4 E3 E2 E1

events stream

response time...

4 t.u. 3 t.u. 2 t.u. 3 t.u. 5 t.u.

If the number of sessions increase in 10% in a 3 minute window and the

average of cpu's usage of the web farm do not

increase in 5% and the number of slow queries in

the database is higher than 10, then we have achieved a

database contention situation. Alarm it!

If the number of sessions increase in 10% in a 3 minute window and the

average of cpu's usage of the web farm do not

increase in 5% and the number of slow queries in

the database is higher than 10, then we have achieved a

database contention situation. Alarm it!

Machine learning“any signal, which is totally predictable, carries no information” - Shannon

Machine learning characteristics

• FRAHST learns to detect anomalous behaviors

• Unsupervised streaming algorithm

• Linear complexity to the number of data streams

FRAHST, state-of-the-art

For further information, see reference [12] in our paper.

Anomaly detection

CEP & Machine Learning Integration

• Users choose the data streams to be correlated

• CEP module aggregates events

• Notifications are raised whether a rank variance is detected

Visualization and User Interface

• Users can create Perspectives

• Real-time dashboard personalizations

• Events history visualization

Dashboards

Conclusion

• Successfully implementation and acceptance in a real use case

• New challenges• improving situational

awareness & prediction• Make creation of queries

more intuitive

This presentation:

http://www.slideshare.net/intelie/debs2010

Our Nagios Plugin source code:

http://github.com/intelie/neb2activemq

Intelligent Monitoring with Esper:

http://esper.codehaus.org/tutorials/tutorial/presentations.html

Denis Vieira Jr. - davieira@gmail.com Ronald Kaiser - ronald@intelie.com.br

Event Driven Solution to monitor Datacenters through continuous queries and machine learning

Technology

Continuous Obstructed Nearest Neighbor Queries in Spatial

Incremental Aggregation on Multiple Continuous Queries

Continuous Queries in Oracle - VLDB Endowment Inc

Data Streams & Continuous Queries The Stanford STREAM Project stanfordstreamdatamanager

C AWARE PROCESSING OF CONTINUOUS LOCATION …€¦ · Query language for navigation-related queries in indoor environments Continuous processing of location-dependent queries 3 DISCUSSION

Efficient Algorithm to Monitor Continuous kNN Queries

Models and Operators for Continuous Queries on Data Streams

Continuous Location Dependent Queries in Mobile Wireless Sensor

Optimization of Continuous Queries in Federated Database and Stream Processing Systems

Efï¬cient Evaluation of Continuous Text Search Queries

CQL: A Language for Continuous Queries over Streams and Relations

Continuous Queries over Append-Only Databases

2005/11/09 Continuous Queries in P2P Networks. Motivation

Continuous Processing of Preference Queries in Data Streams : a Survey

Admission Control Mechanisms for Continuous Queries in the ... · Admission Control Mechanisms for Continuous Queries in the Cloud Keywords the cloud, admission control mechanisms,

Higher SLA Satisfaction in Datacenters with Continuous

Anonymizing continuous queries with delay-tolerant mix ... › bpalan › papers › delaytolerant-dapd.pdfAnonymizing continuous queries with delay-tolerant ... introduce three types

Distributed Evaluation of Continuous Equi-join Queries over Large …20Idreos%20... · Distributed Evaluation of Continuous Equi-join Queries over Large Structured Overlay Networks

The CoQUOS Approach to Continuous Queries in Unstructured Overlays

A Unified Algorithm for Continuous Monitoring of Spatial Queries