Upload
geert-pante
View
676
Download
4
Tags:
Embed Size (px)
DESCRIPTION
Centralized Log Management using the ELK stack: ElasticSearch, Logstash, Kibana
Citation preview
Log Management with ELKELASTICSEARCH, LOGSTASH, KIBANA FOR CENTRALIZED LOGS
Purpose Centralized Log Management
◦ Collect, Parse and Filter using Logstash◦ Store, Index and Search using Elasticsearch◦ Visualize using Kibana
Full open source stack◦ Use for free◦ Support plan from Elasticsearch company
Elasticsearch Real-time search engine
◦ Based on Apache Solr/Lucene◦ Pure Java◦ Document database
◦ Advanced text indexing◦ Fuzzy search
◦ Replication/Sharding for true scalability
Logstash JRuby Based log processor Pluggable event pipeline
◦ Input plugins◦ Filter plugins◦ Codec plugins◦ Output plugins
DevOps Comunity◦ Mix of developers, operations and system administrators
Kibana Browser based dashboard for ElasticSearch Visualization of query results
◦ Time Charts◦ Filter any field◦ Compare subsets
Logstash pipeline Define input, filters and outputs
Simple configuration file
Ruby syntax
Logstash not just for logs Interpretes different log formats
◦ Syslog messages◦ Log4j with full details◦ Apache log files
Other event types too◦ Ganglia server monitoring events◦ SNMP events◦ Windows EventLog
Pre-proces before sending◦ lumberjack
Logstash Inputs Rsyslog via TCP/UDP Log4j appender JMX Listener Logstash-forwarder File tails SNMP Ganglia …
Logstash Filters
Grok Filter Readable regex Predefined patterns for common log data Extract to properties
◦ Indexed properties
Metrics Filter Aggregate metrics
◦ Event rate using sliding windows◦ 1 min◦ 5 min◦ 15 min
◦ Min/max/stddev/percentiles
Logstash Outputs
Statsd output Node.js based
◦ Counters ◦ Timers◦ Graphite frontend
Alert outputsSend alerts
◦ Email◦ Pagerduty◦ XMPP/Jabber◦ Hipchat◦ Nagios
Use treshold from metrics filter
ElasticSearch output Auto-creates new index per day
Index all recognized fields
Full text index, customizable indexer, mapper
Kibana dashboard
Scalability Easy deployment using chef/puppet/docker