27
Logstash find happiness in your logs with Elasticsearch ELK

OSDC 2014: Jordan Sissel - Find Happiness in your Logs

  • Upload
    netways

  • View
    400

  • Download
    0

Embed Size (px)

DESCRIPTION

Got logs? With so much technology powering your business, you need tools to help you identify problems and analyze past behavior. Apache 2.0-licensed Elasticsearch ELK stack is here to help you process, store, and visualize any kind of logging data, in real time, from any source imaginable! Log management seems so boring. Log rotation, retention policy, grep, yuck! What are your servers are doing? Did last night's upgrade break anything? How your users are interacting with your products? Why did the site go down last weekend? Get ready to turn your log pains into awesome visual insights and more! BAM! Elasticsearch ELK! ELK stands for Elasticsearch, Logstash, and Kibana. Each of these three are lovely, open source projects that, together, give you and your business log management superpowers. This talk will primarily be done in three parts: open source and community, technology, and use cases. * The first part will introduce each project and its success as open source software, most notably through supportive and open communities. * The second part will discuss the each project and the problems solved. * The third (and most exciting!) part will highlight a variety of use cases and problem that real humans are using Elasticsearch ELK to solve. Live demos of some use cases will be provided. Attendees will leave the presentation totally full of excitement about this toolset and bursting with fresh ideas about how to tackle their sour logging problems.

Citation preview

Page 1: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Logstashfind happiness in your logs

with Elasticsearch ELK

Page 2: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Open Source

Page 3: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Apache 2.0 License

Open Source

Page 4: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Open and Friendly Community

Open Source

Page 5: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

If a new user has a bad time, it’s a bug.

Open Source

Page 6: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Technology

Page 7: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Logstash

Page 8: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Processing & Transport

Logstash

Page 9: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Logstash

FiltersInputs Outputs42 51 54

Processing & Transport

Page 10: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Logstash Processing & Transport

LAMP Example

ApacheMysqlPHP

Syslog

NagiosElasticsearch

Graphite

IRC

Parse: Latency, Bandwidth, and Errors

Load Balancer

Page 11: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Logstash Example Inputs

Files Graphite

SNMPEmail

Syslog

TwitterTCP

RabbitMQ

Page 12: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Logstash Example Filters

Grok Date

FingerprintGeoIPMultiline

User AgentKey-Value

Page 13: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Logstash Example Outputs

Elasticsearch Graphite

XMPPNagios

Email PagerdutyS3

Page 14: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Elasticsearch

Page 15: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Near Real-Time Search & Analysis

Elasticsearch

Page 16: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Scalable

Elasticsearch

Page 17: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

REST + JSON API

Elasticsearch

Page 18: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Kibana

Page 19: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Visualization & Exploration

Kibana

Page 20: OSDC 2014: Jordan Sissel - Find Happiness in your Logs
Page 21: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Use Cases

Page 22: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Share Logs with Your Tech Support Team

Use Cases

Page 23: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Graph all things!

Use Cases

Page 24: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Exploration by Non-Technical Users

Use Cases

Page 25: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Troubleshooting Latency

?Late

ncy

Time

X

Page 26: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Movie Releases Demo

Use Cases

Page 27: OSDC 2014: Jordan Sissel - Find Happiness in your Logs

Apache Logs Demo

Use Cases