Application performance management with PacketBeat, Elasticsearch and Kibana

Tudor Golubenco (@tudor_g)

What is PacketBeat? ¯\_(ツ)_/¯

What is PacketBeat

• “Open Source Application Monitoring”

• “Monitoring & Troubleshooting for Distributed Applications”

• “Distributed Wireshark with a lot more analytics features”

• “Application Performance Management”

How it works? ಠ_ಠ

How it works

• Captures the wire traffic

• Follows TCP streams, decodes HTTP, MySQL, PgSQL, REDIS, Thrift-RPC

• Looks for requests, waits for the matching response

• Records response time, URLs, response codes, etc

Show me! (￣^￣)

What do we do with the data? ¯\(°_o)/¯

The traditional way

• Decide what metrics you need (requests per second for each server, response time percentiles, etc.)

• Write code to extract these metrics, store them in a DB

• Store the transactions in a DB

• Drilling down is difficult

• Features like “Top 10 method with errors” are difficult to implement

PacketBeat + ELK

Why ELK?

• Already proven to scale and perform for logs

• Clear and simple flow for the data

• You don’t have to pre-create the metrics

• Ad-hoc troubleshooting and analytics by using Kibana

• Drilling down to the problematic transactions is trivial

• Top N features are trivial

• Slicing by different dimensions is easy

Show me! (￣^￣)

Pros of wire data

• Captures a lot of things that other approaches miss

• No changes to the code or to the monitored application

• Minimal knowledge about the monitored app is required

• No latency overhead

• When using tap points, zero CPU/memory overhead on the app servers

Cons of wire data

• There can be, like, tons of data

• Compared to log processing, larger CPU requirements

• Privacy concerns

• Doesn’t work for encrypted protocols

• Doesn’t work for “in-house” protocols

Next steps ( ͡° ͜ʖ ͡°)

More protocols

• Available:

• HTTP

• MySQL

• PostgreSQL

• REDIS

• Thrift-RPC

• Soon (tm):

• DNS

• Memcache

• MongoDB, RethinkDB

• Oracle, MSSQL

• XMLRPC / JSONRPC

• Your suggestions?

Sampling

• Wire data can be huge

• Troubleshooting convenience vs hardware requirements

• Sample by:

• protocol (e.g. store all MySQL requests, sample REDIS 1/10)

• method (e.g. store all PUTs requests, sample GETs 1/10)

• status code (e.g. store all errors, sample successes)

• response time (e.g. store all slow transactions)

String obfuscation

• Replace: select * from users where username=“Tudor” and id=3

• With: select * from users where username=S8 and id=N3

• Makes TopN charts better

• “The Mature Optimisation Handbook” - Carlos Bueno

Bonito

• Our own UI

• Similar to Kibana, but focused more on app performance

• Will be a Kibana 4 plugin

Deploying PacketBeat (´ ▽｀).。ｏ♡

Deploying

• Getting started guide

• packetbeat-deploy

• ansible roles for Packetbeat, Elasticsearch, Logstash, Redis, Kibana

• supports multiple ES nodes or all-in-one server

• ansible-playbook -i hosts site.yml

Thanks! ( ﾟ▽ﾟ)/

Keep in touch

• Twitter: @packetbeat or @tudor_g

• www: packetbeat.com

• github.com/packetbeat/packetbeat