THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

THE ART AND METHODOLOGY OF TROUBLESHOOTING

Alexey Diomin, diominay@gmail.com

Production

Agenda

Logging Offline analysis Online analysis

Monitoring

System monitoring Zabbix Nagios

Application Monitoring New Relic Logstash Kibana

Zabbix. CPU load example

Zabbix. Memory example

Zabbix. CMS Old Gen

Logstash. Kibana

“We need to go deeper”

“We need to go deeper”

jstack (thread dump) jstack <pid> kill -QUIT <pid>

jstack

Very simple profiler Lock detection

jstack example

“We need to go deeper”

jstack (thread dump) jstack <pid> kill -QUIT <pid>

jmap jmap -dump <pid> jmap -histo <pid|file>

jmap

Eclipse Memory Analyzer (MAT) http://www.eclipse.org/mat/

jvisualvm http://visualvm.java.net/

MAT

MAT

Leak Detection

RingBuffer

MAT

“We need to go deeper”

jstack (thread dump) jstack <pid> kill -QUIT <pid>

jmap jmap -dump <pid> jmap -histo <pid|file>

JFR / JMC

JFR

JFR

Event Type

JFR

Basic features: Code Profile Allocation profile

JFR

Basic features: Code Profile Allocation profile

Killer features: Contentions Exceptions rate

Code profile

Allocation profile

Netty’s bug example

Contention

Hbase latency

Contentions example

Exceptions rate

Context Switch

Exception example

com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException

1. “You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ….”

SQL syntax for prepared statements does not support multi-statements (that is, multiple statements within a single string separated by “;” characters).

2. “Can't create more than max_prepared_stmt_count statements”

useServerPrepStmts

Use server-side prepared statements if the server supports them

useServerPrepStmts

Summary

Logging Offline analysis Online analysis

Links

Java Performance: The Definitive Guide http://shop.oreilly.com/product/0636920028499.do

Troubleshooting Guide for Java SE 6 with HotSpot VM

http://www.oracle.com/technetwork/java/javase/tsg-vm-149989.pdf

Black boxes? Not in

troubleshooting!!!