Upload
gwendolyn-cannon
View
219
Download
0
Tags:
Embed Size (px)
Citation preview
THE ART AND METHODOLOGY OF TROUBLESHOOTING
Alexey Diomin, [email protected]
Production
Agenda
Logging Offline analysis Online analysis
Monitoring
System monitoring Zabbix Nagios
Application Monitoring New Relic Logstash Kibana
Zabbix. CPU load example
Zabbix. Memory example
Zabbix. CMS Old Gen
Logstash. Kibana
“We need to go deeper”
“We need to go deeper”
jstack (thread dump) jstack <pid> kill -QUIT <pid>
jstack
Very simple profiler Lock detection
jstack example
“We need to go deeper”
jstack (thread dump) jstack <pid> kill -QUIT <pid>
jmap jmap -dump <pid> jmap -histo <pid|file>
jmap
Eclipse Memory Analyzer (MAT) http://www.eclipse.org/mat/
jvisualvm http://visualvm.java.net/
MAT
MAT
Leak Detection
RingBuffer
MAT
“We need to go deeper”
jstack (thread dump) jstack <pid> kill -QUIT <pid>
jmap jmap -dump <pid> jmap -histo <pid|file>
JFR / JMC
JFR
JFR
Event Type
JFR
Basic features: Code Profile Allocation profile
JFR
Basic features: Code Profile Allocation profile
Killer features: Contentions Exceptions rate
Code profile
Allocation profile
Netty’s bug example
Contention
Hbase latency
Contentions example
Exceptions rate
Context Switch
Exception example
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException
1. “You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ….”
SQL syntax for prepared statements does not support multi-statements (that is, multiple statements within a single string separated by “;” characters).
2. “Can't create more than max_prepared_stmt_count statements”
useServerPrepStmts
Use server-side prepared statements if the server supports them
useServerPrepStmts
Summary
Logging Offline analysis Online analysis
Links
Java Performance: The Definitive Guide http://shop.oreilly.com/product/0636920028499.do
Troubleshooting Guide for Java SE 6 with HotSpot VM
http://www.oracle.com/technetwork/java/javase/tsg-vm-149989.pdf
Black boxes? Not in
troubleshooting!!!