38
THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, [email protected]

THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, [email protected]

Embed Size (px)

Citation preview

Page 1: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

THE ART AND METHODOLOGY OF TROUBLESHOOTING

Alexey Diomin, [email protected]

Page 2: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Production

Page 3: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Agenda

Logging Offline analysis Online analysis

Page 4: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Monitoring

System monitoring Zabbix Nagios

Application Monitoring New Relic Logstash Kibana

Page 5: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Zabbix. CPU load example

Page 6: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Zabbix. Memory example

Page 7: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Zabbix. CMS Old Gen

Page 8: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Logstash. Kibana

Page 9: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

“We need to go deeper”

Page 10: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

“We need to go deeper”

jstack (thread dump) jstack <pid> kill -QUIT <pid>

Page 11: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

jstack

Very simple profiler Lock detection

Page 12: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

jstack example

Page 13: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

“We need to go deeper”

jstack (thread dump) jstack <pid> kill -QUIT <pid>

jmap jmap -dump <pid> jmap -histo <pid|file>

Page 14: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

jmap

Eclipse Memory Analyzer (MAT) http://www.eclipse.org/mat/

jvisualvm http://visualvm.java.net/

Page 15: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

MAT

Page 16: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

MAT

Page 17: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Leak Detection

Page 18: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

RingBuffer

Page 19: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

MAT

Page 20: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

“We need to go deeper”

jstack (thread dump) jstack <pid> kill -QUIT <pid>

jmap jmap -dump <pid> jmap -histo <pid|file>

JFR / JMC

Page 21: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

JFR

Page 22: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

JFR

Page 23: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Event Type

Page 24: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

JFR

Basic features: Code Profile Allocation profile

Page 25: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

JFR

Basic features: Code Profile Allocation profile

Killer features: Contentions Exceptions rate

Page 26: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Code profile

Page 27: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Allocation profile

Page 28: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Netty’s bug example

Page 29: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Contention

Page 30: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Hbase latency

Page 31: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Contentions example

Page 32: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Exceptions rate

Page 33: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Context Switch

Page 34: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Exception example

com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException

1. “You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ….”

SQL syntax for prepared statements does not support multi-statements (that is, multiple statements within a single string separated by “;” characters).

2. “Can't create more than max_prepared_stmt_count statements”

useServerPrepStmts

Use server-side prepared statements if the server supports them

Page 35: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

useServerPrepStmts

Page 36: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Summary

Logging Offline analysis Online analysis

Page 37: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Links

Java Performance: The Definitive Guide http://shop.oreilly.com/product/0636920028499.do

Troubleshooting Guide for Java SE 6 with HotSpot VM

http://www.oracle.com/technetwork/java/javase/tsg-vm-149989.pdf

Page 38: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

Black boxes? Not in

troubleshooting!!!