Performance Schema in MySQL (Danil Zburivsky)

Performance Schema in MySQL Danil Zburivsky

MySQL DBA and Team Lead at Pythian

Tuesday, October 23, 12

About myself• MySQL DBA and Team

Lead at Pythian

• Managing dozens of customers and thousands of MySQL servers

• http://www.pythian.com/news/author/zburivsky/

• @zburivsky


http://www.pythian.com/news/author/zburivsky/




Debugging MySQL Performance


MySQL is a great database, but instrumentation sucks

SHOW ENGINE INNODB STATUS

Slow query log

OS STATSSHOW

PROCESSLIST


I think there is a problem with database...



• Identify timeframe (use trending tools)




• Collect data during incident





• Collect more data






• Collect as much data as you can!







• Try to make sense of it








• Tune settings, SQL, hardware. Did it help?








• Tune settings, SQL, hardware. Did it help?

• I think there is a problem with database...


Tools that might help you

• pt-stalk (http://www.percona.com/doc/percona-toolkit/2.1/pt-stalk.html)

• innotop (http://code.google.com/p/innotop/)

• Or you write your own stuff


http://www.percona.com/doc/percona-toolkit/2.1/pt-stalk.html




http://code.google.com/p/innotop/




What do “Big Boys” have?

• Oracle: SQL-traces, kernel events timings, tons of books on performance tuning

• SQL Server: SQL profiler, Data Management Views, Extended Events,tons of books on performance tuning


Performance schema in MySQL 5.5


The Idea

• Executing a query breaks down to hundreds of smaller tasks

• There are background tasks as well

• We want to instrument it all to know where server is spending time


Implementation

• Instrumentation: measuring when event begins and ends

• Implemented in MySQL code on server end storage engine level

• Can be enabled/disabled or customized


Schema and a storage engine+----------------------------------------------+| Tables_in_performance_schema |+----------------------------------------------+| cond_instances || events_waits_current || events_waits_history || events_waits_history_long || events_waits_summary_by_instance || events_waits_summary_by_thread_by_event_name || events_waits_summary_global_by_event_name || file_instances || file_summary_by_event_name || file_summary_by_instance || mutex_instances || performance_timers || rwlock_instances || setup_consumers || setup_instruments || setup_timers || threads |+----------------------------------------------+17 rows in set (0.00 sec)












Instruments+------------------------------------------------------------+| NAME |+------------------------------------------------------------+| wait/synch/mutex/sql/PAGE::lock || wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_sync || wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_active || wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_pool |..... || wait/synch/rwlock/sql/LOCK_grant || wait/synch/rwlock/sql/LOGGER::LOCK_logger |

......| wait/synch/cond/sql/COND_thread_count | | wait/synch/cond/sql/COND_thread_cache |..... || wait/io/file/sql/binlog || wait/io/file/sql/binlog_index |+------------------------------------------------------------+


Consumers

SELECT * FROM setup_consumers;+----------------------------------------------+---------+| NAME | ENABLED |+----------------------------------------------+---------+| events_waits_current | YES || events_waits_history | YES || events_waits_history_long | YES || events_waits_summary_by_thread_by_event_name | YES || events_waits_summary_by_event_name | YES || events_waits_summary_by_instance | YES || file_summary_by_event_name | YES || file_summary_by_instance | YES |+----------------------------------------------+---------+8 rows in set (0.00 sec)


What are top wait events in my server?

SELECT COUNT_STAR, SUM_TIMER_WAIT, AVG_TIMER_WAIT FROM events_waits_summary_by_instance ORDER BY SUM_TIMER_WAIT DESC LIMIT 10;+---------------------------------------------+------------+-------------------+----------------+| EVENT_NAME | COUNT_STAR | SUM_TIMER_WAIT | AVG_TIMER_WAIT |+---------------------------------------------+-----------------------+---------------+---------|| wait/io/file/innodb/innodb_data_file | 3347631 | 13547848609831118 | 4046995803 || wait/synch/rwlock/innodb/index_tree_rw_lock | 69535441 | 2000129499580014 | 28764173 || wait/synch/mutex/innodb/buf_pool_mutex | 800909600 | 433331246983734 | 541048 || wait/synch/mutex/innodb/log_sys_mutex | 181356697 | 420610626124392 | 2319245 || wait/synch/rwlock/innodb/index_tree_rw_lock | 3060704 | 232839801243104 | 76073936 || wait/io/file/innodb/innodb_data_file | 84259 | 163336572508164 | 1938505946 || wait/synch/rwlock/innodb/btr_search_latch | 192529196 | 161285917556962 | 837721 || wait/synch/mutex/sql/LOCK_open | 285022231 | 154822982324406 | 543196 || wait/synch/mutex/innodb/kernel_mutex | 403820715 | 130291049763820 | 322645 || wait/synch/mutex/mysys/THR_LOCK::mutex | 285017708 | 121061567338910 | 424751 |+---------------------------------------------+------------+-------------------+----------------+10 rows in set (0.03 sec)


innodb_buffer_pool_instances=10

SELECT * FROM events_waits_summary_by_instance ORDER BY SUM_TIMER_WAIT DESC LIMIT 10;+-----------------------------------------------+------------+-------------------+----------------+| EVENT_NAME | COUNT_STAR | SUM_TIMER_WAIT | AVG_TIMER_WAIT |+-----------------------------------------------+------------+-------------------+----------------+| wait/io/file/innodb/innodb_data_file | 3600080 | 14130028463169872 | 3924920685 || wait/synch/rwlock/innodb/index_tree_rw_lock | 69313641 | 2045091134703870 | 29504886 || wait/synch/mutex/innodb/trx_doublewrite_mutex | 6432834 | 592011487365760 | 92029654 || wait/synch/mutex/innodb/log_sys_mutex | 183016201 | 459928639055248 | 2513048 || wait/synch/rwlock/innodb/index_tree_rw_lock | 3239574 | 277212671866944 | 85570717 || wait/io/file/innodb/innodb_data_file | 400484 | 219787308922610 | 548804219 || wait/synch/rwlock/innodb/btr_search_latch | 190363676 | 183251774563990 | 962640 || wait/synch/mutex/sql/LOCK_open | 285022522 | 144911560909258 | 508421 || wait/synch/mutex/mysys/THR_LOCK::mutex | 285018012 | 130317830152708 | 457226 || wait/synch/mutex/innodb/kernel_mutex | 401734769 | 126815854421630 | 315670 |+-----------------------------------------------+------------+-------------------+----------------+10 rows in set (0.02 sec)

buf_pool_mutex is no longer in top 10 wait events


Which files are being accessed the most?

mysql> SELECT SUM_NUMBER_OF_BYTES_READ, SUM_NUMBER_OF_BYTES_WRITE FROM file_summary_by_instance ORDER BY SUM_NUMBER_OF_BYTES_READ+SUM_NUMBER_OF_BYTES_WRITE DESC LIMIT 10;+------------------------------+--------------------------+---------------------------+| FILE_NAME |SUM_NUMBER_OF_BYTES_READ | SUM_NUMBER_OF_BYTES_WRITE |+------------------------------+--------------------------+---------------------------+| /var/lib/mysql/ibdata1 | 166920192 | 363649155072 || /var/lib/mysql/WIR.ibd | 180310720512 | 114136367104 || /var/lib/mysql/EMIWPR.ibd | 51803815936 | 71776468992 || /var/lib/mysql/EWAPR.ibd | 38596755456 | 54104932352 || /var/lib/mysql/OFPR.ibd | 24171593728 | 33964572672 || /var/lib/mysql/OFIR.ibd | 38002688000 | 14265303040 || /var/lib/mysql/EQPO.ibd | 12718882816 | 21407596544 || /var/lib/mysql/FIR.ibd | 19646103552 | 10337140736 || /var/lib/mysql/EIR.ibd | 14708899840 | 12814794752 || /var/lib/mysql/FPR.ibd | 6360350720 | 9764585472 |+------------------------------+--------------------------+---------------------------+10 rows in set (0.01 sec)


Threads table(performance_schema) > SELECT * FROM threads;+-----------+----------------+----------------------------------------+| THREAD_ID | PROCESSLIST_ID | NAME |+-----------+----------------+----------------------------------------+| 0 | 0 | thread/sql/main || 12 | 0 | thread/innodb/srv_lock_timeout_thread || 14 | 0 | thread/innodb/srv_monitor_thread || 16 | 0 | thread/innodb/srv_purge_thread || 13 | 0 | thread/innodb/srv_error_monitor_thread || 2 | 0 | thread/innodb/io_handler_thread || 1 | 0 | thread/innodb/io_handler_thread || 9 | 0 | thread/innodb/io_handler_thread || 25 | 8 | thread/sql/one_connection || 3 | 0 | thread/innodb/io_handler_thread || 15 | 0 | thread/innodb/srv_master_thread || 10 | 0 | thread/innodb/io_handler_thread || 5 | 0 | thread/innodb/io_handler_thread || 8 | 0 | thread/innodb/io_handler_thread || 7 | 0 | thread/innodb/io_handler_thread || 17 | 0 | thread/sql/signal_handler || 4 | 0 | thread/innodb/io_handler_thread || 6 | 0 | thread/innodb/io_handler_thread |+-----------+----------------+----------------------------------------+18 rows in set (0.00 sec)


Average wait per thread (I)

SELECT e.THREAD_ID, e.EVENT_NAME, MAX(e.AVG_TIMER_WAIT)FROM events_waits_summary_by_thread_by_event_name e LEFT JOIN threads t ON t.THREAD_ID = e.THREAD_IDWHERE event_name LIKE 'wait/synch/mutex/innodb/%' AND t.NAME='thread/sql/one_connection'AND AVG_TIMER_WAIT > 0GROUP BY t.thread_id, e.EVENT_NAMEORDER BY t.thread_id, MAX(e.AVG_TIMER_WAIT)


Average wait per thread (II)

+-----------+-------------------------------------------------------+-------------------------+| THREAD_ID | EVENT_NAME | MAX(AVG_TIMER_WAIT) |+-----------+-------------------------------------------------------+-------------------------+| 27 | wait/synch/mutex/innodb/mutex_list_mutex | 49463 || 27 | wait/synch/mutex/innodb/purge_sys_bh_mutex | 83247 |

.....

.....| 27 | wait/synch/mutex/innodb/ibuf_bitmap_mutex | 115681 || 27 | wait/synch/mutex/innodb/ibuf_pessimistic_insert_mutex | 168537 || 27 | wait/synch/mutex/innodb/ibuf_mutex | 282400 || 27 | wait/synch/mutex/innodb/log_sys_mutex | 1690694 |----------------------------------------------------------------------------------------------| 28 | wait/synch/mutex/innodb/mutex_list_mutex | 49504 || 28 | wait/synch/mutex/innodb/flush_list_mutex | 52375 || 28 | wait/synch/mutex/innodb/log_flush_order_mutex | 62040 |

.....

.....| 28 | wait/synch/mutex/innodb/autoinc_mutex | 63012 || 28 | wait/synch/mutex/innodb/dict_sys_mutex | 181203 || 28 | wait/synch/mutex/innodb/ibuf_mutex | 222376 || 28 | wait/synch/mutex/innodb/innobase_share_mutex | 239514 || 28 | wait/synch/mutex/innodb/ibuf_pessimistic_insert_mutex | 266684 || 28 | wait/synch/mutex/innodb/log_sys_mutex | 1718503 |


Average wait per thread (III)

+-----------+--------------------------------------------------+-------------------------+| THREAD_ID | EVENT_NAME | MAX(ebt.AVG_TIMER_WAIT) |+-----------+--------------------------------------------------+-------------------------+| 27 | wait/synch/mutex/innodb/ibuf_mutex | 55872 || 27 | wait/synch/mutex/innodb/flush_list_mutex | 59495 || 27 | wait/synch/mutex/innodb/mutex_list_mutex | 59499 || 27 | wait/synch/mutex/innodb/autoinc_mutex | 64136 |

.....

.....| 27 | wait/synch/mutex/innodb/ibuf_bitmap_mutex | 88242 || 27 | wait/synch/mutex/innodb/buf_pool_mutex | 101598 || 27 | wait/synch/mutex/innodb/kernel_mutex | 148332 || 27 | wait/synch/mutex/innodb/log_sys_mutex | 4671112 |-----------------------------------------------------------------------------------------| 28 | wait/synch/mutex/innodb/ibuf_mutex | 42886 || 28 | wait/synch/mutex/innodb/mutex_list_mutex | 56646 || 28 | wait/synch/mutex/innodb/flush_list_mutex | 58105 || 28 | wait/synch/mutex/innodb/autoinc_mutex | 66873 |

....

....| 28 | wait/synch/mutex/innodb/purge_sys_bh_mutex | 83545 || 28 | wait/synch/mutex/innodb/ibuf_bitmap_mutex | 87697 || 28 | wait/synch/mutex/innodb/buf_pool_mutex | 101365 || 28 | wait/synch/mutex/innodb/kernel_mutex | 149135 || 28 | wait/synch/mutex/innodb/log_sys_mutex | 4829134 |

innodb_change_buffering = none [Don’t try this at home! ]


Picoseconds, shmicosecond! ps_helper to rescue!

• http://www.markleith.co.uk/ps_helper/

• Great examples of using performance_schema

• Useful tools for converting time, bytes, paths into human readable format


http://www.markleith.co.uk/ps_helper/

http://www.markleith.co.uk/ps_helper/

Example: top_global_io_consumers_by_bytes_usage

(ps_helper) > SELECT event_name, total_read, avg_read, total_written, avg_written,avg_latency FROM top_global_io_consumers_by_bytes_usage;+-------------------------+------------+-----------+---------------+-------------+-------------+| event_name | total_read | avg_read | total_written | avg_written | avg_latency |+-------------------------+------------+-----------+---------------+-------------+-------------+| innodb/innodb_data_file | 10.76 GiB | 16.00 KiB | 16.05 GiB | 30.90 KiB | 10.96 ms || innodb/innodb_log_file | 68.00 KiB | 11.33 KiB | 1.03 GiB | 369.66 KiB | 2.33 ms || myisam/dfile | 197.79 KiB | 8.60 KiB | 187.04 KiB | 5.84 KiB | 9.89 µs || sql/FRM | 142.28 KiB | 491 bytes | 4.50 KiB | 242 bytes | 3.66 µs || sql/ERRMSG | 43.68 KiB | 14.56 KiB | 0 bytes | 0 bytes | 15.42 µs || myisam/kfile | 13.22 KiB | 226 bytes | 5.96 KiB | 24 bytes | 8.26 µs || mysys/charset | 17.83 KiB | 17.83 KiB | 0 bytes | 0 bytes | 10.68 µs || sql/partition | 1.06 KiB | 32 bytes | 0 bytes | 0 bytes | 4.38 µs || sql/pid | 0 bytes | 0 bytes | 6 bytes | 6 bytes | 34.78 µs || sql/global_ddl_log | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 3.69 µs || archive/data | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 4.35 µs || sql/dbopt | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 4.55 µs || sql/casetest | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 28.08 µs |+-------------------------+------------+-----------+---------------+-------------+-------------+13 rows in set (0.00 sec)


No free lunch: PFS overhead

• Instrumentation doesn’t come for free

• In CPU-bound workloads overhead can be ~15-20%

• In IO-bound — 5%-8%

• Significant improvements in 5.6


Performance schema in MySQL 5.6


What’s new in 5.6?

• performance_schema enabled by default!

• Less overhead: 5%-10% for CPU-bound workload

• Statements, Stages, Actors and Objects


Actors: filter events by user

(performance_schema) > SELECT * FROM setup_actors;+------+------+------+| HOST | USER | ROLE |+------+------+------+| % | % | % |+------+------+------+1 row in set (0.00 sec)


Objects: filter out events by database/table

(performance_schema) > SELECT * FROM setup_objects;+-------------+--------------------+-------------+---------+-------+| OBJECT_TYPE | OBJECT_SCHEMA | OBJECT_NAME | ENABLED | TIMED |+-------------+--------------------+-------------+---------+-------+| TABLE | mysql | % | NO | NO || TABLE | performance_schema | % | NO | NO || TABLE | information_schema | % | NO | NO || TABLE | % | % | YES | YES |+-------------+--------------------+-------------+---------+-------+4 rows in set (0.00 sec)


StatementsSELECT * FROM events_statements_summary_by_digest\G

*************************** 4. row *************************** DIGEST: f60b0866d7cbd78d39caf5fb52f43bfb DIGEST_TEXT: SELECT * FROM `table_io_waits_summary_by_index_usage` ORDER BY `SUM_TIMER_WAIT` DESC LIMIT ? COUNT_STAR: 1 SUM_TIMER_WAIT: 511539000 MIN_TIMER_WAIT: 511539000 AVG_TIMER_WAIT: 511539000 MAX_TIMER_WAIT: 511539000 SUM_LOCK_TIME: 72000000 SUM_ERRORS: 0 SUM_WARNINGS: 0 SUM_ROWS_AFFECTED: 0 SUM_ROWS_SENT: 5 SUM_ROWS_EXAMINED: 91SUM_CREATED_TMP_DISK_TABLES: 0 SUM_CREATED_TMP_TABLES: 0 SUM_SELECT_FULL_JOIN: 0 SUM_SELECT_FULL_RANGE_JOIN: 0 SUM_SELECT_RANGE: 0 SUM_SELECT_RANGE_CHECK: 0 SUM_SELECT_SCAN: 1 SUM_SORT_MERGE_PASSES: 0 SUM_SORT_RANGE: 0 SUM_SORT_ROWS: 5 SUM_SORT_SCAN: 1 SUM_NO_INDEX_USED: 1 SUM_NO_GOOD_INDEX_USED: 0 FIRST_SEEN: 2012-10-16 21:50:13 LAST_SEEN: 2012-10-16 21:50:13


Summary. Pros.

• Provides insight into internal server metric

• Flexible

• Relational model allows to build your own views on top of this data


Summary. Cons.

• Steep learning curve: not very well documented

• Overhead for CPU-bound loads can be significant


Thank you!


Documents

Performance Schema in MySQL (Danil Zburivsky)