Upload
mihaidoru4353
View
228
Download
0
Embed Size (px)
Citation preview
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
1/103
Using Cacti To Graph MySQLs MetricsKenny Gryp
Principal Consultant @ Percona
Collaborate 2011
1
mailto:[email protected]:[email protected]8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
2/103
Percona
MySQL/LAMP Consulting
MySQL Support
Percona Server (XtraDB)
Percona XtraBackup InnoDB Recovery Toolkit, tcprstat
(maatkit, innotop, aspersa, mysql-mmm, mysql-cacti-templates)
...
http://www.percona.com
http://www.mysqlperformanceblog.com
2
http://www.mysqlperformanceblog.com/http://www.mysqlperformanceblog.com/http://www.percona.com/http://www.mysqlperformanceblog.com/http://www.mysqlperformanceblog.com/http://www.percona.com/http://www.percona.com/8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
3/103
Topics
Why Graph?
What is Cacti and mysql-cacti-templates?
How Cacti works
InstallationAdd Graphs
Demo
Graphs!
Extras
3
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
4/103
Topics
Why Graph?
What is Cacti and mysql-cacti-templates?
How Cacti works
InstallationAdd Graphs
Demo
Graphs!
Extras
4
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
5/103
Why Graph?
Troubleshooting
-behavior over time
-sudden spikes in graphs after application updates
Capacity Planning
-traffic patterns
-predictions
Not only for Operations, also for Development and
Management
5
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
6/103
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
7/103
Cacti?
open source graphing solution
uses RRDTool
advanced graphs, scriptable
web interfacequite complex to add graphs
adding servers is a manual process
7
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
8/103
mysql-cacti-templates?
collection of templates for cacti
graph MySQL metrics
Has a lot of non-MySQL Metrics too: IO stats,
apache, memcache, mongodb, java...
8
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
9/103
Topics
Why Graph?
What is Cacti and mysql-cacti-templates?
How Cacti works
InstallationAdd Graphs
Demo
Graphs!
Extras
9
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
10/103
How does Cacti work?
10
Database Server
mysqld
sshd
snmpd
/proc/diskstats
Cacti
MySQL(contains cacticonfiguration)
.rrd Files(contains rrd data)
Apache(serves web
interface)
Poller(cronjob which
fetches data fromservers)
Web Server
httpd
sshd
snmpd
Web Browser
what
stores
fetch
df
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
11/103
Topics
Why Graph?
What is Cacti and mysql-cacti-templates?
How Cacti works
InstallationAdd Graphs
Demo
Graphs!
Extras
11
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
12/103
Installation
Cacti
Mysql-cacti-templates
Configuring Servers
12
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
13/103
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
14/103
Setting up Cacti: Requirements
httpd
php, php-mysql, php-snmp
mysql-server (server and client utilities)
net-snmp (snmpget)rrdtool
14
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
15/103
Setting up Cacti: Installation
Install rpm/deb/... package or extract yourself
Configure webserver: php5, cacti scripts
Create cacti mysql user
Import cacti databaseEdit configuration
Setup cronjob for poller
Finish installation using your webbrowser
15
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
16/103
Setup: Extract
Recommended is using apt/yum/... to install cacti
if not:
16
cacti# cd /var/www/html/ # may varycacti# tar xzvf cacti-0.8.7g.tar.gzcacti# ln -s cacti-0.8.7g cacticacti#chown -R cactiuser rra/ log/
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
17/103
Setup: Webserver
Apache (may be something else)
Enable php
Set DirectoryIndex index.php
17
LoadModule php5_module modules/libphp5.soAddHandler php5-script .phpAddType text/html .phpDirectoryIndex index.php
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
18/103
Setup: MySQL
Create Cacti MySQL user:
Import Cacti Schema
Edit include/config.php
18
cactimysql> GRANT ALL PRIVILEGES ON cacti.* TOcactiuser@localhost IDENTIFIED BY 'password';
cacti#mysql -e CREATE DATABASE cacti;cacti#mysql cacti < cacti.sql
$database_type = "mysql";$database_default = "cacti";$database_hostname = "localhost";$database_username = "cactiuser";
$database_password = "password";
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
19/103
Setup: Poller
Add cronjob (/etc/cron.d/cacti)
19
*/5 * * * * cactiuser php /var/www/html/cacti/poller.php > /dev/null 2>&1
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
20/103
Setup: UI
20
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
21/103
Setup: UI
21
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
22/103
Setup: UI
22
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
23/103
Setup: UI
23
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
24/103
Setup: UI
24
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
25/103
Setup: UI
25
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
26/103
Setup: UI
26
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
27/103
Installation
Cacti
Mysql-cacti-templates
Configuring Servers
27
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
28/103
download & installation
28
cacti#wget http://mysql-cacti-templates.googlecode.com/files/better-cacti-templates-1.1.8.tar.gzcacti# tar -xzf better-cacti-templates-1.1.8.tar.gzcacti# cd better-cacti-templates-1.1.8/cacti#cp \scripts/{ss_get_mysql_stats.php,ss_get_by_ssh.php}\ /var/www/html/cacti/scripts/
http://mysql-cacti-templates.googlecode.com/files/better-cacti-templates-1.1.8.tar.gzhttp://mysql-cacti-templates.googlecode.com/files/better-cacti-templates-1.1.8.tar.gzhttp://mysql-cacti-templates.googlecode.com/files/better-cacti-templates-1.1.8.tar.gzhttp://mysql-cacti-templates.googlecode.com/files/better-cacti-templates-1.1.8.tar.gzhttp://mysql-cacti-templates.googlecode.com/files/better-cacti-templates-1.1.8.tar.gzhttp://mysql-cacti-templates.googlecode.com/files/better-cacti-templates-1.1.8.tar.gz8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
29/103
install: mysql script
29
cacti# vi /var/www/html/cacti/scripts/\ss_get_mysql_stats.php...
$mysql_user = 'cactiuser';$mysql_pass = 'password';...
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
30/103
install: generate ssh key
30
cacti#mkdir /etc/cacticacti# ssh-keygenGenerating public/private rsa key pair.Enter file in which to save the key (/root/.ssh/id_rsa): /etc/cacti/id_rsaEnter passphrase (empty for no passphrase):Your identification has been saved in /etc/cacti/id_rsa.Your public key has been saved in /etc/cacti/id_rsa.pub.cacti# chown -R www-data: /etc/cacti/{id_rsa,id_rsa.pub}cacti# chmod -R 0400 /etc/cacti/id_rsa
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
31/103
install: ssh script
31
cacti# vi /var/www/html/cacti/scripts/ss_get_by_ssh.php...
$ssh_user= 'cacti'; # SSH username$ssh_port= 22; # SSH port$ssh_iden='-i /etc/cacti/id_rsa';# SSH identity...
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
32/103
install: templates
32
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
33/103
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
34/103
install: templates
34
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
35/103
Installation
Cacti
Mysql-cacti-templates
Configuring Servers
35
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
36/103
Configure Servers
configure net-snmpd
add a cacti user in MySQL
configure ssh access
36
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
37/103
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
38/103
Configure Servers: mysql user
Add cacti user
Verify if login works from cacti servers
38
servermysql>GRANT SUPER, PROCESS ON *.* TO'cactiuser'@'cactiserver' IDENTIFIED BY"password";
cacti#mysql -h ubuntu-server-test -e "SELECTVERSION()" -u cactiuser -ppassword
+-------------------+| VERSION() |+-------------------+| 5.1.49-1ubuntu8.1 |
+-------------------+
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
39/103
Configure Servers: ssh access
Add shell user cacti
Copy the public key
Verify
39
server# useradd cactiserver# getent passwd cacticacti:x:1001:1002::/home/cacti:/bin/sh
server#mkdir /home/cacti/.ssh/server# cat > /home/cacti/.ssh/authorized_keys
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
40/103
Topics
Why Graph?
What is Cacti and mysql-cacti-templates?
How Cacti works
Installation
Add Graphs
Demo
Graphs!
Extras
40
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
41/103
Adding Graphs
41
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
42/103
Adding Graphs
42
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
43/103
Adding Graphs
43
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
44/103
Adding Graphs
44
Apply wanted templates on the device-MySQL & GNU/Linux templates
Click on Create Graphs:
Select graphs to create and Add!
Might ask some more questions
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
45/103
Adding Graphs
45
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
46/103
Adding Graphs
46
Assign ucd/net SNMP template to have:-cpu usage/load
-memory used/available
-network statistics
-free disk space
Different way of creating
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
47/103
Adding Graphs
47
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
48/103
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
49/103
Adding Graphs: Graph Trees
49
Tree to browse through graphs
By host/individual graph, mixed
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
50/103
Topics
Why Graph?
What is Cacti and mysql-cacti-templates?
How Cacti works
Installation
Add Graphs
Demo
Graphs!
Extras
50
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
51/103
Topics
Why Graph?
What is Cacti and mysql-cacti-templates?
How Cacti works
Installation
Add Graphs
Demo
Graphs!
Extras
51
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
52/103
MySQL Handlers
52
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
53/103
MySQL Handlers
53
Table scans, backup?
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
54/103
MySQL Handlers
54
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
55/103
MySQL Handlers
55
Take backup here
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
56/103
MySQL Connections
56
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
57/103
MySQL Connections
57
max_connections not reached yet
Why more incoming connections?
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
58/103
MySQL Replication
58
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
59/103
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
60/103
MySQL Replication
60
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
61/103
MySQL Replication
61
In 2 hours, almost 6900seconds replication lag
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
62/103
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
63/103
MySQL Temporary Objects
63
keep an eye on:* temp tables* temp disk tablesif it grows > query optimization
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
64/103
MySQL Select Types
64
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
65/103
MySQL Select Types
65
Why this sudden peak in rangescans?
SQ S
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
66/103
MySQL Select Types
66
M SQL S l T
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
67/103
MySQL Select Types
67
Query behavior changed?Other explain plan chosen?
M SQL S t
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
68/103
MySQL Sorts
68
M SQL S t
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
69/103
MySQL Sorts
69
The peaks should be understood,not necessarily be removed as theymight be really fast in terms of rows
M SQL C d C t
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
70/103
MySQL Command Counters
70
M SQL Bi L
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
71/103
MySQL Binary Logs
71
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
72/103
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
73/103
M SQL Q R Ti
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
74/103
MySQL Query Response Time
74
M SQL Q R Ti
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
75/103
MySQL Query Response Time
75
I DB B ff P l
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
76/103
InnoDB Buffer Pool
76
I DB B ff P l
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
77/103
InnoDB Buffer Pool
77
DB Restart
Time to fill buffer
I DB Ch k i t A
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
78/103
InnoDB Checkpoint Age
78
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
79/103
InnoDB Buffer Pool Activity
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
80/103
InnoDB Buffer Pool Activity
80
InnoDB I/O
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
81/103
InnoDB I/O
81
InnoDB I/O
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
82/103
InnoDB I/O
82
More readrequests Increase in
load
InnoDB Row Operations
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
83/103
InnoDB Row Operations
83
InnoDB Row Operations
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
84/103
InnoDB Row Operations
84
Good view on capacity of the system
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
85/103
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
86/103
86
more row reads,coming from disk
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
87/103
87
this spike in IO readsdid not cause more disk reads
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
88/103
88
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
89/103
CPU Usage
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
90/103
CPU Usage
90
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
91/103
Disk Operations
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
92/103
Disk Operations
92
Topics
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
93/103
Topics
Why Graph?What is Cacti and mysql-cacti-templates?
How Cacti works
InstallationAdd Graphs
Demo
Graphs!
Extras
93
non-MySQL mysql-cacti-templates
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
94/103
non-MySQL mysql-cacti-templates
ApacheJMX
Memcached
MongodbNginx
OpenVZ
Redis
Unix
94
non-MySQL mysql-cacti-templates
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
95/103
non-MySQL mysql-cacti-templates
95
non-MySQL mysql-cacti-templates
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
96/103
non-MySQL mysql-cacti-templates
96
non-MySQL mysql-cacti-templates
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
97/103
non-MySQL mysql-cacti-templates
97
non-MySQL mysql-cacti-templates
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
98/103
non-MySQL mysql-cacti-templates
98
Other Graphing solutions
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
99/103
Other Graphing solutions
Munin and munin-mysqlOpenNMS and MySQL-SNMP
Zabbix with appaloosa-zabbix-templates
99
Creating Your Own Graphs
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
100/103
Creating Your Own Graphs
Creating cacti graphs not trivial/portablemysql-cacti-templates to the rescue!
http://code.google.com/p/mysql-cacti-templates/wiki/CreatingGraphs
100
Using Cacti To Graph MySQLs Metrics
http://code.google.com/p/mysql-cacti-templates/wiki/CreatingGraphshttp://code.google.com/p/mysql-cacti-templates/wiki/CreatingGraphshttp://code.google.com/p/mysql-cacti-templates/wiki/CreatingGraphshttp://code.google.com/p/mysql-cacti-templates/wiki/CreatingGraphs8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
101/103
Using Cacti To Graph MySQL s Metrics
Powerful ToolEasy to install
Open source!
101
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
102/103
Percona Live MySQL Conference NYCMay 26, 2011
8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics
103/103
May 26, 2011
http://www.percona.com/live/nyc-2011/
http://www.percona.com/live/nyc-2011/http://www.percona.com/live/nyc-2011/