Upload
mats-kindahl
View
151
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Presentation on replication tips and tricks given at the MySQL Connect 2012 conference.
Citation preview
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 122
Replication Tips & TricksMats [email protected]
3 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Program Agenda
Replication setup and status checking
Binary log analysis
Crash-safe slaves
Multi-source replication
Slave fail-over using GTID
4 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
About the Presentation
This presentation will introduce you to some replication features and also briefly show tips and tricks on how to work with replication. The focus is on short ideas and each item does not go deeper into details.
5 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Replication Architecture
Master Slave
6 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Reading Slave Status
● I/O Thread Status
– Slave_IO_Running
– Last_IO_Errno
– Last_IO_Error
– Last_IO_Error_Timestamp
● SQL Thread Status
– Slave_SQL_Running
– Last_SQL_Errno
– Last_SQL_Error
– Last_SQL_Error_Timestamp
● Master being replicated from
– Master_Host
– Master_Port
Understanding the fields of SHOW SLAVE STATUS
7 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Reading Slave Status
● Next event to execute – in master log coordinates
– Relay_Master_Log_File + Exec_Master_Log_Pos
● Next event to execute – in relay log coordinates
Relay_Log_File + Relay_Log_Pos
● Next event to read from master
Master_Log_File + Read_Master_Log_Pos
Understanding the fields of SHOW SLAVE STATUS
8 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Reading the Binary Log
● Use mysqlbinlog
--hexdump Show hex dump of event as comment
--start-position Start dumping at a position
--stop-position Stop dumping after this position
# at 275#120927 23:11:58 server id 3 end_log_pos 373 # Position Timestamp Type Master ID Size Master Pos Flags # 113 1e c1 64 50 02 03 00 00 00 62 00 00 00 75 01 00 00 00 00
Query Little-endian
17516=37310
Decoding the binary log
9 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Reading the Binary Log
# at 275#120927 23:11:58 server id 3 end_log_pos 373 # Position Timestamp Type Master ID Size Master Pos Flags # 113 1e c1 64 50 02 03 00 00 00 62 00 00 00 75 01 00 00 00 00# 126 c5 03 00 00 00 00 00 00 04 00 00 1a 00 00 00 00 |................|# 136 00 00 01 00 00 00 00 00 00 00 00 06 03 73 74 64 |.............std|# 146 04 21 00 21 00 08 00 74 65 73 74 00 69 6e 73 65 |.......test.inse|# 156 72 74 20 69 6e 74 6f 20 74 32 20 76 61 6c 75 65 |rt.into.t2.value|# 166 73 20 28 31 2c 27 74 65 73 74 69 6e 67 27 29 |s..1..testing..|# Query thread_id=965 exec_time=0 error_code=0SET TIMESTAMP=1348780318/*!*/;insert into t2 values (1,'testing')/*!*/;
Decoding the binary log
10 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Reading the Binary Log
● Use mysqlbinlog
--verbose Decode row events into pseudo-SQL
# at 849# at 893#120928 15:19:24 server id 3 end_log_pos 893 Table_map: `test`.`t2` mapped to number 48#120928 15:19:24 server id 3 end_log_pos 941 Write_rows: table id 48 flags: STMT_END_F
BINLOG '3KNlUBMDAAAALAAAAH0DAAAAADAAAAAAAAEABHRlc3QAAnQyAAIDDwIoAAM=3KNlUBcDAAAAMAAAAK0DAAAAADAAAAAAAAEAAv/8AwAAAA1yb3dzIGFyZSBjb29s'/*!*/;
Decode row events
11 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Reading the Binary Log
● Use mysqlbinlog
--verbose Decode row events into pseudo-SQL
# at 849# at 893#120928 15:19:24 server id 3 end_log_pos 893 Table_map: `test`.`t2` mapped to number 48#120928 15:19:24 server id 3 end_log_pos 941 Write_rows: table id 48 flags: STMT_END_F
BINLOG '3KNlUBMDAAAALAAAAH0DAAAAADAAAAAAAAEABHRlc3QAAnQyAAIDDwIoAAM=3KNlUBcDAAAAMAAAAK0DAAAAADAAAAAAAAEAAv/8AwAAAA1yb3dzIGFyZSBjb29s'/*!*/;### INSERT INTO test.t2### SET### @1=3### @2='rows are cool'
Decode row events
12 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Cloning a Slave
1. Stop the slave
2. Write down where slave has stopped
– SHOW SLAVE STATUS– Relay_Master_Log_File + Exec_Master_Log_Pos
3. Backup slave
– mysqldump – slow but safe– Physical file copy – fast but you need to shut down the server
Creating a slave image
13 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Cloning a Slave
4. Restore backup on new slave
5. Direct the slave to the master
– CHANGE MASTER– Master_Host + Master_Port– Relay_Master_Log_File + Exec_Master_Log_Pos
6. Start slave
– START SLAVE
Setting up the new slave
14 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Cloning a Slave
● Using mysqldbcopy
mysqldbcopy replication=slave \ locking=snapshot \ source=root:[email protected] \ destination=root:[email protected] \ database ...
The easy way – using MySQL Utilities
15 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Cloning a Slave
● Using mysqldbcopy
mysqldbcopy replication=slave \ locking=lockall \ source=root:[email protected] \ destination=root:[email protected] \ database ...
The easy way – using MySQL Utilities
16 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Point in time recovery
● Use mysqlbinlog
--to-datetime Stop reading when reaching date● Example
mysqlbinlog --read-from-remote-server \ --host=master -uroot --position=192 \
--to-datetime=”2009-04-11 12:36:56” \master-bin.00002[2-4] |
mysql -uroot --host=slave
Warning
: times
tamps
are
not mo
notonic
ally inc
reasing
!
17 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Using Relay Servers
● Relieve master server by creating relay slaves
● Just keep binary logs
● Do not store data in tables
– Use BLACKHOLE engine– --log-slave-updates
SET STORAGE_ENGINE = BLACKHOLEALTER TABLE table ENGINE = BLACKHOLE
18 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Using Relay Servers
● Filtering replication stream
● Filtering done on relay servers
● replicatewild*table
– Work with cross-database queries
replicate-wild-do-table-wild=*.%_east replicate-wild-do-table-wild=*.%_west
19 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Write Your Own Library
● Create a utility library to easier work with replication
● Common functions to manage replication:
– Start/stop slave– Change master– Fetch master position– Fetch slave execute/read position– Fetch master host information
Write your own utility library
20 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Write Your Own Library
# Usage: mysql_exec socket sql ... mysql_exec () { sock=$1 shift 1 mysql uroot vertical batch \ skipcolumnnames \ socket="$sock" \ exec "$*"}
# Usage: stop_slave socket [ thread ]stop_slave () { mysql_exec $1 STOP SLAVE $2}
# Usage: start_slave socket [ thread ]start_slave () { mysql_exec $1 START SLAVE $2}
Basic functions using Bourne shell
21 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Write Your Own Library
# Usage: change_master socket [ host [ port [ file [ pos ] ] ] ]change_master () { host=${2:+MASTER_HOST=\'$2\'} port=${3:+,MASTER_PORT=$3} file=${4:+,MASTER_LOG_FILE=\'$4\'} pos=${5:+,MASTER_LOG_POS=$5} mysql_exec $1 CHANGE MASTER TO $host $port $file $pos}
# Usage: fetch_master_pos socketfetch_master_pos () { mysql_exec $1 SHOW MASTER STATUS | grep '\<File\|\<Pos' | cut f2 d:}
Basic functions using Bourne shell
22 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Write Your Own Library
# Usage: fetch_slave_exec_pos socketfetch_slave_exec_pos () { mysql_exec $1 SHOW SLAVE STATUS | grep '\<Relay_Master_Log_File\|\<Exec_Master_Log_Pos' | cut f2 d:}
# Usage: fetch_slave_read_pos socketfetch_slave_read_pos () { mysql_exec $1 SHOW SLAVE STATUS | grep '\<Master_Log_File\|\<Read_Master_Log_Pos' | cut f2 d:}
Basic functions using Bourne shell
23 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Round-Robin Replication
● Slave can only replicate from single master at a time
● Use time sharing to replicate from several masters
Switcher
master1.example.com master2.example.com
slave1.example.com
Stop
Save
Restore
Start
24 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
# Usage: fetch_host_and_pos socketfetch_host_and_pos () { mysql_exec $1 SHOW SLAVE STATUS | grep '\<Master_\(Host\|Port\|Log_File\)\|\<Read_Master_Log_Pos' | cut -f2 -d:}
# Usage: stop_and_save socketstop_and_save () { sock="/var/run/mysqld/$1.sock" stop_slave $socket fetch_host_and_pos $sock >$1.savepos}
Round-Robin Replication
A version using Bourne Shell
25 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
restore_and_start () { socket="/var/run/mysqld/$1.sock" cat $1.savepos | { read host read port read file read pos change_master $socket \ $host $port $file $pos start_slave $socket }}
Round-Robin Replication
cnt=1
while truedo stop_and_save mysqld.$cnt cnt=`expr $cnt % 5 + 1` restore_and_start mysqld.$cnt sleep 60done
A version using Bourne Shell
26 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Crash-safe Slaves
● Traditional Replication
– Position stored in file– Update after transaction– Crash can lose update
● Transactional Replication
– Positions stored in table– Update part of transaction– Crash-safe
● Repository location FILE or TABLE
master_info_repositoryrelay_log_info_repository
FILE TABLE
Keeping replication information in sync with data
27 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Crash-safe Slaves
● Direct server to use table-based repository
– Defaults to FILE
master_info_repository=TABLErelay_log_info_repository=TABLE
● Ensure that a transactional engine is used
– InnoDB default since 5.6.6– Before 5.6.6: set the storage engine
ALTER TABLE slave_master_info ENGINE = InnoDBALTER TABLE slave_relay_log_info ENGINE = InnoDB
Steps to set up crash-safe slaves
28 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CREATE TABLE current_master ( idx INT) ENGINE=InnoDB
CREATE TABLE my_masters ( idx INT PRIMARY KEY, host CHAR(50) NOT NULL, port INT NOT NULL DEFAULT 3306, log_file CHAR(50), log_pos LONG, UNIQUE INDEX (host,port,user)) ENGINE=InnoDB
Round-Robin Replication
CREATE PROCEDURE save_position()BEGIN DECLARE l_idx INT UNSIGNED;
UPDATE my_masters AS mi, mysql.slave_relay_log_info AS rli SET mi.log_pos = rli.master_log_pos, mi.log_file = rli.master_log_name WHERE idx = (SELECT idx FROM current_master);END
A version using pure SQL
29 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Round-Robin Replication
In pure SQL
CREATE PROCEDURE fetch_next_master(OUT p_host CHAR(50), OUT p_port INT UNSIGNED, OUT p_file CHAR(50), OUT p_pos BIGINT)BEGIN DECLARE l_next_idx INT DEFAULT 1;
SELECT idx INTO l_next_idx FROM my_masters WHERE idx > (SELECT idx FROM current_master) ORDER BY idx LIMIT 1;
SELECT idx INTO l_next_idx FROM my_masters WHERE idx >= l_next_idx ORDER BY idx LIMIT 1;
UPDATE current_master SET idx = l_next_idx;
SELECT host, port, log_file, log_pos INTO p_host, p_port, p_file, p_pos FROM my_masters WHERE idx = l_next_idx;END
Select next index, if there is one
Fetch master info using the index
Select the first index, if there were no next index
30 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Round-Robin Replication
In pure SQL
CREATE EVENT multi_source ON SCHEDULE EVERY 10 SECOND DOBEGIN DECLARE l_host CHAR(50); DECLARE l_port INT UNSIGNED; DECLARE l_file CHAR(50); DECLARE l_pos BIGINT;
SET SQL_LOG_BIN = 0;
STOP SLAVE; START TRANSACTION; CALL save_current_position(); CALL fetch_next_master(l_host, l_port, l_file, l_pos); CALL change_master(l_host, l_port, 'repl_user', 'xyzzy', l_file, l_pos); COMMIT; START SLAVE;END
31 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Different Tables on Master and Slave
● Upgrading
– Example: Upgrading a topology without downtime● Saving space
– Not keeping big columns on slave● Hiding data
– Keeping data away from users
– Note: still available in binary log and relay log – just not applied to table● Debugging and Auditing
– Adding data for auditing
32 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CREATE TABLE employee ( name CHAR(40), email CHAR(40), changed TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP)
CREATE TABLE employee ( name CHAR(40), email CHAR(40)
)
Different Tables on Master and Slave
● Same initial columns
● Statement-based or row-based replication
More columns on slave
Master Slave
33 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CREATE TABLE employee ( name CHAR(40), email CHAR(40), password CHAR(40))
CREATE TABLE employee ( name CHAR(40), email CHAR(40), password CHAR(40))
Different Tables on Master and Slave
● Same initial columns
● Row-based replication only
More columns on master
Master Slave
34 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CREATE TABLE employee ( emp_id TINYINT, name CHAR(40), email CHAR(40))
CREATE TABLE employee ( emp_id INT, name CHAR(40), email CHAR(40))
Different Tables on Master and Slave
● Table definitions are identical
… except that some column type are different
Different types of columns on master and slave
Master Slave
35 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CREATE TABLE employee ( emp_id TINYINT, name CHAR(40), email CHAR(40))
CREATE TABLE employee ( emp_id INT, name CHAR(40), email CHAR(40))
Different Tables on Master and Slave
● Consider this statement:
INSERT INTO employee VALUES (1, 'Bob', '[email protected]'), (500, 'Alice', '[email protected]')
Slave type conversions
Master Slave
Statement-based
ReplicationWorks fine
Fails silently!
36 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CREATE TABLE employee ( emp_id INT, name CHAR(40), email CHAR(40))
CREATE TABLE employee ( emp_id TINYINT, name CHAR(40), email CHAR(40))
Different Tables on Master and Slave
● Consider this statement:
INSERT INTO employee VALUES (1, 'Bob', '[email protected]'), (100, 'Alice', '[email protected]')
Slave type conversions
Master Slave
Row-basedReplicationThrows an error!
37 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Different Tables on Master and Slave
● @@GLOBAL.SLAVE_TYPE_CONVERSIONS
– Type: SET('ALL_LOSSY', 'ALL_NON_LOSSY')
– Require slave restart
● Non-lossy conversions
– TINYINT INT→
– VARCHAR(32) CHAR(64)→
– FLOAT DOUBLE→
● Lossy conversions
– INT TINYINT→
– CHAR(64) VARCHAR(32)→
– DOUBLE FLOAT →
Slave type conversions
38 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Different Tables on Master and Slave
● Conversions within same domain possible
– INT, TINYINT, SMALLINT, MEDIUMINT, BIGINT– CHAR(n), VARCHAR(n), TINYTEXT, TEXT, …– DECIMAL(n,m), DOUBLE, FLOAT
● ALL_NON_LOSSY in SLAVE_TYPE_CONVERSIONS
– Conversion to larger domains allowed
● ALL_LOSSY in SLAVE_TYPE_CONVERSIONS
– Conversion to smaller domains allowed (truncation/rounding may occur)
Slave type conversions
39 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CREATE TABLE employee ( emp_id INT, name VARCHAR(40), email VARCHAR(40))
CREATE TABLE employee ( emp_id TINYINT, name VARCHAR(40), email VARCHAR(40))
Different Tables on Master and Slave
● Consider this statement:
INSERT INTO employee VALUES (1, 'Bob', '[email protected]'), (100, 'Alice', '[email protected]')
Slave type conversions
Master Slave
Row-basedReplicationWorks fine!
SLAVE_TYPE_CONVERSIONS = 'ALL_NON_LOSSY'
40 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CREATE TABLE employee ( emp_id TINYINT, name VARCHAR(40), email VARCHAR(40))
CREATE TABLE employee ( emp_id INT, name VARCHAR(40), email VARCHAR(40))
Different Tables on Master and Slave
● Consider this statement:
INSERT INTO employee VALUES (1, 'Bob', '[email protected]'), (100, 'Alice', '[email protected]')
Slave type conversions
Master Slave
Row-basedReplicationThrows an error!
SLAVE_TYPE_CONVERSIONS = 'ALL_NON_LOSSY'
41 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Master and Slave out of sync
● Slave stopped with a strange error
ERROR 1062 (23000): Duplicate entry '3' for key 'PRIMARY'
● Compare two databases using mysqldbcompare
mysqldbcompare [email protected] [email protected] employee
42 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Master and Slave out of sync
$ mysqldbcompare --quiet --server1=root@localhost:3309 --server2=root@localhost:3310 world# Checking databases world on server1 and world on server2#
## Data differences found among rows:--- world.City+++ world.City@@ -1,5 +1,5 @@-+-------+------------+--------------+-----------+-------------+-| ID | Name | CountryCode | District | Population |-+-------+------------+--------------+-----------+-------------+-| 3048 | Stockholm | SWE | Lisboa | 750348 |-+-------+------------+--------------+-----------+-------------+++-------+-----------+--------------+-----------+-------------++| ID | Name | CountryCode | District | Population |++-------+-----------+--------------+-----------+-------------++| 3048 | Helsinki | SWE | Lisboa | 750348 |++-------+-----------+--------------+-----------+-------------+
Comparing two databases and finding the differences
43 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Master and Slave out of sync
$ mysqldbcompare --quiet --difftype=sql --changes-for=server2 \> [email protected] [email protected] \> world# Checking databases world on server1 and world on server2#
## Transformation for --changes-for=server2:#
# Data differences found among rows:UPDATE world.City SET Name = 'Stockholm' WHERE ID = '3048';
Synchronizing master and slave databases
44 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Master and Slave out of sync
$ mysqldbcompare --quiet --difftype=sql --changes-for=server1 \> [email protected] [email protected] \> world# Checking databases world on server1 and world on server2#
## Transformation for --changes-for=server1:#
# Data differences found among rows:UPDATE world.City SET Name = 'Helsinki' WHERE ID = '3048';
Synchronizing master and slave databases
45 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Replication Event ChecksumsMaster Slave
EventChecksums
46 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Replication Event Checksums
● Controlling checksum generation (default is CRC32 since 5.6.6)
SET GLOBAL BINLOG_CHECKSUM = CRC32
● Enable verification on events read by dump thread
SET GLOBAL MASTER_VERIFY_CHECKSUM = ON
● Enable verification on events read by SQL thread
SET GLOBAL SLAVE_SQL_VERIFY_CHECKSUM = ON
Configurations and optionsConfigurations and options
47 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CHANGE MASTER TO MASTER_HOST='master1.example.com', MASTER_PORT=3306, MASTER_USER='repl_user', MASTER_PASSWORD='xyzzy', MASTER_LOG_FILE='master-bin.00001', MASTER_LOG_POS=22145;
Using Global Transaction ID for Fail-over
● Binary log positions manually handled
● Each server has it's own position
● Failing over slaves to new masters difficult
– What is the position to fail over to?
Handling positions in MySQL 5.5
Different fordifferent servers
48 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Using Global Transaction ID for Fail-over
● MySQL 5.6 has Global Transaction ID
– Positions independent of server● GTID handshake
– Done on connection with master– Negotiate position automatically
CHANGE MASTER TO MASTER_HOST='master1.example.com', MASTER_PORT=3306, MASTER_USER='repl_user', MASTER_PASSWORD='xyzzy', MASTER_LOG_FILE='master-bin.00001', MASTER_LOG_POS=22145; MASTER_AUTO_POSITION=1;
Enabling Global Transaction IDs
[mysqld]…gtid-mode=ondisable-gtid-unsafe-statementslog-bin=master-binlog-slave-updates
49 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
STOP SLAVE;CHANGE MASTER TO MASTER_HOST='master2.example.com', MASTER_PORT=3306;START SLAVE;
Using Global Transaction ID for Fail-over
● Failing over slave to another master
– Switch over – old master server is online– Fail over – old master server is down
● GTID handshake
– Automatically done when changing master
Failing over to a new master
50 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.