Embed Size (px)
DESCRIPTION
Tungsten Replicator is an innovative and reliable tool that can solve your most complex replication problems. In this webinar we introduce Replicator installation and show you how to use key Replicator features effectively with MySQL. Course Topics: - Checking host and MySQL prerequisites - Downloading code from http://code.google.com/p/tungsten-replicator/ - Installation using the tpm utility - Transaction filtering using standard filters as well as customized filters you write yourself - Enabling and managing parallel replication - Configuring multi-master and fan-in using multiple replication services - Backup and restore integration - Troubleshooting replication problems - Logging bugs and participating in the Tungsten Replicator community Replication is a powerful technology that takes knowledge and planning to use effectively. This webinar gives you the background that makes replication easier to set up and allows you to take full advantage of the Tungsten Replicator benefits.
Citation preview
©Continuent 2014
Tungsten University: Set Up and Operate
Tungsten Replication Robert Hodges, CEO
©Continuent 2014
Introducing Continuent
2
• The leading provider of clustering and replication for open source DBMS
• Our Product: Continuent Tungsten
• Clustering - Commercial-grade HA, performance scaling and data management for MySQL
• Replication - Flexible, high-performance data movement
©Continuent 2014
Introducing Tungsten Replicator
3
©Continuent 2014
Tungsten Replicator Overview
4
Master
(Transactions + Metadata)
Slave
THL
DBMS Logs
Replicator
(Transactions + Metadata)
THLReplicator
Download transactions via network
Apply using JDBC
©Continuent 2014
Master Replication Service
5
Extract Filter Apply
StageExtract Filter Apply
Stage
Pipeline
MySQL Master
Transaction History Log
In-Memory Queue
Slave ReplicatorsBinlog
tcp/ip
©Continuent 2014
Slave Replication Service
6
Extract Filter Apply
StageExtract Filter Apply
StageExtract Filter Apply
Stage
Pipeline
Transaction History Log
In-Memory Queue
Slave DBMS
Master Replicator
tcp/
ip
©Continuent 2014
EC2 Installation Using Github project
1.Set up Github vagrant project
git clone https://github.com/continuent/continuent-vagrant
2.Pick EC2 Vagrantfile and fill in keys
3.Copy examples/MasterSlave/default.pp to manifests/
4.Run launch.sh
7
(Follow directions in README.md)
©Continuent 2014
Tpm installation overview
8
/opt/continuent/etc/tungsten/tungsten.ini
check prereqs [copy code]
configure topology
[defaults]!user=tungsten!home-directory=/opt/continuent!datasource-user=tungsten!datasource-password=secret!skip-validation-check=MySQLPermissionsCheck!start-and-report=true!![university]!master=db1!slaves=db2,db3!topology=master-slave!
tpm update
(Installation)
©Continuent 2014
What Is Tungsten Doing?
9
©Continuent 2014
Commands to Check Replication
• trepctl services -- All services
• trepctl status -- Overall pipeline status
• trepctl status -name stages -- Stage summary
• trepctl status -name stores -- Store summary
• trepctl status -name tasks -- Task detail
• trepctl [-service name] heartbeat -- Is it Alive?
10
©Continuent 2014
Changing Replicator Configuration
11
©Continuent 2014
Reconfiguring a Replicator
• Edit /etc/tungsten/tungsten.ini
• No restart on local host
trepctl offline!
tpm update --no-restart!
trepctl configure!
trepctl online!
!
• With automatic restart
tpm update
12
Don’t change replicator configuration files directly!
©Continuent 2014
Enabling Parallel Replication
13
©Continuent 2014
Parallel Replication of Sharded Data
14
Extract Filter Apply
StageExtract Filter Apply
StageStage
Pipeline
Transaction History Log
Parallel Queue
Slave DBMS
Extract Filter ApplyExtract Filter ApplyExtract Filter Apply
Master Replicator
tcp/
ip
©Continuent 2014
Fixing Problems
15
©Continuent 2014
The Scene of the Crime
16
trepctl -host db2 status Processing status command... NAME VALUE ---- ----- ... pendingError : Event application failed: seqno=1957469 fragno=0 message=java.sql.SQLException: Statement failed on slave but succeeded on master pendingErrorCode : NONE pendingErrorEventId : mysql-bin.000008:0000000461736666;0 pendingErrorSeqno : 1957469 pendingExceptionMessage: java.sql.SQLException: Statement failed on slave but succeeded on master create table trouble(id int primary key) /* ___SERVICE___ = [university] */ ... Finished status command...
©Continuent 2014
Correcting problems
1.Look at the transaction (thl list -seqno NNN)
2.Look at the replicator message log
3.Fix the problem and go online
OR
4.Skip over the problem and go online
OR
5.Reprovision the slave
17
©Continuent 2014
Tungsten Logs
18
©Continuent 2014
Simple Facts about Logs
• Replicator log = THL (Transaction History Log)
• One THL per replication service
• 2x the size of MySQL binlog
• Default retention of 7 days
• Take service offline to move or delete logs safely
• The thl tool shows you log contents
19
©Continuent 2014
Resetting Replication
20
©Continuent 2014
Resetting/Reinstalling Automatically
Reset
1.tpm reset
Reinstall
1.tpm uninstall --i-am-sure
2.Rerun installation
21
©Continuent 2014
Resetting/Reinstalling Manually
Reset
1.Stop replicator
2.Remove THL files
3.Remove tungsten_<svc> databases
Reinstall
1.All of the above, plus delete /opt/continuent/*
2.Rerun installation
22
©Continuent 2014
Backup and Restore
23
©Continuent 2014
What Backup Tools Does MySQL Have?
24
MySQL Server
Process
File System
DBMS Files
mysqldump (dataset < 10Gb)
xtrabackup (10Gb < dataset < 1Tbtar/rsync (dataset < 1Tb)
file system snapshot (1Tb << dataset)
©Continuent 2014
Advanced Installation Patterns
25
©Continuent 2014
How Do I Install Fan-In Replication?
26
db1Replicator
db3
Service db1
db2Replicator
Service db2
Replicator
Service db1
Service db2
©Continuent 2014
How Do I install Multi-Master?
27
db1 Replicator
Service db1
Service db2
Service db3db3
Service db1
Service db2
Service db3db2
Replicator
Service db1
Service db2
Service db3
©Continuent 2014
Tpm Understands Topologies!
• Use --topology argument!
28
Fan-In StarAll-Masters
--topology=fan-in --topology=all-masters --topology=star
©Continuent 2014
Tpm install from staging directory
29
db1
db2
db3
Staging copy of files
check prereqs copy code
configure topology
tpm configure ... tpm update
©Continuent 2014
Commands to Perform Staging Install
30
$ wget http://downloads.tungsten-replicator.org/download.php?file=tungsten-replicator-2.2.1-403.tar.gz !$ tar -xvzf tungsten-replicator-2.2.1-403.tar.gz !$ ln -s tungsten-replicator-2.2.1-403 tungsten !$ tungsten/tools/tpm configure defaults --reset \ --user=tungsten --home-directory=/opt/continuent \ --datasource-user=tungsten --datasource-password=secret \ --skip-validation-check=MySQLPermissionsCheck \ --start-and-report=true !$ tungsten/tools/tpm configure fanin \ --members=db1,db2,db3 \ --masters=db1,db2 \ --master-services=db1,db2 \ --topology=fan-in !$ tungsten/tools/tpm update
©Continuent 2014
Wrapping Up
31
©Continuent 2014
Tungsten Replicator is 100% Open Source
32
• Project home:
http://code.google.com/p/tungsten-replicator/
• Log bugs, find builds, post in discussion group
• Documentation:
https://docs.continuent.com
• Install automation:
https://github.com/continuent
!
©Continuent 2014
HeterogeneousOracleMySQL Oracle MySQL MySQL
star
master-slave
fan-in slave all-masters
MySQLOracle Oracle
©Continuent 2014
We Do Clustering Too
34
Tungsten clusters combine off-the-shelf open source DBMS servers into data services with: !
• 24x7 data access • Scaling of load on replicas • Simple management commands !...without app changes or data migration
Amazon US West
apache /php
GonzoPortal.com
Connector Connector
©Continuent 2014
Conclusion
35
• Tungsten Replicator is an advanced, open source replicator
• Use tpm to install either from staging or using batch commands
• Use trepctl to manage replication
• Tungsten Replicator has capabilities far beyond what we showed today
Use your imagination!
©Continuent 2014
Continuent Web Page: http://www.continuent.com
!
Tungsten Replicator: http://code.google.com/p/tungsten-replicator
Our Blogs: http://scale-out-blog.blogspot.com http://datacharmer.org/blog http://www.continuent.com/news/blogs http://flyingclusters.blogspot.com/
560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: [email protected]
