Failover, or not Failover,that is the questionPercona Live MySQL Conference and Expo 2013Massimo Brignoli, SkySQL Henrik Ingo, Nokia
Please share and reuse this presentation licensed under the Creative Commonse Attribution License
Agenda
● Why HA is more difficult for databases● Steps to failover● Monitoring● Automating failover● Sounds great!
What could possibly go wrong?● Amazon Dynamo● Galera and NDB
Fault tolerance = redundancy
● RAID● 2 power units per server● Cluster of servers● 2 kidneys per person● Redudancy at all levels:
Software, Hardware, Network, Electricity...
A chain is as strong as the weakest link.
Durability
"Durability is an interesting concept.If I flush a transaction to disk,
it is said to be durable.But if I then take a backup, it is even more durable."
Heikki Tuuri
Why High Availability is More Difficult for Databases
Redundancy of serverAND
Redundancy of data
WHILEPerforming thousands of write operations
per second onto the dataset
What failover?
1. Primary server2. Secondary / Standby server
for redundancy3. In case Primary fails,
Secondary server must become the new Primary
Steps to failover (theory)
1. Notice failure2. Move VIP3. Continue
Automating failoverGeneric Clustering Solutions
● Pacemaker/Corosync
● Linux Heartbeat
● Red Hat Cluster Suite
● Solaris Cluster
● Windows Server Failover Clustering
● etc...
MySQL Specific Solutions
● MMM
● PRM
● MHA
● JDBC connector
Steps to failover (DRBD)
VIPVIP
1. Have DRBD2. Notice failure3. Shutdown MySQL on primary4. Unmount disk on primary5. Mount disk on secondary6. Start MySQL on secondary7. Wait for InnoDB recovery8. Wait for InnoDB recovery9. Wait for InnoDB recovery
10. Unset VIP on primary11. Set VIP on secondary12. Continue13. Should you add a new secondary?
Steps to failover (MySQL replication)
VIPVIP
1. Have replication2. Notice failure3. Make slave writable4. Make master read-only5. Unset VIP on master6. Set VIP on slave7. Continue8. Should you add a new slave?
What if you have more than 2 servers? (MySQL replication)
VIPVIP
?
● MySQL replication failover with more than 2
servers can be a hassle.
● Which slave should become the new master?
● All slaves must be pointed to the new master.
● They must figure out where to continue
replication (binlog position)
● MySQL 5.6 GTID helps.
MHA and SkySQL...
● Combination of resource manager + scripts
● Automating failover process:○ New Master
selection○ Slaves
reconfiguration○ VIP management○ Missing binlogs
retrieval
Sounds great, what could possibly go wrong?
Sounds great, what could possibly go wrong?
VIP
1. Have replication○ Ok, is it working? What if it's not working?○ Is it replicating in the right direction?○ Does your bash script handle binlog positions correctly?○ Asynchronous?
2. Notice failure○ Polling interval○ Who is polling?○ ...and from where?○ How is he handling failure himself?○ False positives○ Is failover the right response to every failure?
3. STONITH○ Shutdown MySQL on Primary? How? It's not responding...○ Unmount disk on Primary? How? It's not responding...○ "You need a STONITH device"! Hehe, nice try...
4. Move VIP○ Unset VIP on Master/Primary? How? It's not responding...○ Set VIP on Secondary/Slave. This will work fine. Unfortunately.
5. Continue6. Add back new/same Secondary
○ Automatically of course. Even if it just failed 15 seconds ago.
VIP
Case Githubhttps://github.com/blog/1261-github-availability-this-week● MySQL replication, Pacemaker, Corosync, Percona Replication Manager● PRM health check fails due to high load during schema migration.● Failover!● New node has cold caches, so even worse performance.● Failover! (back)● Disable PRM● A slave is found outdated as replication is not happening● Enable PRM and hope it will fix it● Pacemaker segfaults, causing cluster partition● PRM selects the outdated node as master, shuts down others● All kinds of data inconsistencies● Restart PRM on all nodes● ...
Case GithubLesson learned:
Automated failover is dangerous
Cold cache is dangerous
But... Not automating is also dangerous
Baron Schwartz: 75% of replication failures are human errorshttp://www.percona.com/about-us/mysql-white-paper/causes-of-downtime-in-production-mysql-servers
80% of Aviation accidents are caused by human errorshttp://asasi.org/papers/2004/Shappell%20et%20al_HFACS_ISASI04.pdf
80% Events caused by human errors, 70% of them due to organization weaknesseshttp://www.hss.doe.gov/sesa/corporatesafety/hpc/fundamentals.html
Are we solving the right problem?
Instead of automating the problem...Eliminate the problem!
Amazon Dynamo
R + W > N
Voldemort, Cassandra, RIAK, DynamoDB, S3http://openlife.cc/blogs/2012/september/failover-evil
N=3, R=W=2
R + W > N
Eventual consistency is internal only
R + W > N
Failover?
Single node failure is a non-event!
For relational databases?
Synchronous replication isspecial case of Dynamo:
W=N & R=1
Or is there a failover after all?
Due to W=N, writers actually notice node failures! Cluster reconfiguration needed.
(Readers are ok.)
?
Example: Galera
OKTimeout
Example: MySQL NDB Cluster
What have we learned?
● Failover with DRBD is painful because it is slow.
● Failover with MySQL replication is painful because it's a mess.
● Amazon Dynamo has no failover● Galera Cluster has no failover but needs
cluster reconfiguration. Same thing...● MySQL NDB Cluster has failover but you
can't see it.