MySQL Replication vs DRBD Battles

1Like 3 TweetTweet 0StumbleUpon

February 26, 2012

FacebookTwitterLinkedInKindle Edition

MySQL Performance Blog

Percona's MySQL & InnoDB performance and scalability blog

MySQL ConsultingMySQL SupportMySQL TrainingConferencesPercona ServerPercona XtraBackupCustom DevPostsComments

BenchmarksInsight for DBAsInsight for DevelopersHardware and StorageCloud and NoSQLPercona SoftwareEvents and Announcements

MySQL Replication vs DRBD BattlesApril 28, 2008 By Peter Zaitsev 12 Comments

Well these days we see a lot of post for and against (more, more) using of MySQL and DRBD as ahigh availability practice.

I personally think DRBD has its place but there are far more cases when other techniques wouldwork much better for variety of reasons.First let me start with Florian’s comments on the issue as I think they are most interested ones.

First lets get to the point what we’re comparing here – it is mainly DRBD to MySQL Replicationbased techniques (lets leave MySQL Cluster and Continnuent alone for a while as these are a bitfrom different league). It is not the question if DRBD is better than SAN – it offers moreindependence compared to SAN and in my view surely superior from HA point of view but this isnot the point of comparison.

ShareShare

MySQL Replication vs DRBD Battles - MySQL Performance Blog http://www.mysqlperformanceblog.com/2008/04/28/mysql-replic...

1 of 10 2/26/12 8:38 PM

â€œFailback could destroy the original master tooâ€ , however, is plain false. DRBDwonâ€™t â€œdestroy the original masterâ€ any more than it already was if thefilesystem on top of DRBD was fried beforehand.

Let us again compare MySQL Replication to DRBD in this case – in both cases due to some reasonsyou can have systems to run out of sync and have conflicting updates applied to them. With DRBDyou have the choice of killing one of the nodes and re-syncing from another one while with MySQLReplication you can use Maatkit to merge the changes after all, also you can review binary logs tosee which updates were applied to different nodes.

Transaction log replay, yes. But fsck? These days this amounts to running a journalreplay. Takes under a second in most circumstance

I would put it 10 seconds but it does not matter. The transactional log reply is likely to take muchlonger than that. This is very bad property of DRBD – besides the well understood overhead ofcommitting on both nodes instead of one you also meat the tough choice – you’ve got to pick eitherlong recovery time or further degraded performance. In the large databases I run in productionrelaying on MySQL Replication for HA I often have 15+ minutes reply of Innodb Transactional logwhich would be a huge bummer with DRBD.

I would also say this implies hidden danger – the time it will take your database to do transactionallog recovery is invisible until you get a crash, meaning if your production database size growths loadchanges or you happen to have failure during activity of certain kind you might have it taking muchlonger when expected. recovery time depends on a lot of variables.

The side question about it is of course the fact you have to be picky about storage engines you’reusing – DRBD does not work with MyISAM (check required) so you need to have processes toensure your application does not uses this storage engine which may be hard to guaranty in manyenvironments when development has too much autonomy.

I must note in this aspect however DRBD goes on par with MySQL Statement Based Replication – itis also well too easy to use MySQL features which break replication.

The failover node is a hot standby, itâ€™s just not a running slave node from thedatabaseâ€™s standpoint. And, nothing stops you from running two databases on twoservers on two DRBD devices laid out in a â€œcriss-crossâ€ fashion, converging onone node in case of node failure.

This actually goes to two topics. First – hot vs cold. If you’re using decent hardware and care aboutperformance you use O_DIRECT with Innodb which makes it to bypass file cache. If you have itDRBD slave will be fully cold. But lets assume you’re ready to pay for yet another penalty DRBDintroduce and do not use this option wast memory and CPU cycles on double caching. Even in thiscase DRBD slave node can’t be called hot because write load often does not touches the same data asread load. Here is simple example – assume you’re inserting the data at the same time as runningreporting queries on last month. All last month will be hot on the slave which is doing reads but onlylast few hours will be hot on the standby box.

Running two instances on the server allows to reduce hardware waste with DRBD, though noteliminate because you get some disks which you can’t really use for anything else rather than HA.Two instances also complicate things – depending on infrastructure it can be seen as almost nocomplication or quite serious complication.

“Cannot do maintenance on cold standby database.”


2 of 10 2/26/12 8:38 PM

1Like

But you can do anything you want with a database that you run off a DRBD LVMsnapshot. Works on a Secondary node too.

I’m not sure Florian understood what was meant here. With MySQL Master-Master replication I canadd and index on passive node, wait it to catch up and switch the roles (see another post) you can’treally do this with DRBD as this requires logical level of operation to work.

This is not to mention other things you can do with MySQL Replication, such as filtered replicationor cross storage engine replication, though which are not typically used for HA purposes. TimeDelayed replication is however something quite helpful for some environments. Though DRBD alsocould be extended to support one if needed.

Now, do not get me wrong DRBD is great, and thanks to Florian for following up and making suremyths about DRBD do not spread too wide.

So when I would recommend to use DRBD with MySQL ?

There some good reasons to use DRBD with MySQL though as I mentioned I do not view it as firstchoice solution.

First it is good choice for organizations which are got use to SAN based high availability solutionswith active-passive management software. Quite often these guys would have be familiar with suchHA concept and it would be very natural for them to use same approach for MySQL as they use forPostgreSQL for example instead of investing time to learn about MySQL Replication or just lookingto keep MySQL infrastructure to be as close to one for other databases in use as possible.

Second – it is often inevitable choice when you can’t avoid loosing any transaction – period. Somepeople would rather stand longer fallback time (as with DRBD) but would not like to have losttransactions which may happen with async replication. Another similar case is when you’re lookingto ensured consistency – MySQL Replication can out of sync – and there is bunch of tips indocumentation of how to do it. With DRBD the chance of nodes running out of sync is minimal andcan be caused by software and hardware bugs rather than known limitations.

You could argue depending on cases you spend most time working with how much cases do thesecorrespond – some people mainly deal with systems which can’t accept any transactions loss and forthese DRBD often comes as a first choice if you have more experience with traditional web shops –these usually would prefer to lose one user comment a year instead of paying extra performancecosts.

It is worth to note DRBD also allows building very nice mixed environments with MySQL – forexample you can replicate binary logs using DRBD so if master node fails you have not losttransactions – when you can use such logs to do point in time recovery or to catch up to last few notcommitted transactions on the slave. We should spend some time implementing such script sometimewhich could be helping to get best of both worlds.

But currently – You can’t have it all

The state of High Availability Solutions for MySQL these days is – you can’t have it all. There is noOpenSource solution out where which would offer you full redundancy use of both nodes at least forreads no transaction loss and automated fail over. Whenever you’re using MySQL Cluster, DRBD orMySQL Replication you have to have some compromises.

Filed Under: Insight for DBAs


3 of 10 2/26/12 8:38 PM

3 TweetTweet 0StumbleUpon

Tagged With:Production

About Peter Zaitsev

Peter managed the High Performance Group within MySQL until 2006, when he founded Percona.Peter has a Master's Degree in Computer Science and is an expert in database kernels, computerhardware, and application scaling.

Comments

Wei says:April 28, 2008 at 9:59 pm

Should’nt Google’s MySQL solution an HA option:. Semi-sync replication to wait for replica’s responses. Mirror binlog on the replicas. Make replicas serve the mirrored binlogs during the failover

The code is there. But, it is not smoothly integrated yet.

1.

peter says:April 28, 2008 at 11:30 pm

Wei,

Google Patches is Semi-Synchronous solution though which is good enough for most cases.Though indeed this will be great solution once it is integrated

2.

Wei says:April 29, 2008 at 12:22 am

The patch also includes mirroring binlogs on the replicas and serve the mirrored binlogs fromthe replicas. The patch should also make the hierarchal replication available. I wrote it, butcould not find an easy way to integrate it smoothly.

3.

Paul McCullagh says:April 29, 2008 at 4:35 am

Hi Peter,

As I discussed at the conference (http://www.primebase.org/download/pbxt-uc-2008.pdf), I amplanning to add HA capabilities to PBXT using synchronous replication at the engine level.

4.

ShareShare


4 of 10 2/26/12 8:38 PM

What do you think of this approach?

Best regards,

Paul

Peter Romianowski says:April 29, 2008 at 5:31 am

Great post summing up all pros and cons.

I would add another point: Backup from slave / secondary. With DRBD this is hard /impossible in many situations.

We use both, replication and DRBD in production. One database which is doing nothing but“spooling” working sets runs on DRBD (no slave lag, 100% sure all transactions are there, noneed to backup this database). The main database is configured in Master-Master replicationfashion. We evaluated using DRBD here too but did not do it because we’d loose the ability todo “online schema change” or LVM snapshot backup. And finally we were not able to get adescent InnoDB recovery time (26G buffer pool). So it would not even be a cold stand-by buta frozen one.

So all the pros and cons stroke us.

5.

Gregory Haase says:April 29, 2008 at 6:52 am

I’m not sure what Peter R. means about not being able to do an lvm snapshot of the DRBDsecondary node. Surely you can. If DRBD sits on top of LVM, then you snapshot the LVM andyou get a snapshot of the blocks on the drbd device. Whether DRBD is in primary orsecondary at this point is irrelevant.

Another thing I’d like to add is that there are some database/application designs that just don’tallow for friendly multi-master setups. More real-time data warehouse applications that logtons of data and frequently summarize and aggregate will run into problems. Do you runaggregation scripts only on one server and propagate them? Do you run aggregates on bothservers and “assume” that all the numbers add up in the end – this can be very dangerous.

Also, merely having a slave sitting ready to take master can be troublesome in some situationsbecause the slave is still single-threaded. You end up getting a beefier slave than yourproduction system just so that a single thread on the slave can keep up with all yourtransactional threads on your master. Now you have to ask yourself – maybe I can use theslave for other things, so it doesn’t bother me too much that my slave is actually morepowerful than my master. Or maybe it’s better to have two equally setup systems and useDRBD instead.

I think given the argument that we all agree on one thing – it really depends on yourapplication/database environment, your requirements for recovery speed, data loss, etc.

Finally, I will leave with this note – we actually do both – drbd + ha cluster for

6.


5 of 10 2/26/12 8:38 PM

high-availability and a big beefy slave for redundancy and read only queries.

Michael Monashev says:April 29, 2008 at 10:49 am

Ð¯ Ð½ÐµÐ´Ð°Ð²Ð½Ð¾ Ð¿Ð¸Ñ Ð°Ð» Ð¾Ð± Ð°Ð»ÑŒÑ‚ÐµÑ€Ð½Ð°Ñ‚Ð¸Ð²Ðµ DRBDÐ½Ð° FreeBSD: http://michael.mindmix.ru/366-392-ggate-kak-al-ternativa-memcached-i-memory-appliance.zhtml

Ð¡Ð°Ð¼ Ð½Ðµ Ð¿Ñ€Ð¾Ð±Ð¾Ð²Ð°Ð» Ñ‚Ð°ÐºÑƒÑŽ Ñ Ñ…ÐµÐ¼ÑƒÑ€ÐµÐ°Ð»Ð¸Ð·Ð¾Ð²Ñ‹Ð²Ð°Ñ‚ÑŒ, Ð½Ð¾ Ð²Ð¾Ð·Ð¼Ð¾Ð¶Ð½Ð¾ Ð¾Ð½Ð°Ð¾ÐºÐ°Ð¶ÐµÑ‚Ñ Ñ Ð±Ñ‹Ñ Ñ‚Ñ€ÐµÐµ Ð¿Ð¾ Ñ‡Ñ‚ÐµÐ½Ð¸ÑŽ.ÐžÑ Ð¾Ð±ÐµÐ½Ð½Ð¾ Ð¿Ð¾ Ñ Ð»ÑƒÑ‡Ð°Ð¹Ð½Ð¾Ð¼Ñƒ.

7.

Mark Callaghan says:May 1, 2008 at 8:12 am

I think DRBD is amazing. The concern about long crash recovery times after failover withInnoDB and DRBD can be resolved. One method to do this is a config option that enforces themaximum number of crash-recovery IO (dirty pages in buffer cache + pending IO from insertbuffer + pending IO for purge thread). When the max is exceeded the server can favorbackground IO over transactions to let the server catch up. Another option is to let Maatkitwatch the server and run commands (might need more FLUSH commands) to force morebackground IO when there is too much pending IO in a server.

Is there any parallelism or concurrency (multiple threads or multiple IOs pending) in InnoDBcrash recovery code?

The Google patch had more than semi-sync for an HA solution with MySQL. It also includes afeature to keep the InnoDB transaction log and slave replication state in sync after a crash(transactional replication). But the most interesting feature was mirror binlog. This maintainedmirrors of the master’s binlog on a slave (same filename as used on the master, byte for byteequivalent). This was done using the same replication protocol — the IO thread would copyevents into the relay log and the mirrored binlog. It allows hierarchical replication andtransparent slave failover.

We are trying to redo support for hierarchical replication and transparent slave failover withglobal transaction IDs.

8.

peter says:May 1, 2008 at 3:34 pm

Mark,

I do not think it is as easy as you say. you can limit number of io for REDO but there is alsoUNDO part which depends only on size of transaction. It is indeed background in 5.0 but thedata remains locked until rollback is complete which can affect operation.

9.


6 of 10 2/26/12 8:38 PM

Plus you still have warmup problem.

Speaking about Google batch – I think there is a lot of cool stuff out where besides HA, andthanks for explanations.

Mark Callaghan says:May 1, 2008 at 4:23 pm

Don’t allow long running transactions and undo is not a problem. Of course, that is easier saidthan done. But you don’t want long running transactions. They are an instant source ofreplication delay and force applications to go back to the primary to get current data or stopusing the slave.

10.

peter says:May 1, 2008 at 4:44 pm

Right. You do not want long running transactions in the perfect world, if you can avoid them itis not a big issue.By the way it would be cool if Innodb could allow to restrict number of changes transactioncould do – enabling such stuff would allow to catch users running bad transactions easily.

Again my point is not to say generally it is good or bad but highlight the problems you can runinto

11.

joelb says:June 30, 2008 at 12:26 pm

hello, we are debating DRBD vs MySqL replication. The idea would be two servers, onemaster, one slave running drbd. Second server would never be live unless the first servercrashed, or needed maintainance. There would be a third server not at the colo that would usemysql replication to keep as closely up to date as possible with the master server at the colo,and be used as a failover in the event of an outage.

I’m confused by what I’ve read: After a crash, the slave is now live. How long does it take toresync the master with the slave? Can the slave simply become the master and the master nowbe considered the hot spare slave that will take over if the new “master” machine ever crashes?

Thanks,

Any comments (& the sooner the better) would be helpful, also any direction to a communityof folks who are deeply involved with these technologies would be helpful as well.

– Joel

12.

Speak Your Mind

Name *


7 of 10 2/26/12 8:38 PM

Email *

Website

Post Comment

Notify me of followup comments via e-mail

Search

SearchAdvanced

Related Searches

mysql replication with drbdmysql cluster vs drbdmysql replication vs clusterdrbd mysql replicationmysql cluster vs replicationmysql community vs enterprise versionmysql community vs enterprisemysql btree vs rtreemysql 4 vs 5difference of mysql enterprise vs mysql community

ABOUT PERCONA

Percona is the oldest and largest independent MySQL support and consulting company, with a global24×7 staff of over 50 people serving more than 1000 customers in 50+ countries since 2006.

We offer consulting, training, support, and custom development services.


8 of 10 2/26/12 8:38 PM

We also create Percona Server, XtraDB and XtraBackup.

Percona Webinars

April 4, 2012 - A New High Availability Solution for MySQLMarch 14, 2012 - Optimizing MySQL Configuration

All Webinars »

Careers at Percona

Sales and Marketing Operations EngineerTrainer (Worldwide)Software EngineerRegional Support Sales RepresentativeMySQL Consultant (American timezones)

All Careers »

MySQL Training from Percona

San Francisco, USA on Monday, 12 March 2012

All Courses »

Related Posts

MySQL Replication and Slow Queries1.Making MySQL Replication Parallel2.Figuring out what limits MySQL Replication3.Managing Slave Lag with MySQL Replication4.State of MySQL Market and will Replication live ?5.No more MySQL Crash Safe Replication in 5.0 ?6.Jeremy Cole on MySQL Replication7.Thoughts on MySQL Replication8.Filtered MySQL Replication9.Using LVM for MySQL Backup and Replication Setup10.

SOFTWARE FROM PERCONA

Percona's open-source software leads the industry in performance and features.

Percona Server: an enhanced version of MySQLPercona XtraDB: our fork of InnoDBPercona XtraBackup: hot InnoDB Backups


9 of 10 2/26/12 8:38 PM

About This BlogContent LicensePresentations on MySQLMySQL PatchesTools for MySQL UsersConsulting for MySQL

Careers at PerconaPercona.TV VideosPerformance ForumsWeb Scaling BlogSSD Performance Blog

SERVICES FOR MYSQL USERS

Percona offers professional services for MySQL users

MySQL ConsultingMySQL SupportMySQL TrainingCustom MySQL Development

ABOUT THIS BLOG AND PERCONA

Copyright © Percona Inc. • Contact Us• MySQL and InnoDB are trademarksof Oracle Corp.


10 of 10 2/26/12 8:38 PM

Documents

MySQL Replication vs DRBD Battles