High Availability & Replica Setswith

Gareth Davies

@ShaolinTiger

www.shaolintiger.com

Who am I?

- Blogger (shaolintiger.com/darknet.org.uk)

- Community Starter (security-forums.com/shutterasia.com)

- Geek/Sys-admin

- WordPress/Web Publishing Scaling Expert

- Recent MongoDB user

- Currently working at Mindvalley

Why I <3 MongoDB

- It's FAST- It's relatively easy to setup- It's a LOT easier to scale than say..MySQL

- Does anyone know about scaling MySQL?

Scaling MySQL Is Like...

Basic Concepts – Master Slave

The Next Level – Replica Set

Master Slave vs Replica Set

Replica Sets – Things to Grok

- The primary AKA Master is auto-elected

- Drivers and mongos can detect the primary

- Replica sets provide you:

- Data Redundancy

- Automated Failover AKA High-availaiblity

- Distributed Read Load/Read Scaling

- Disaster Recovery

Replica Sets - Caveats

- You must have an odd number of mongos at all times to avoid vote locking & primary becoming read only

- You can have a maximum of 12 nodes in a set, if you need more read capacity – look to sharding

- If you are reading from the secondary servers you need to understand Eventual Consistency

Getting Started

- I <3 Linode!

- Easy scaling/Nodebalancers/Cloning/Fast Roll-up/Fastest IO in the industry/Ubuntu 12.04LTS etc

– Examples are done on Linode

Add 3 new Nodes

- Chose your location, put 2 in your primary DC and 1 in a different geographical DC

- In my case this would be 2 servers in Atlanta, GA and 1 in Dallas TX

– This gives you a replica set that works if a whole datacenter goes down

Select The OS

- Ubuntu 12.04LTS 64-bit – this gives you package support for the next 5 years

- Also gives you the ability to grow your MongoDB instance above 2GB safely

Do your basic shizzles

- Set the hostname/local IP address etc

- Disable Swap

- Change SSH port

- Remove password based login

- Block root SSH acess

- aptitude update; aptitude safe-upgrade

- Install base packages (munin/iotop/sysstat etc)

- Configure unattended security updates

Install MongoDB

- I don't recommend installing from the regular repo as it will be out of date after some time

- Install direct from the 10gen repo

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10

sudo nano /etc/apt/sources.list – add this:

deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen

aptitude update; aptitude install mongodb-10gen

- That's it – it's installed!

Clone that bad boy!

- Bear in mind you only have to do all of that stuff once! When it's done – just clone it over to your two new nodes.

- Remember to delete the config & disk images from your target first & power down the initial machine.

* Do note when copying to the remote DC it will take quite a long time

Get the Replica Set StartedDo:

sudo nano /etc/mongodb.conf

Find the line like so:

# in replica set configuration, specify the name of the replica set

# replSet = setname

Change it to:

# in replica set configuration, specify the name of the replica set

replSet = yoursetname

Do this on all your Mongos and then restart them

sudo service mongodb restart

Configure the Replica Set

- After restarting if you check the logs you'll see something like this:

- This basically means the Replica set is running, but it's not yet aware of the other nodes

Add the member nodes

- You'll need to run 'mongo' to get the mongo shell then:

rs.initiate({_id: 'yoursetname', members: [

{_id: 0, host: '192.168.1.1:27017'},

{_id: 1, host: '192.168.1.2:27017'},

{_id: 2, host: '192.168.1.3:27017'}]

})

- This will spin up the set

Check that it worked

- I suggest running tail -f on the logs on one of the other nodes, you'll see a bunch of messages about replSet & rsStart (hopefully)

- If you see all that in /var/log/mongodb/mongodb.log – you're good!

That's It!

- Yah I know, too easy right?

- That's how hard it is to set up a fully scalable, high availability database cluster with MongoDB

Things to Consider

- ALWAYS monitor, make decisions made on statistics and numbers not on assumptions

- I like (and very actively use) munin

- munin works well with MongoDB and a myriad of other software

Further Learnings

- Think about security (Bind address/IPTables/Authentication/Cluster Keys)

- If you have a write heavy application and you need to scale writes – look to sharding

- Sharding and replica sets work well together (but each shard needs a replica set)

- Try and give your MongoDB instance enough RAM to keep the hot index in memory

THE END!

Questions?

This presentation will be available at http://slideshare.net/shaolintiger