MongoDB in Production - Amazon Web...

Chris Harris - charris@10gen.com

MongoDB in Production

HIGH AVAILABILITY SCENARIOS

Single Node

•Will have downtime•If node crashes human intervention might be needed

How MongoDB Replication works

Member 1

Member 2

Member 3

•Set is made up of 2 or more nodes

How MongoDB Replication works

Member 1

Member 2

PRIMARY

Member 3

•Election establishes the PRIMARY•Data replication from PRIMARY to SECONDARY

How MongoDB Replication works

Member 1

Member 2

DOWN

Member 3

negotiate new master

•PRIMARY may fail•Automatic election of new PRIMARY if majority exists

How MongoDB Replication works

Member 1

Member 2

DOWN

Member 3

PRIMARY

•New PRIMARY elected•Replication Set re-established

How MongoDB Replication works

Member 1

Member 2RECOVERING

Member 3

PRIMARY

•Automatic recovery

How MongoDB Replication works

Member 1

Member 2

Member 3

PRIMARY

•Replication Set re-established

DEMO

Replica Set 0•Two Node•Network failure can cause the nodes to slip which will result in the the whole system going READ_ONLY

NOT RECOMMENDED!

Replica Set 1

Arbiter

•Single datacenter•Single switch & power•Points of failure:•Power•Network•Datacenter•Two node failure

•Automatic recovery of single node crash

Replica Set 2

Arbiter

•Single datacenter•Multiple power/network zones•Points of failure:•Datacenter•Two node failure

•Automatic recovery of single node crash•w=2 not viable as losing 1 node means no writes

Replica Set 3•Single datacenter•Multiple power/network zones•Points of failure:•Datacenter•Two node failure

•Automatic recovery of single node crash•w=2 viable as 2/3 online

Replica Set 4

•Multi datacenter

•DR node for safety

•Can’t do multi data center durable write safely since only 1 node in distant DC

Replica Set 5

delayed •Three data centers•Can survive full data center loss •Can do w= { dc : 2 } to guarantee write in 2 data centers

HARDWARE SIZING

Collection 1

Index 1

Virtual Address Space 1

Collection 1

Index 1

Virtual Address Space 1

Collection 1

Index 1 This is your virtual memory size (mapped)

Virtual Address Space 1

Physical RAM

Collection 1

Index 1

Virtual Address Space 1

Physical RAM

Collection 1

Index 1

This is your resident memory size

Virtual Address Space 1

Physical RAM

DiskCollection 1

Index 1

Virtual Address Space 1

Physical RAM

DiskCollection 1

Index 1

100 ns

10,000 ns

=

=

Sizing RAM and Disk• Working set• Document Size• Memory versus disk• Data lifecycle patterns

• Long tail• pure random• bulk removes

Figuring out working Set> db.wombats.stats() { "ns" : "test.wombats", "count" : 1338330, "size" : 46915928, "avgObjSize" : 35.05557523181876, "storageSize" : 86092032, "numExtents" : 12, "nindexes" : 2, "lastExtentSize" : 20872960, "paddingFactor" : 1, "flags" : 0, "totalIndexSize" : 99860480, "indexSizes" : { "_id_" : 55877632, "name_1" : 43982848 },

Size of data

Size on disk (and in memory!)

Size of all indexes

Average document size

Size of each index

Disk configurations

~200 seeks / second

Single Disk

Disk configurations

~200 seeks / second

~200 seeks / second ~200 seeks / second ~200 seeks / second

Single Disk

RAID 0

Disk configurations

~200 seeks / second

~200 seeks / second ~200 seeks / second ~200 seeks / second

~400 seeks / second ~400 seeks / second ~400 seeks / second

Single Disk

RAID 0

RAID 10

SSD?• Seek time of 0.1ms vs 5ms

(200 seeks / sec => 10000 seeks / sec)

• But expensive

Takeaway• Know how important page faults are

–If you want low latency, avoid page faults• Size memory appropriately

–To avoid page faults, fit everything in RAM–Collection Data + Index Data

• Provision disk appropriately–RAID10 is recommended–SSD’s are fast, if you can afford them

MONITORING

Monitoring Toolsmongostat -

MMS! - http://mms.10gen.com

munin, cacti, nagios -http://www.mongodb.org/display/DOCS/Monitoring+and+Diagnostics

Monitoring is your friend!

SUMMARY

Typical DeploymentsUse? Set

sizeData Protection

High Availability Notes

X One No No Must use --journal to protect against crashes

Two Yes No On loss of one member, surviving member is read only

Three Yes Yes - 1 failureOn loss of one member, surviving two members can elect a new primary

X Four Yes Yes - 1 failure** On loss of two members, surviving two members are read only

Five Yes Yes - 2 failuresOn loss of two members, surviving three members can elect a new primary

Summary

• Reads from Primary are always consistent

• Reads from Secondaries are eventually consistent

• Automatic failover if a Primary fails

• Understand your Working Set

• Monitoring is your Friend!