1. CTO, 10genEliot Horowitz#MongoDBDaysHow to keep your data safein MongoDB
2. What can go wrong? Network breaks in transit Server crashes while processing Server blows up after processing a write beforereplication Server processes, crashes, and then aconflicting write happens elsewhere All copies burn in a fire 20 years later, no one remembers how to read it
3. What is data safety?
4. Version 1Probability that a givenwrite or piece of data isaccessible given humanintervention and infinitetime.
5. Version 2Probability that a givenwrite or piece of data isvisible in a query.
6. Single Server How a WriteWorks Client sends a write operation to server Received by servers tcp stack MongoDB process queues write Write happens in memory Depending on what Write Concern asks for Respond immediately Wait for data to be journaled, then respond
7. Single Server What can gowrong Network can go down once message hits other side Client doesnt know what happens without going back and checking Write could fail for logical reason (unique key exception) Server could crash before journaled Write is lost journaled Server could crash after journaled When server is recovered, write is replayed and safe Hard drive can crash irrecoverably Data center could lose power for large period of time
8. Any single server will failReplica Sets
9. Replica Set - Reminders N nodes Each node has a fully copy of the data Replication is asynchronous
10. Replica Set -Acknowledgements w : how many servers must apply write beforeacknowledged w=2 : do not acknowledge until write is on twoservers If primary fails, election guaranteesnew primary has all writesacknowledge w=2 w=majority : do not acknowledge until writes is on amajority of nodes in a replica set If any primaryis elected automatically,all writes acknowledged withw=majoritywill be on primary.
11. Good, but not enoughWhat if I lose an entiredata center?
12. Replica Set - tags A node can have a set of tags region=us-east color=blue Operator configures write level Critical has to be in 3 regions Important has to be in 2 regions w=critical Do not acknowledge write until its in 3 data centers Losing an entire data center causes no data loss
13. What about sharding? Same rules apply Given a series of writes, they may go to differentshards Aw=majority at the end means all writes on that socketare acknowledge by a majority of the relevant replica set Config servers have no impact on faulttolerance/durability, only on admin uptime (orreal uptime in a disaster)
15. Personal Blog Single server No replication Hourly backups If server crashes Down until back up All acknowledge writes safe If server is destroyed Have to recover from backup Lose up to 1 hour of writes
16. Departmental App Single replica set 3 nodes in a single server If any single node goes down System is still readable/writeable Writes done with w=2 are safe If 2 nodes go down at the same time Only writes with w=3 are safe (bad idea) No primary, last node is read-only
17. Core User Database Single replica set 3 data centers Primary data center: 3 node (p=2) 2 alternates with 2 nodes each (p=1) Different types of operations Password change (w=majority) Adds a like (w=2) Login count (w=1)
18. Core User Database contd Lose any single server Can only lose a login count Lose any 2 servers Could lose a like if you are unlucky Lose a data center Still have a majority All password changes are safe
19. Choice is a doubleedged sword
20. When to give a choice? Give choice over semantics Developers and Operators know their needs Tuning parameters are dangerous System should be smart enough to avoid thousands ofknobs Defaults should be Intuitive and sensible Changing is hard Always changing a little
21. Semantics Knobs toomany?
22. Already have them in differentarchitecture components Caching Worker queues Asynchronous replication Synchronous replication Two-phase commit
23. MongoDB gives you the choice ofdurability semantics from manysystems in one. Control per write One source of truth in architecture
24. What should you do? Pick a default write level for your app Only deviate with good reason Test disaster scenario so you know whatsgoing to happen