Embed Size (px)
Citation preview
MongoDB Athens User GroupAthens, Greece 16/1/2013
Alex [email protected]
Replication and High Availability
(c) Alex Giamas, Persado Inc, All rights reserved
History
● Oracle shop
● Non existent OLAP
● Queries in live DB
(c) Alex Giamas, Persado Inc, All rights reserved
Initial investigation
● CouchDB
● Riak
● Hbase
● Cassandra
● MongoDB
● Voldemort
History
● Missed all the fun while being in the States
● Hadoop / HBase / llama / Pig
● Sharded MySQL
● Voldemort
● Huge RAC deployments
● Following MongoDB since 1.4 (Replica sets? Nah.. Sharding?..alpha)
(c) Alex Giamas, Persado Inc, All rights reserved
Reporting and Analytics
● Settled on MongoDB
● Document oriented
● No clue about schema at the time
● No clue about what are we going to do with client data
(c) Alex Giamas, Persado Inc, All rights reserved
Prototyping, version 0.5
● MT, MO, User collections
● Sync Map Reduce for reporting
● One DB to rule them all
(c) Alex Giamas, Persado Inc, All rights reserved
Show stoppers
● Single server deployment
● Global write lock
● MR in real time
(c) Alex Giamas, Persado Inc, All rights reserved
Results
● Demo Christmas Eve 2010
(c) Alex Giamas, Persado Inc, All rights reserved
Results
● Demo Christmas Eve 2010
● Slow....
(c) Alex Giamas, Persado Inc, All rights reserved
Reporting, version 1.0
● Spring Batch for async computations
● Quartz scheduler firing every 3 minutes
● Separate nodes for OLTP and OLAP Dbs
● Custom cloneCollection()
(c) Alex Giamas, Persado Inc, All rights reserved
Real world kicks in
● Everything designed for online integration
● Huge client coming in offering offline integration
● Ride the cloud wagon!
(c) Alex Giamas, Persado Inc, All rights reserved
Reporting Version 2 (the real world)
● Files coming in via FTP containing all sorts of time inconsistencies
● No longer a linear timeline of events, more like a soup of results
(c) Alex Giamas, Persado Inc, All rights reserved
MongoDB on EC2
● 2 replica sets of 2 nodes+arb
● Arbiters crossed wrt replica sets
● Third node could be different availability zone
(c) Alex Giamas, Persado Inc, All rights reserved
Replica Sets Configuration
(c) Alex Giamas, Persado Inc, All rights reserved
Replica Sets Configuration
“server seen down”
(c) Alex Giamas, Persado Inc, All rights reserved
Replica Sets Configuration
“server seen down”
(c) Alex Giamas, Persado Inc, All rights reserved
Replica Sets Configuration
● Can afford 1 failure with fully functional cluster
● Can afford 2 failures with partially functional cluster**
** Terms and conditions may apply
(c) Alex Giamas, Persado Inc, All rights reserved
Replica Sets Configuration
● Rolling upgrades without DB downtime
● Schemaless, document oriented offers great flexibility in application terms
(c) Alex Giamas, Persado Inc, All rights reserved
Replica Sets Configuration
● Unix level tweaks:
– raise ulimit
– raise tcp timeout
– Noatime nodiratime
– XFS, ext4
– LVM for snapshotting
● Mongo level tips:
– Use journaling. USE JOURNALING
(c) Alex Giamas, Persado Inc, All rights reserved
Replica Sets Configuration
EC2 specific tips:
– Can and will steal back time, plan for it
– Can get flaky at times..
– Design around EBS
(c) Alex Giamas, Persado Inc, All rights reserved
Replica Sets Configuration
● EC2 storage:
– Local storage. Ephemeral
– EBS storage. Lasts but not strong durability guarantees.
– S3 storage. Lasts more, slower
(c) Alex Giamas, Persado Inc, All rights reserved
Replica Sets Configuration
● Settled for EBS storage.
● Nightly backups, 30 day window
(c) Alex Giamas, Persado Inc, All rights reserved
Reporting Version 3
● Aggregation Framework effort led by Chris Westin
– Simpler way to perform Map Reduce jobs without all the pain of JS
– Integrates cleanly with our business logic
● Initial design on sharding
– More on that next..
(c) Alex Giamas, Persado Inc, All rights reserved
Reporting Version 3
● Aggregation framework for both storing and retrieving aggregate data
– New collection for double checking results with MR.
● Faster, simpler, most of the times fits in our problem domain.
● Worked better in dev than production versions ;)
(c) Alex Giamas, Persado Inc, All rights reserved
Reporting Version 3
● More fine grained write semantics.
– WriteConcern.SAFE for most write queries
– .REPLICAS_SAFE for non idempotent queries that are costly to recompute
● Do you feel lucky punk?
– Reactive Mongo
● Asynchronous & Non-Blocking Scala Driver for MongoDB
– Brings the best of WriteConcern.SAFE and WriteConcern.NORMAL
(c) Alex Giamas, Persado Inc, All rights reserved
Replication and High Availability Take aways
● Use delayed members
● Size your oplog
● Use writeconcern and readpreference to balance between providing fresh data and overloading servers
● Failover happens automagically but not instantaneously
● Think your security model
(c) Alex Giamas, Persado Inc, All rights reserved
Replication and High Availability Take aways
● More important: Think who has access to your systems.
– No commit, no rollback
● Prepare people for change
– Educate non engineers
– Use morphia
(c) Alex Giamas, Persado Inc, All rights reserved
Replication and High Availability Take aways
● Audit – audit – audit
– Monitor closely your MongoDB servers for potential bottlenecks
● mms.10gen.com great tool to do so
– Github is your friend:
● https://github.com/mongolab/dex
(c) Alex Giamas, Persado Inc, All rights reserved
Q&A
Ask me anything...
or drop me a line:
