MongoDB on SolidFireTuning MongoDB for Next Generation Storage Systems

MongoDB Has Changed The Landscape

● Scale-out Databases Have Changed Storage Expectations○ Expand infrastructure incrementally vs. forklift system upgrades○ Path to start small and scale extremely large (e.g. Internet sized applications)○ Avoid vendor lock-in -> not a factor due to open platform options○ Proven performance in both dedicated and virtual environments

● Schemaless Design offers Ultimate Flexibility○ Developers are not forced to code to a relational schema○ Applications can store and retrieve data in a natural format○ Functional development is simplified and streamlined ○ MongoDB can help applications scale, regardless of original dbms

Traditional Storage Can’t Keep Up

● Limited by their scale-up architectures

● Spinning disks have huge latency and throughput limitations

● HDD performance ceiling is lower than capacity ceiling

● Legacy storage systems require a PhD to configure and maintain

● Severe lack of provisioning flexibility

● Immutable configurations do not allow in-line modification

Next Gen Storage Systems Embrace the New Reality

● Scale-out storage architecture (clusters nodes together)

● Horizontal scaling approach allows for incremental, online expansion

● All SSD storage systems provides superior query performance

● Inline compression of data and deduplication maximize usable capacity

● Storage design flexibility enable flexible/adaptive database designs

○ Virtualize storage resources (GBs, IOPS)

○ Provision elastically

○ Adjust resources independently and as needed

Real World Example - Large Web 2.0 Enterprise

● Private Cloud environment with a SolidFire storage layer

● MongoDB Sharded Cluster, ~42 nodes

● Backend for a global e-commerce platform

● High volume, extremely strict operational tolerances

● Low latency critical to performance at scale

● IOPS crucial for sharded cluster stability and balancing

Architecture of a MongoDB Cluster on SolidFire (simplified)

● Cloud Environment backed by SolidFire supporting a MongoDB sharded cluster● All mongod, mongos, and config nodes, as well as storage volumes, are virtualized● No cookie-cutter limitations, the entire environment has been tailored to suit the application needs

mongos

mongod (C1)

mongos

mongod (C2)

mongod (C3)

mongod (s1)mongod (s1)

mongod (s1)

mongod (s2)mongod (s2)

mongod (s2)

mongod (s3)mongod (s3)

mongod (s3)

Cloud Environment

Single Shard

Config ServersQuery Routers

Performance Troubleshooting

● Query latency inconsistent and above tolerance requirements: avg 15-17ms

● Affecting all mongod instances in the cluster, not isolated to a single deployment, node, or shard

● What is happening during that 15-17ms?

● What configuration options can be adjusted that affect overall latency?

● Tools: mongostat , iostat, top, bash plus SolidFire API calls

● Worked with customer DBA and Ops teams to gather metrics and implement configuration optimizations on the production system

Latency Analysis and Contributor Isolation

● Identify any potential i/o queue depth bottlenecks

● Determine main contributors to end-to-end latency on the chain

● View the chain as an overall ‘latency loop’ and break down sections

● MongoDB > OS > NIC > Network > Storage > Network > NIC > OS > MongoDB

● iostat showed i/o queue depths of 100+, while iSCSI limit was 32

● Frames stalled at the OS layer while waiting to move down the chain

● Implemented mpio to combine 4 iSCSI sessions, increasing queue capacity to 128

Latency Reduction After Implementing mpio

Optimizing the Stack to Reduce Remaining Latency

● Verified Linux best practices were being followed:

○ Adjust /etc/limits.conf○ XFS agcount equal to number of cpu cores○ Readahead value set very low (4k)

● Verified SolidFire best practices were being followed:

○ I/O scheduler set to[noop]○ Volume mount options: nobarrier, noatime, nodiratime○ MPIO in use for high performance applications○ IOPS values configured high enough for MongoDB needs

● If the OS and the Storage System are optimal, and the Network latency is a static, what’s left?

Zeroing in on the Latency Source

● The latency loop from MongoDB, end-to-end, averaging 3.9ms MongoDB > OS > NIC > Network > Storage > Network > NIC > OS > MongoDB

● The inner subset loop of the OS back to the OS averaging ~1.1msOS > NIC > Network > Storage > Network > NIC > OS

● The storage subset of the OS loop was averaging 0.7ms

● The application layer, MongoDB, was the main contributor at ~2.8ms

● Latency spikes at ~60s intervals

Fine-tuning to Reduce Latency

● I/O queue depth showing periodic elevations to 100+

● MongoDB flushes pages to disk, by default, every 60s

● Latency spikes lasting ~5-6 seconds

● Tuning syncdelay to 10s lowered the flushing queue depth, reducing latency

● Final end-to-end MongoDB response times consistently 1.2ms

● 4-5k IOPS per mongod, no measurable increase in extraneous i/o

DBA and Ops Teams VERY HAPPY

Avoid Gotchas

● Be sure to use the optimal i/o scheduler for your storage subsystem

● Ensure that the correct mount options are used to avoid i/o bottlenecks

● For high-volume databases, align queue depth maximums (multipath)

● Provision adequate IOPS and burst capacity per volume for throughput

● Traverse the entire stack to mitigate latency, don’t over-focus on one area

● Consider syncdelay tuning, but evaluate this on a per-deployment basis

● Depending on storage characteristics, journal splitting may be necessary

● DO NOT RUN OUT OF IOPS. THIS KILLS THE DB CLUSTER.

Take Aways

New Approach = Amazing Results

● MongoDB has changed the database / storage relationship

● Great results can happen with database / storage architecture alignment

The Path There

● Horizontal scaling in storage is critical for optimal throughput and latency

● Understand your options for controlling performance before designing

● Have a plan for scaling both capacity and performance

1620 Pearl Street,Boulder, Colorado 80302

Phone: 720.523.3278Email: info@solidfire.com

www.solidfire.com