PetaMongo: A Petabyte Database for as Little as $200

© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

PetaMongo: A Petabyte Database for as Little as $200

Chris Biow, MongoDB

Miles Ward, AWS

November 13, 2013

Agenda

• MongoDB on AWS review– Guidance, Storage, Architecture

• MongoDB at PetaScale on AWS

• Whitepaper• Marketplace• CloudFormation

Tools to simplify your design

http://media.amazonwebservices.com/AWS_NoSQL_MongoDB.pdf

• Easy to start a single node

• Correctly configured PIOPS EBS Storage

• No extra cost

https://aws.amazon.com/marketplace/pp/B00COAAEH8/ref=srh_res_product_title?ie=UTF8&sr=0-6&qid=1383897659043

mongodb.org/display/DOCS/Automating+Deployment+with+CloudFormation

• Nested Templates

• Nodes and Storage

• Configurable Scale

• CloudFormation: Your Infrastructure belongs in your source control

CloudFormation

AWS Storage Options

• EBS – Provisioned IOPS volumes• Deliver predictable, high performance for I/O intensive workloads• Specify IOPS required upfront, and EBS provisions for lifetime of volume– 4000 IOPS per volume, can stripe to get thousands of IOPS to an EC2 instance

• High IO Instances – hi1.4xlarge• For some applications that require tens of thousands of IOPS• Eliminates network latency/bandwidth as a performance constraint to storage

EBSPIOPS

SSD

AWS Storage OptionsTesting: random 4k reads

EBS

SSD

PIOPS+

One Volume: ~200 MongoOPS with some variability, <1mb/sLoaded instance: ~ 1000 MongoOPS with some variability <10mb/s

One Volume: 200 0 MongoOPS with <1% variability, 16mb/sLoaded Instance: 16,000 MongoOPS with <1% variability, 64mb/s

Loaded Cluster Instance: MongoOPS, 320mb/s

Hi1.4xlarge ephemeral: ~64,000 MongoOPS with low variability, ~245mb/s

4,000

80,000

Testing: random 4k reads

EBS

SSD

PIOPS+

Sta

ble

Stability Tips

• Ext4 or XFS, nodiratime, noatime

• Raise file descriptor limits

• Set disk read-ahead

• No large virtual memory pages

• SNAPSHOT SNAPSHOT SNAPSHOT

• Retain a PIOPS EBS node for snapshot backups

• Snapshots allow cross-AZ and cross-region recovery

• SSD hosts as primary

• Shard for scale

244gb cr1.8xlargeAnother option…

So, about that Petabyte

v.cheap

• Spot Market• m1.small• 1024 shards• 1TB EBS from snapshot• PowerBench reader• Aggregation queries

v.fast

• AutoScaling On-Demand• cc2.8xlarge• 44 instances x 24 shards

each• 24TBx1K PIOPS indexed• YCSB loader• Aggregation queries

The naming of parts

Amazon Terms

• Provisioned IOPS• Elastic Compute Cloud• EC2 Spot Instances• Auto Scaling groups

Nicks

• PIOPS• EC2• Here, Spot!• ASG

Players

MongoDB• Document-model,

NoSQL database

• Dev adoption is STRONG

• MongoDB Inc. trending toward zero h/w

• Scale-up with commodity h/w• Scale-out with sharding• Scale-around with replication

Dev Activity: stackoverflow.com

AWS

• PIOPS for an IO-hungry client• 40% of MongoDB customer usage• 90% of MongoDB internal usage• More ports :2701[79] than :[15]521

PB & ChocolateDifferentiators for mutual customers

• Fast time-to-solution• Easy global distribution• Document model• Secondary indexes• Geo, text, security• Fast analytic aggregation

Challenge

Motivation: IWBCI…

• Test scale-out of MongoDB beyond typical• Learn massive scale-out on AWS • Do it as cheaply as possible• Apply customer data• Break the petabarrier

m1.small us-east1 Spot Market

m1.small us-east1d Spot Market

ProposalItem Units Time Unit Cost Net Cost

m1.small Spot 1050 3hr $0.007/hr $22.05

m1.large 3 48hrs $0.056/hr $8.07

S3 1TB 1wk $95/TB/mo 23.75

EBS 1024 x 1TB 1hr $100/TB/mo 142.22

S3 EBS 1PB lazy $0/TB 0.00

Total $196.09

http://calculator.s3.amazonaws.com/G77798SS77SH72

Initial Directions

• Spot instance requests– m1.small market, mostly us-east-1 (my zone “d”)– Net: $0.007 / hour = $7 / hr / K-shard

• Perl– use Net::Amazon::EC2;– gaps: parse EC2 command-line API

• Defer Chef, Puppet, CloudFormation• YCSB• userdata.sh• t1.micro / m1.small / cr1.8xlarge

MongoDB Architecture

• 3x Config Servers– mongod --configsvr

• Routing– mongos --configdb a,b,c

• Replica sets (not used)• Shards

– mongod

• Client load – java -cp [] com.yahoo.ycsb.Client

Range-based sharding

Hash-based sharding

Process Flow

Spot Instance Request (sir-)

• Rejected• Awaiting evaluation• Awaiting fulfillment

– Partial– Launch intervals

• Fulfilled

Instances (i-)

• Requested• Initializing (i)• Config running (C)• MongoS starting (s)• MongoS running (S)• MongoD starting (D)• Failed/slow response (X)

Spot Instance Lifecycle

sir-

Config

Sharded

MongoD

Shard

MongoS

Progress

Scale Out Experience

• Sharding by magnitude: 4, 16, 64, 256, 1024• 4: functional validation• 16: startup variation, process flow• 64: full speed ahead!• 256: chunk distribution time, single Config• 1024: market dependence, client wire saturation

Lessons Learned

• Code defensively• Monitor: MongoDB Mgt Svc, top, iftop, iostat,

mongostat• Avoid sentimental attachment (i-8bad8bee)• Prototype / refactor• Make the instances do the work• Mitigate chunk migration

Refactor

• BenchPress YCSB• Auto Scaling Groups request-spot-instances • use VM::EC2; Net::Amazon::EC2 • gsh monolithic Perl• serf polling

Secure Cloud Networking

Enable customers to easily connect, manage and secure applications across VPCs, regions, and hybrid infrastructures.

Cloud-scale your VPC connectivity!

VPC 1 VPC 2

ApplicationServiceMesh

After the Session: Survey - $500 Gift CardOr schedule a demoInfo@unionbaynetworks.com

5:16:48 5:45:36 6:14:24 6:43:12 7:12:00 7:40:480

200,000,000

400,000,000

600,000,000

800,000,000

1,000,000,000

1,200,000,000

1,400,000,000

1,600,000,000

1,800,000,000

1KB Docs Loaded, 512 shards

^ 1X RAM

4:19:12 5:31:12 6:43:12 7:55:12 9:07:12 10:19:12 11:31:12 12:43:12 13:55:120

500,000,000

1,000,000,000

1,500,000,000

2,000,000,000

2,500,000,000

1KB Docs Loaded, 1035 shards, 2 jobs conflicting

^ 1X RAM

Dee-Luxe

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

3,000,000

3,500,000

cc2.8xlarge, 24 x 1TB-4K PIOPS EBS, bulk-load 64KB docs

64KB-docsus-latency

100% RAM

12:00:00 AM 02:24:00 AM 04:48:00 AM 07:12:00 AM 09:36:00 AM 12:00:00 PM 02:24:00 PM 04:48:00 PM 07:12:00 PM0

20,000,000

40,000,000

60,000,000

80,000,000

100,000,000

120,000,000

140,000,000

cc2.8xlarge, 24 x 1TB-4K PIOPS EBS, bulk-load 64KB docs

64KB-docsus-latency

Further Work

• Completion• Replication• Self-healing• MongoDB-appropriate benchmarks• Customer data• Self-hosting cluster

Please give us your feedback on this presentation

As a thank you, we will select prize winners daily for completed surveys!

BDT307