Michael DelNegroPrincipal Database Administrator

AOL

“Operationalizing”

MongoDB@AOL

About Me

• DBA at AOL (Dulles) for 6+ years• Original DBMS Background is in Sybase• Now MySQL, MongoDB, PostgreSQL and

NoSQL (Neo4J, Redis, etc)• Check out www.meetup.com/NOVA-

MySQL

The First Mongo at AOL• A MapQuest Developer in the

Summer of 2010 Replaced a MySQL Datastore with Mongo

• That Developer left in early 2011 :-/• Was Handed Over to Operations and

Yours Truly

Never Forget Your First• Not Small: 40+ Hosts• Upgrade Challenges• Connection Floods• Support to the Rescue• Magnified the Reasons to

Operationalize

Operation “Operationalize”

Operation “Operationalize”

• Take Advantage of 10Gen Support• Establish Standards• Monitoring/Alerting• Backups• Information Sharing• Challenges

10Gen Support

• JIRA Documents Cases Enabling Information Sharing and Referencing

• Serves Us Well in Both “Emergencies” and “Hey Quick Question”

• Training Classes: Administration & Development

Standard Host Setup

• CentOS 6• XFS/EXT4• Deadline I/O Scheduler• dirty_ratio = 10 (down from 40)• dirty_background_ratio = 5 (down

from 10)• noatime

Standard Build Scripts

• Use 64bit Linux Binaries Stored in AOL Repository

• mongodb_install.sh– Parameters Include:• MongoDB Version• Replica Set Name• Sharded? (Y/N)• Port (27017 is standard)• Type (DB, Arbiter, Config, Mongos)

Standard Directory/Layouts

• Based on our standard MySQL layout• Binaries in /opt/local/pkg• /db/mongodb/<port> base directory

– Ex. /db/mongodb/27017– Can Override

• /db/mongodb/27017-repset1

• Under base directory– data– etc (config files)– logs– scripts (ex stop/start)

ORB

• AOL Technologies’ Configuration Management Database (CMDB)

• Integrated with many authoritative data repositories

• Unique namespace for many operations data points

• Data model for operations management• Projects, Assets, People, Applications,

Network Data, HCM• SQL Interface

Monitoring/Alerting

• Argus• Nagios• Component Test Tool• MMS (MongoDB Monitoring Service)

Argus

• Metric and Event– Collector– Thresholds –Management– Data Viewing

Argus

• Capture serverStatus Metrics (once a minute)• Capture Replication Lag• Great for Trending• Great for Capacity Planning• Great for Troubleshooting• We Also Use for Host Metrics (CPU, I/O, etc)• Administrated By a Small and very Busy Group– Requested Additions/Changes Can Be Slow

Nagios

• Great For Fault-detection/Alerting• Great For Show Me What Is Currently

Broken• Great For Service Availability Metrics• Flexible• Reduces Pressure On NOC• Integrates With Netcool, Ignore Tool• We Write Our Own Plug-Ins

Component Test Tool

• Runs Parallel HTTP GETs Across a Group of Hosts

• Matches Regular Expressions On the Page Content (or HTTP Headers)

• Displays Number (or Content) of Matches in a HTML Table

MMS

Backups

• Scripts Delivered With Install• Choice Between LVM (default) and

Standard (mongodump)• Ability to Backup Sharded Setup

Mongo @ AOL Currently

Mongo @ AOL Currently

• 30+ Active MongoDB Projects• 500+ Servers in Production• Developed in:

– Java– PHP– Ruby– C#– Erlang– Scala– Perl– C++

Information Sharing

• Internal Wikis/Websites• UnUniversity (Organized Brown Bags)• Architecture Reviews• 10Gen JIRA Access

Challenges

• Developers Love It • BUT Requires More Upfront Planning

Than They May Realize• Should NOT Be Our Default Datastore• Better Developer OnBoarding• Aggressive Upgrading Is Required• Coordination of Driver Upgrades

The Future

• “MongoDB As a Service”• Puppet or Chef• Internal Administration Console– Already has Visual Topologies

• Internal MongoDB Operations Class– First One is in July

Resources

• Recommended Presentation: Todd Dampier’s Rock-Solid Mongo Ops (mongolab)

• Check out Jon Reed’s MongoDB for Online Advertising at AOL (MongoDB UK 2012)

• www.meetup.com/Washington-DC-MongoDB-Users-Group/

Thank You!

• www.slideshare.net/radiocats• @radiocats• www.linkedin.com/in/mdelnegro• Thank you, SeniorGif