Migration from SQL to MongoDB - A Case Study at TheKnot.com

XO Group Inc.

Membership and Community Team

Alexander Copquin - Senior Software Engineer

Vladimir Carballo - Senior Software Engineer

Favorites API Re-platforming

…a case study

Favorites API Re-platforming

• Architectures SQL .NET / Ruby Mongo• Reasons for migration• Schema design• RoR model design and implementation• Migration strategies and systems• Lessons learned

Our Favorites Feature

Favorites API

• Add / Edit / Delete Object.• Manage Boards• Get counts & stats• RESTful API• Rails • JavaScript • Ios• Android

Features

• + 100,000,000 “favorited” objects• + 760,000 boards• Avg. 55,000 new objects per day

Legacy Architecture

• Database 55 GB and growing…

• Avg 45 rpm on peak times

• Avg 80 msec response POST

• Avg 460 msec response GET

Legacy Benchmarks

• Db Reaching max. capacity for setup• Scalability problems• Hard to modify schema• Bad response times• Very complex caching layer• Out of line with company’s strategy

New Architecture

• Easy to scale• Flexible schema• Fast Response• No Cache Layer• Fast Iteration / Deploy• TDD first and foremost• At a glance monitoring of all layers

Scalable

Implementation

What we persisted in the legacy schema

• UserId (primary key)• UniqueId• Url (unique per user)• ImageUrl• Name• Description• ObjectId (unique per application adding favorites)

• Category• Timestamps• Other

Favorites DB Legacy Schema

select top 10 UserFavoriteId, Name, Description, Url, ImageUrl from userFavorites where userId = '5174181997807393'

Sample queries

select top 5 grp.groupId, grp.Name as GroupName, fav.userFavoriteId, fav.name, fav.Description, fav.Url, fav.ImageUrlfrom userFavoritesGroups grpinner join userFavoritesGroupsItems grpItm on grp.GroupId = grpItm.GroupIdinner join userFavorites fav on grpItm.userFavoriteId = fav.userFavoriteIdwhere grp.userId = '5174181997807393'

Sample queries

Towards a new Schema and Persistance Layer

• Start with a clean slate• Break with the past• Persist only relevant minimum data points• Think and rethink relationships• High Performance• Flexible• Prototype different scenarios

First attempt

UserFavorites

• Document contains embedded documents which are required to be accessed on its own

• Documents would grow without bound• Most queries would be slow• Indexes would be very expensive• Tries too hard to imitate legacy

Second attempt

Board document with one recent favorite

Board document with more recent favorites

Favorite document located on different boards

• Document structure matches the data required on the view• A Board document includes the 4 most recent favorites.• A Favorite document includes the list of boards it was

added to• Faster queries.• More control on the size of each document• Better implementation of UX intent

Sample queriesdb.favorites.find({'member': 'e1606ed5-4ac8-48b4-aee6-bc4203937903'})

.limit(1)

db.favorites.find({'boards': '7557acf8-b7b1-4eab-a64d-57449034cfc6'})

.limit(1)

db.favorites.find({'application': 'marketplace'}) .limit(1)

db.boards.find({'member': 'e1606ed5-4ac8-48b4-aee6-bc4203937903'}) .limit(1)

db.boards.find({'member': 'e1606ed5-4ac8-48b4-aee6-bc4203937903', 'default_board': true})

db.boards.find({'name' : 'Simple Reception Decor'}).limit(1)

● Rails web application framework● We speak RoR and JS● mongoDB as a data repository (we love NoSQL)● Two collections, one for Boards and one for

Favorites● No joins, no foreign keys● Referential integrity is handled in a different fashion.● MongoId Gem (Pros & Cons)

Some implementation details

Favorites re-platform

Board class

Favorite class

Scaling reads with replica sets

Scaling reads with sharding

Migration

Clients switchover

NewAPI

Legacy API

Client

ONE WAY

Migration Timeline

new API.Continuous

Migr.Implement Monitors

Turn on Continuous

Data Catch-up

Plug ClientsBulk Migr.

Development

Bulk Migr.

Migration

Bulk Migration

FavoritesUserFavorites

SQL Tables Mongo Collections

BoardsUserFavoritesGroups

UserFavoritesGroupsItems

Favorites Job

Pentaho Steps

Auto-increment Id vs. UUID

UserFavoriteId

GroupId

Favorites UUID

Groups UUID

Continuous Migration

Get UUID from the Get Go

• Add a column to legacy Db (+ 100M recs!!) with new Mongo UUID

• Then migrations will take care of inserting into new documents

SQL has all new

idsxxxxx-xxxx-xxxxxxxxx-xxxx-xxxxxxxxx-xxxx-xxxx

Ids are inserted

Migration Systems

Add UUID Columns in SQL

100 M recs!!!Alter table add UUID uniqueidentifier

New Favs TempTablewith UUID

SELECT *, uuid = NEWID() INTO NewUserFavorites

FROM UserFavorites

Add Indexes

Rename & drop

• SQL needed some sanitation• SQL prep scripts approx. 4 hs• Pentaho ETL on local Workstation: 8hs• Restore into production Mongo Cluster: 4hs

We’ve got data!!

Continuous Migration Architecture

Clients

Legacy API New API

SQS Queue Messenger

ONE WAY SYNC…

Continuous Migration Favorites Legacy Messenger

• Ruby• Consumer of an SQS queue coming from Legacy

that generates 1 message per operation• Issues API call to new app per each operation• Runs as a worker in the background

Legacy API

SQSLegacy

Messenger

NewAPI Mongo

SQS Queue is not a FIFO Friend

Sent by Legacy

Consumed by Messenger

• Queue is not FIFO• Objects don’t exist• Queue bloats fast• Can get like not-real-time• Data is different

Challenges

• Verify if entity exists (API call),

otherwise, throw back in queue• Set message expiration• Sanitize data• Get multiple workers to achieve

near real-time syncing.

Solutions

• Favor a simple document structure• Try different schema paradigms• Bypass native objectId generation in favor of UUID• Break with the past• Queues can be deceiving• Gems can simplify application layer impl.• Manage ref. integrity in app. layer• No cache required Take away

• Avg 85 rpm on peak times

• Avg 58 msec response POST

• Avg 18 msec response GET

New Benchmarks

New vs. Legacy

• Overall Performance Increase• 18 ms vs. 460 ms for GET• 58 ms vs. 80 ms for POST • Easy Schema Changes• Scalable• Simpler architecture• No Cache layer• Fast Code iteration, testing and deployment• In-line with company’s technology strategy

Acknowledgments

• Dmitri Nesterenko• Jason Sherman• Nelly Santoso• Phillip Chiu• Sean Lipkin• George Taveras• Alison Fay• Diana Taykhman• Rajendra Prashad• Josh Keys• Lewis DiFelice

contact, questions, inquiries?

memcomtech@xogrp.com

Migration from SQL to MongoDB - A Case Study at TheKnot.com

Technology

Introduction to MongoDB Basics from SQL to NoSQL

No SQL MSATS MongoDB | SolidQ Summit 2014

One Tool to Rule Them All- Seamless SQL on MongoDB, MySQL ... · MongoDB, MySQL and Redis with Apache Spark. ... So why not Apache Drill? ... One Tool to Rule Them All- Seamless SQL

Big Data Analytics with Hadoop, MongoDB and SQL Server

Mapping SQL to MongoDB - GitHub Pages€¦ · Mapping SQL to MongoDB Converting to MongoDB Terms MYSQL EXECUTABLE ORACLE EXECUTABLE MONGODB EXECUTABLE mysqld oracle mongod mysql sqlplus

No SQL databases An experience with MongoDB

Mapping SQL to MongoDB · 2020. 7. 21. · Mapping SQL to MongoDB Converting to MongoDB Terms MYSQL EXECUTABLE ORACLE EXECUTABLE MONGODB EXECUTABLE mysqld oracle mongod mysql sqlplus

NoSql ve MongoDB - Kodcu.com – Değer katan bilgiler vs. Sql Server 2008 MongoDB C# API vs. LINQ to SQL Neden NoSql? MongoDb vs. Sql Server 2008 MongoDB C# API vs. LINQ to SQL Neden

Mapping SQL to MongoDB - Amazon S3s3.amazonaws.com/info-mongodb-com/sql_to_mongo.pdf · Below are examples of SQL statements and the analogous statements in MongoDB JavaScript shell

No-SQL, Azure Table Storage y MongoDB

Introdução no sql mongodb java

Introduction to MongoDB NGUYEN VO – A02213331 1. Content I. Overview II. SQL vs MongoDB III. MongoDB Data Model IV. MongoDB Queries V. Installation VI

進階使用Nodejs 淺談no sql(mongodb)

No SQL and MongoDB - Hyderabad Scalability Meetup

Node.js and MongoDB Compared to C#/Java/SQL

MongoDB for the SQL Server Professional

Simba ODBC Driver with SQL Connector for MongoDB ... · Simba ODBC Driver with SQL Connector for MongoDB Installation and Configuration Guide 3 8. In the Host field, type the name

Moving Hudl from MS SQL to MongoDB: A Two Year Journey

Visualizing NoSQL data with tableau and mongodb€¦ · The MongoDB BI connector connects to the BI tools, translates the SQL statements issued by the BI tool into equivalent MongoDB

SQL, NoSQL, MongoDB - Computer Science and · PDF fileSQL, NoSQL, MongoDB CSE-291 (Distributed Systems) Winter 2017 Gregory Kesden