View
550
Download
3
Category
Preview:
Citation preview
SCHEMA DESIGN WORKSHOP
Jeremy Mikola@jmikola
AGENDA1. Basic schema design principles for MongoDB2. Schema design over an application's lifetime3. Common design patterns4. Sharding
GOALSLearn the schema design process in MongoDBPractice applying common principles via exercisesUnderstand the implications of sharding
WHAT IS A SCHEMA AND WHY IS ITIMPORTANT?
SCHEMAMap concepts and relationships to dataSet expectations for the dataMinimize overhead of iterative modificationsEnsure compatibility
NORMALIZATIONusers
usernamefirst_namelast_name
← bookstitleisbnlanguagecreated_byauthor
→ authorsfirst_namelast_name
DENORMALIZATIONusers
usernamefirst_namelast_name
← bookstitleisbnlanguagecreated_byauthorfirst_namelast_name
WHAT IS SCHEMA DESIGN LIKE INMONGODB?
Schema is defined at the application-levelDesign is part of each phase in its lifetimeThere is no magic formula
MONGODB DOCUMENTSStorage in BSON → BSONSpec.org
ScalarsDoublesIntegers (32 or 64-bit)UTF-8 stringsUTC Date, timestampBinary, regex, codeObject IDnull
Rich typesObjectsArrays
TERMINOLOGY{ "mongodb" : "relational db", "database" : "database", "collection" : "table", "document" : "row", "index" : "index", "sharding" : { "shard" : "partition", "shard key" : "partition key" }}
THREE CONSIDERATIONS IN MONGODBSCHEMA DESIGN
1. The data your application needs2. Your application's read usage of the data3. Your application's write usage of the data
CASE STUDYLIBRARY WEB APPLICATION
Different schemas are possible
AUTHOR SCHEMA{ "_id": int, "first_name": string, "last_name": string}
USER SCHEMA{ "_id": int, "username": string, "password": string}
BOOK SCHEMA{ "_id": int, "title": string, "slug": string, "author": int, "available": boolean, "isbn": string, "pages": int, "publisher": { "city": string, "date": date, "name": string }, "subjects": [ string, string ], "language": string, "reviews": [ { "user": int, "text": string }, { "user": int, "text": string } ],}
EXAMPLE DOCUMENTS
AUTHOR DOCUMENT> db.authors.findOne(){ _id: 1, first_name: "F. Scott", last_name: "Fitzgerald"}
USER DOCUMENT> db.users.findOne(){ _id: 1, username: "emily@10gen.com", password: "slsjfk4odk84k209dlkdj90009283d"}
BOOK DOCUMENT> db.books.findOne(){ _id: 1, title: "The Great Gatsby", slug: "9781857150193‐the‐great‐gatsby", author: 1, available: true, isbn: "9781857150193", pages: 176, publisher: { name: "Everyman's Library", date: ISODate("1991‐09‐19T00:00:00Z"), city: "London" }, subjects: ["Love stories", "1920s", "Jazz Age"], language: "English", reviews: [ { user: 1, text: "One of the best…" }, { user: 2, text: "It's hard to…" } ]}
EMBEDDED OBJECTSAKA EMBEDDED OR SUB-DOCUMENTS
What advantages do they have?
When should they be used?
EMBEDDED OBJECTS> db.books.findOne(){ _id: 1, title: "The Great Gatsby", slug: "9781857150193‐the‐great‐gatsby", author: 1, available: true, isbn: "9781857150193", pages: 176, publisher: { name: "Everyman's Library", date: ISODate("1991‐09‐19T00:00:00Z"), city: "London" }, subjects: ["Love stories", "1920s", "Jazz Age"], language: "English", reviews: [ { user: 1, text: "One of the best…" }, { user: 2, text: "It's hard to…" } ]}
EMBEDDED OBJECTSGreat for read performanceOne seek to load the entire documentOne round trip to the databaseWrites can be slow if constantly adding to objects
LINKED DOCUMENTSWhat advantages does this approach have?
When should they be used?
LINKED DOCUMENTS> db.books.findOne(){ _id: 1, title: "The Great Gatsby", slug: "9781857150193‐the‐great‐gatsby", author: 1, available: true, isbn: "9781857150193", pages: 176, publisher: { publisher_name: "Everyman's Library", date: ISODate("1991‐09‐19T00:00:00Z"), publisher_city: "London" }, subjects: ["Love stories", "1920s", "Jazz Age"], language: "English", reviews: [ { user: 1, text: "One of the best…" }, { user: 2, text: "It's hard to…" } ]}
LINKED DOCUMENTSMore, smaller documentsCan make queries by ID very simpleAccessing linked document data requires extra readWhat effect does this have on the system?
DATA, RAM AND DISK
ARRAYSWhen should they be used?
ARRAY OF SCALARS> db.books.findOne(){ _id: 1, title: "The Great Gatsby", slug: "9781857150193‐the‐great‐gatsby", author: 1, available: true, isbn: "9781857150193", pages: 176, publisher: { name: "Everyman's Library", date: ISODate("1991‐09‐19T00:00:00Z"), city: "London" }, subjects: ["Love stories", "1920s", "Jazz Age"], language: "English", reviews: [ { user: 1, text: "One of the best…" }, { user: 2, text: "It's hard to…" } ]}
ARRAY OF OBJECTS db.books.findOne(){ _id: 1, title: "The Great Gatsby", slug: "9781857150193‐the‐great‐gatsby", author: 1, available: true, isbn: "9781857150193", pages: 176, publisher: { name: "Everyman's Library", date: ISODate("1991‐09‐19T00:00:00Z"), city: "London" }, subjects: ["Love stories", "1920s", "Jazz Age"], language: "English", reviews: [ { user: 1, text: "One of the best…" }, { user: 2, text: "It's hard to…" } ],}
EXERCISE #1Design a schema for users and their book reviews
Usersusername (string)email (string)
Reviewstext (string)rating (integer)created_at (date)
Usernames are immutable
EXERCISE #1: SOLUTION AReviews may be queried by user or book
// db.users (one document per user){ _id: ObjectId("…"), username: "bob", email: "bob@example.com"}
// db.reviews (one document per review){ _id: ObjectId("…"), user: ObjectId("…"), book: ObjectId("…"), rating: 5, text: "This book is excellent!", created_at: ISODate("2012‐10‐10T21:14:07.096Z")}
EXERCISE #1: SOLUTION BOptimized to retrieve reviews by user
// db.users (one document per user with all reviews){ _id: ObjectId("…"), username: "bob", email: "bob@example.com", reviews: [ { book: ObjectId("…"), rating: 5, text: "This book is excellent!", created_at: ISODate("2012‐10‐10T21:14:07.096Z") } ]}
EXERCISE #1: SOLUTION COptimized to retrieve reviews by book
// db.users (one document per user){ _id: ObjectId("…"), username: "bob", email: "bob@example.com"}
// db.books (one document per book with all reviews){ _id: ObjectId("…"), // Other book fields… reviews: [ { user: ObjectId("…"), rating: 5, text: "This book is excellent!", created_at: ISODate("2012‐10‐10T21:14:07.096Z") } ]}
SCHEMA DESIGN OVER AN APPLICATION'SLIFETIME
DevelopmentProductionIterative Modifications
DEVELOPMENT PHASEBasic CRUD functionality
CREATE
The _id field is unique and automatically indexedMongoDB will generate an ObjectId if not provided
RUD author = { _id: 2, first_name: "Arthur", last_name: "Miller" };
db.authors.insert(author);
READC UD> db.authors.find({ "last_name": "Miller" }){ _id: 2, first_name: "Arthur", last_name: "Miller"}
READS AND INDEXINGExamine the query after creating an index.
> db.books.ensureIndex({ "slug": 1 })
> db.books.find({ "slug": "the‐great‐gatsby" }).explain(){ "cursor": "BtreeCursor slug_1", "isMultiKey" : false, "n" : 1, "nscannedObjects" : 1, "nscanned" : 1, "scanAndOrder" : false, "indexOnly" : false, "nYields" : 0, "nChunkSkips" : 0, "millis" : 0, // Other fields follow…}
MULTI-KEY INDEXESIndex all values in an array field.
> db.books.ensureIndex({ "subjects": 1 });
INDEXING EMBEDDED FIELDSIndex an embedded object's field.
> db.books.ensureIndex({ "publisher.name": 1 })
QUERY OPERATORSConditional operators
$gt, $gte, $lt, $lte, $ne, $all, $in, $nin, $size,$and, $or, $nor, $mod, $type, $exists
Regular expressionsValue in an array
$elemMatchCursor methods and modifiers
count(), limit(), skip(), snapshot(), sort(),batchSize(), explain(), hint()
UPDATECR D review = { user: 1, text: "I did NOT like this book." };
db.books.update( { _id: 1 }, { $push: { reviews: review }} );
ATOMIC MODIFIERSUpdate specific fields within a document
$set, $unset$push, $pushAll$addToSet, $pop$pull, $pullAll$rename$bit
DELETECRU > db.books.remove({ _id: 1 })
PRODUCTION PHASEEvolve schema to meet the application's read and write
patterns
READ USAGEFinding books by an author's first name
authors = db.authors.find({ first_name: /̂f.*/i }, { _id: 1 });
authorIds = authors.map(function(x) { return x._id; });
db.books.find({author: { $in: authorIds }});
READ USAGE"Cache" the author name in an embedded document
Queries are now one step
> db.books.findOne(){ _id: 1, title: "The Great Gatsby", author: { first_name: "F. Scott", last_name: "Fitzgerald" } // Other fields follow…}
> db.books.find({ author.first_name: /̂f.*/i })
WRITE USAGEUsers can review a book
Document size limit (16MB)Storage fragmentation after many updates/deletes
review = { user: 1, text: "I thought this book was great!", rating: 5};
> db.books.update( { _id: 3 }, { $push: { reviews: review }});
EXERCISE #2Display the 10 most recent reviews by a userMake efficient use of memory and disk seeks
EXERCISE #2: SOLUTIONStore users' reviews in monthly buckets
// db.reviews (one document per user per month){ _id: "bob‐201210", reviews: [ { _id: ObjectId("…"), rating: 5, text: "This book is excellent!", created_at: ISODate("2012‐10‐10T21:14:07.096Z") }, { _id: ObjectId("…"), rating: 2, text: "I didn't really enjoy this book.", created_at: ISODate("2012‐10‐11T20:12:50.594Z") } ]}
EXERCISE #2: SOLUTIONAdding a new review to the appropriate bucket
myReview = { _id: ObjectId("…"), rating: 3, text: "An average read.", created_at: ISODate("2012‐10‐13T12:26:11.502Z")};
> db.reviews.update( { _id: "bob‐2012‐10" }, { $push: { reviews: myReview }});
EXERCISE #2: SOLUTIONDisplay the 10 most recent reviews by a user
cursor = db.reviews.find( { _id: /̂bob‐/ }, { reviews: { $slice: 10 }}).sort({ _id: ‐1 });
num = 0;
while (cursor.hasNext() && num < 10) { doc = cursor.next();
for (var i = 0; i < doc.reviews.length && num < 10; ++i, ++num) { printjson(doc.reviews[i]); }}
EXERCISE #2: SOLUTIONDeleting a review
cursor = db.reviews.update( { _id: "bob‐2012‐10" }, { $pull: { reviews: { _id: ObjectId("…") }}});
ITERATIVEMODIFICATIONS
Schema design is evolutionary
ALLOW USERS TO BROWSE BY BOOKSUBJECT
How can you search this collection?Be aware of document size limitationsBenefit from hierarchy being in same document
> db.subjects.findOne(){ _id: 1, name: "American Literature", sub_category: { name: "1920s", sub_category: { name: "Jazz Age" } }}
TREE STRUCTURES> db.subjects.find(){ _id: "American Literature" }
{ _id : "1920s", ancestors: ["American Literature"], parent: "American Literature"}
{ _id: "Jazz Age", ancestors: ["American Literature", "1920s"], parent: "1920s"}
{ _id: "Jazz Age in New York", ancestors: ["American Literature", "1920s", "Jazz Age"], parent: "Jazz Age"}
TREE STRUCTURESFind sub-categories of a given subject
> db.subjects.find({ ancestors: "1920s" }){ _id: "Jazz Age", ancestors: ["American Literature", "1920s"], parent: "1920s"}
{ _id: "Jazz Age in New York", ancestors: ["American Literature", "1920s", "Jazz Age"], parent: "Jazz Age"}
EXERCISE #3Allow users to borrow library books
User sends a loan requestLibrary approves or notRequests time out after seven days
Approval process is asynchronousRequests may be prioritized
EXERCISE #3: SOLUTIONNeed to maintain order and stateEnsure that updates are atomic
// Create a new loan request> db.loans.insert({ _id: { borrower: "bob", book: ObjectId("…") }, pending: false, approved: false, priority: 1,});
// Find the highest priority request and mark as pending approvalrequest = db.loans.findAndModify({ query: { pending: false }, sort: { priority: ‐1 }, update: { $set: { pending: true, started: new ISODate() }}, new: true});
EXERCISE #3: SOLUTIONUpdated and added fieldsModified document was returned
{ _id: { borrower: "bob", book: ObjectId("…") }, pending: true, approved: false, priority: 1, started: ISODate("2012‐10‐11T22:09:42.542Z")}
EXERCISE #3: SOLUTION// Library approves the loan request> db.loans.update( { _id: { borrower: "bob", book: ObjectId("…") }}, { $set: { pending: false, approved: true }});
EXERCISE #3: SOLUTION// Request times out after seven dayslimit = new Date();limit.setDate(limit.getDate() ‐ 7);
> db.loans.update( { pending: true, started: { $lt: limit }}, { $set: { pending: false, approved: false }});
EXERCISE #4Allow users to recommend books
Users can recommend each book only onceDisplay a book's current recommendations
EXERCISE #4: SOLUTION// db.recommendations (one document per user per book)> db.recommendations.insert({ book: ObjectId("…"), user: ObjectId("…")});
// Unique index ensures users can't recommend twice> db.recommendations.ensureIndex( { book: 1, user: 1 }, { unique: true });
// Count the number of recommendations for a book> db.recommendations.count({ book: ObjectId("…") });
EXERCISE #4: SOLUTIONIndexes in MongoDB are not countingCounts are computed via index scansDenormalize totals on books
> db.books.update( { _id: ObjectId("…") }, { $inc: { recommendations: 1 }}});
COMMON DESIGNPATTERNS
ONE-TO-ONERELATIONSHIP
Let's pretend that authors only write one book.
LINKINGEither side, or both, can track the relationship.
> db.books.findOne(){ _id: 1, title: "The Great Gatsby", slug: "9781857150193‐the‐great‐gatsby", author: 1, // Other fields follow…}
> db.authors.findOne({ _id: 1 }){ _id: 1, first_name: "F. Scott", last_name: "Fitzgerald" book: 1,}
EMBEDDED OBJECT> db.books.findOne(){ _id: 1, title: "The Great Gatsby", slug: "9781857150193‐the‐great‐gatsby", author: { first_name: "F. Scott", last_name: "Fitzgerald" } // Other fields follow…}
ONE-TO-MANYRELATIONSHIP
In reality, authors may write multiple books.
ARRAY OF ID'SThe "one" side tracks the relationship.
Flexible and space-efficientAdditional query needed for non-ID lookups
> db.authors.findOne(){ _id: 1, first_name: "F. Scott", last_name: "Fitzgerald", books: [1, 3, 20]}
SINGLE FIELD WITH IDThe "many" side tracks the relationship.
> db.books.find({ author: 1 }){ _id: 1, title: "The Great Gatsby", slug: "9781857150193‐the‐great‐gatsby", author: 1, // Other fields follow…}
{ _id: 3, title: "This Side of Paradise", slug: "9780679447238‐this‐side‐of‐paradise", author: 1, // Other fields follow…}
ARRAY OF OBJECTS
Use $slice operator to return a subset of books
> db.authors.findOne(){ _id: 1, first_name: "F. Scott", last_name: "Fitzgerald", books: [ { _id: 1, title: "The Great Gatsby" }, { _id: 3, title: "This Side of Paradise" } ] // Other fields follow…}
MANY-TO-MANYRELATIONSHIP
Some books may also have co-authors.
ARRAY OF ID'S ON BOTH SIDES> db.books.findOne(){ _id: 1, title: "The Great Gatsby", authors: [1, 5] // Other fields follow…}
> db.authors.findOne(){ _id: 1, first_name: "F. Scott", last_name: "Fitzgerald", books: [1, 3, 20]}
ARRAY OF ID'S ON BOTH SIDESQuery for all books by a given author
Query for all authors of a given book
> db.books.find({ authors: 1 });
> db.authors.find({ books: 1 });
ARRAY OF ID'S ON ONE SIDE> db.books.findOne(){ _id: 1, title: "The Great Gatsby", authors: [1, 5] // Other fields follow…}
> db.authors.findOne({ _id: { $in: [1, 5] }}){ _id: 1, first_name: "F. Scott", last_name: "Fitzgerald"}
{ _id: 5, first_name: "Unknown", last_name: "Co‐author"}
ARRAY OF ID'S ON ONE SIDEQuery for all books by a given author
Query for all authors of a given book
> db.books.find({ authors: 1 });
book = db.books.findOne( { title: "The Great Gatsby" }, { authors: 1 });
db.authors.find({ _id: { $in: book.authors }});
EXERCISE #5Tracking time series data
Graph recommendations per unit of timeCount by: day, hour, minute
EXERCISE #5: SOLUTION A// db.rec_ts (time series buckets, hour and minute sub‐docs)> db.rec_ts.insert({ book: ObjectId("…"), day: ISODate("2012‐10‐11T00:00:00.000Z") total: 0, hour: { "0": 0, "1": 0, /* … */ "23": 0 }, minute: { "0": 0, "1": 0, /* … */ "1439": 0 }});
// Record a recommendation created one minute before midnight> db.rec_ts.update( { book: ObjectId("…"), day: ISODate("2012‐10‐11T00:00:00.000Z") }, { $inc: { total: 1, "hour.23": 1, "minute.1439": 1 }}});
BSON STORAGESequence of key/value pairsNot a hash mapOptimized to scan quickly
minute[0] [1] … [1439]
What is the cost of updating the minute before midnight?
BSON STORAGEWe can skip sub-documents
hour0[0] [1] … [59]
… hour23[1380] … [1439]
How could this change the schema?
EXERCISE #5: SOLUTION B// db.rec_ts (time series buckets, each hour a sub‐doc)> db.rec_ts.insert({ book: ObjectId("…"), day: ISODate("2012‐10‐11T00:00:00.000Z") total: 148, hour: { "0": { total: 7, "0": 0, /* … */ "59": 2 }, "1": { total: 3, "60": 1, /* … */ "119": 0 }, // Other hours… "23": { total: 12, "1380": 0, /* … */ "1439": 3 } }});
// Record a recommendation created one minute before midnight> db.rec_ts.update( { book: ObjectId("…"), day: ISODate("2012‐10‐11T00:00:00.000Z") }, { $inc: { total: 1, "hour.23.total": 1, "hour.23.1439": 1 }}});
SINGLE-COLLECTION INHERITANCETake advantage of MongoDB's features
Documents need not all have the same fieldsSparsely index only present fields
SCHEMA FLEXIBILITY
Find all books that are part of a series
> db.books.findOne(){ _id: 47, title: "The Wizard Chase", type: "series", series_title: "The Wizard's Trilogy", volume: 2 // Other fields follow…}
db.books.find({ type: "series" });
> db.books.find({ series_title: { $exists: true }});
> db.books.find({ volume: { $gt: 0 }});
INDEX ONLY PRESENT FIELDSDocuments without these fields will not be indexed.> db.books.ensureIndex({ series_title: 1 }, { sparse: true })
> db.books.ensureIndex({ volume: 1 }, { sparse: true })
EXERCISE #6Users can recommend at most 10 books
EXERCISE #6: SOLUTION// db.user_recs (track user's remaining and given recommendations)> db.user_recs.insert({ _id: "bob", remaining: 8, books: [3, 10]});
// Record a recommendation if possible> db.user_recs.update( { _id: "bob", remaining: { $gt: 0 }, books: { $ne: 4 }}, { $inc: { remaining: ‐1 }, $push: { books: 4 }}});
EXERCISE #6: SOLUTIONOne less unassigned recommendation remainingNewly-recommended book is now linked
> db.user_recs.findOne(){ _id: "bob", remaining: 7, books: [3, 10, 4]}
EXERCISE #7Statistic buckets
Each book has a listing page in our applicationRecord referring website domains for each bookCount each domain independently
EXERCISE #7: SOLUTION A> db.book_refs.findOne(){ book: 1, referrers: [ { domain: "google.com", count: 4 }, { domain: "yahoo.com", count: 1 } ]}
> db.book_refs.update( { book: 1, "referrers.domain": "google.com" }, { $inc: { "referrers.$.count": 1 }});
EXERCISE #7: SOLUTION AUpdate the position of the first matched element.
What if a new referring website is used?
> db.book_refs.update( { book: 1, "referrers.domain": "google.com" }, { $inc: { "referrers.$.count": 1 }});
> db.book_refs.findOne(){ book: 1, referrers: [ { domain: "google.com", count: 5 }, { domain: "yahoo.com", count: 1 } ]}
EXERCISE #7: SOLUTION B
Replace dots with underscores for key namesIncrement to add a new referring websiteUpsert in case this is the book's first referrer
> db.book_refs.findOne(){ book: 1, referrers: { "google_com": 5, "yahoo_com": 1 }}
> db.book_refs.update( { book: 1 }, { $inc: { "referrers.bing_com": 1 }}, true);
SHARDING
SHARDINGAd-hoc partitioningConsistent hashing
Amazon DynamoDBRange based partitioning
Google BigTableYahoo! PNUTSMongoDB
SHARDING IN MONGODBAutomated managementRange based partitioningConvert to sharded system with no downtimeFully consistent
SHARDING A COLLECTION
Keys range from −∞ to +∞Ranges are stored as chunks
> db.runCommand({ addshard : "shard1.example.com" });
> db.runCommand({ shardCollection: "library.books", key: { _id : 1}});
SHARDING DATA BY CHUNKS
[ −∞, +∞) → [−∞, 40)[40, +∞)
→ [−∞, 40)[40, 50)[50, +∞)
Ranges are split into chunks as data is inserted
> db.books.save({ _id: 35, title: "Call of the Wild" });> db.books.save({ _id: 40, title: "Tropic of Cancer" });> db.books.save({ _id: 45, title: "The Jungle" });> db.books.save({ _id: 50, title: "Of Mice and Men" });
ADDING NEW SHARDSshard1[−∞, 40)[40, 50)[50, 60)[60, +∞)
ADDING NEW SHARDS
shard1[−∞, 40) [50, 60)
shard2 [40, 50) [60, +∞)
Chunks are migrated to balance shards
> db.runCommand({ addshard : "shard2.example.com" });
ADDING NEW SHARDS
shard1[−∞, 40)
shard2 [40, 50) [60, +∞)
shard3 [50, 60)
> db.runCommand({ addshard : "shard3.example.com" });
SHARDING COMPONENTSmongosConfig serversShards
mongodReplica sets
SHARDED WRITESInserts
Shard key requiredRouted
Updates and removesShard key optionalMay be routed or scattered
SHARDED READSQueries
By shard key: routedWithout shard key: scatter/gather
Sorted queriesBy shard key: routed in orderWithout shard key: distributed merge sort
EXERCISE #8Users can upload images for books
imagesimage_id: ???data: binary
The collection will be sharded by image_id.
What should image_id be?
EXERCISE #8: SOLUTIONSWhat's the best shard key for our use case?
Auto-increment (ObjectId)MD5 of dataTime (e.g. month) and MD5
Right-balanced Access
Random Access
Segmented Access
SUMMARYSchema design is different in MongoDB.Basic data design principles apply.It's about your application.It's about your data and how it's used.It's about the entire lifetime of your application.
THANKS!QUESTIONS?
Recommended