Lessons learnt building

@RossC0http://github.com/rozza

WHAT IS MONGODB?

A document database

Highly scalable

Developer friendly

http://mongodb.org

In BSON

{ _id : ObjectId("..."), author : "Ross", date : ISODate("2012-07-05..."), text : "About MongoDB...", tags : [ "tech", "databases" ], comments : [{ author : "Tim", date : ISODate("2012-07-05..."), text : "Best Post Ever!" }], comment_count : 1}

WHAT IS MONGODB?

In Python In BSON

{ "_id" : ObjectId("..."), "author" : "Ross", "date" : datetime(2012,7,5,10,0), "text" : "About MongoDB...", "tags" : ["tech", "databases"], "comments" : [{ "author" : "Tim", "date" : datetime(2012,7,5,11,35), "text" : "Best Post Ever!" }], "comment_count" : 1}

{ _id : ObjectId("..."), author : "Ross", date : ISODate("2012-07-05..."), text : "About MongoDB...", tags : [ "tech", "databases" ], comments : [{ author : "Tim", date : ISODate("2012-07-05..."), text : "Best Post Ever!" }], comment_count : 1}

http://education.10gen.com

Want to know more?

http://www.flickr.com/photos/51838104@N02/5841690990W H Y D O Y O U E V E N N E E D A N O D M ?

MongoDB a good fit

Documents schema in codeEnforces schemaData validationSpeeds up developmentBuild tooling off itCan DRY up code...

SCHEMA LESS != CHAOS

Inspired by Django's ORM

Supports Python 2.5 - Python 3.3

Originally authored by Harry Marr 2010

I took over development in May 2011

Current release 0.7.5

http://github.com/MongoEngine/mongoengine

INTRODUCING MONGOENGINE

class Post(Document): title = StringField(max_length=120, required=True) author = ReferenceField('User') tags = ListField(StringField(max_length=30)) comments = ListField(EmbeddedDocumentField('Comment'))

class Comment(EmbeddedDocument): content = StringField() name = StringField(max_length=120)

class User(Document): email = StringField(required=True) first_name = StringField(max_length=50) last_name = StringField(max_length=50)

CREATING A MODEL

class Post(Document): title = StringField(max_length=120, required=True) author = ReferenceField('User') tags = ListField(StringField(max_length=30)) comments = ListField(EmbeddedDocumentField('Comment'))

Define a class inheriting from Document

Map a field to a defined data type strings, ints, binary, files, lists etc..

By default all declared fields aren't required

Pass keyword arguments to apply constraints eg set if unique, max_length, default values.

INSERTING DATA

# Pass data into the constructoruser = User(email="ross@10gen.com", name="Ross").save()

# Create instance and edit / update in placepost = Post()post.title = "mongoengine"post.author = userpost.tags = ['odm', 'mongodb', 'python']post.save()

Create instance of the object

Update its attributes

Call save, insert, update to persist the data

QUERYING DATA

# An `objects` manager is added to every `Document` classusers = User.objects(email='ross@10gen.com')

# Pass kwargs to commands are lazy and be extended as neededusers.filter(auth=True)

# Iterating evaluates the querysetprint [u for u in users]

Documents have a queryset manager (objects) for querying

You can continually extend it

Queryset evaluated on iteration

6 LESSONS LEARNT

LE S S O N 1 : D I V E I N !http://www.flickr.com/photos/jackace/565857899/

In May 2011

>200 forks>100 issues~50 pull requests

I needed it

Volunteered to helpStarted reviewing issuesSupported Harry and community

PROJECT STALLED

LE S S O N 2 : M E TA C L A S S E Shttp://www.flickr.com/photos/ubique/135848053

WHATS NEEDED TO MAKE AN ORM?

Instance methods

validation datamanipulate dataconvert data to and from mongodb

Queryset methods

Finding dataBulk changes

METACLASSES

class Document(object): __metaclass__ = TopLevelDocumentMetaclass ...

class EmbeddedDocument(object): __metaclass__ = DocumentMetaclass ...

Needed for:

1. inspect the object inheritance

2. inject functionality to the class

Its surprisingly simple - all we need is: __new__

METACLASSES 101

TopLevelDocument

Document

python's type

cls, name, bases, attrs

IN

new class

Out

METACLASSES

TopLevelDocument

Document

python's type

Creates default meta datainheritance rules, id_field, index information, default ordering.Merges in parents metaValidationabstract flag on an inherited class collection set on a subclass

Manipulates the attrs going in.

IN

METACLASSES

TopLevelDocument

Document

python's type

Merges all fields from parentsAdds in own field definitionsCreates lookups_db_field_map_reverse_db_field_mapDetermine superclasses (for model inheritance)

IN

METACLASSES

TopLevelDocument

Document

python's type

Adds in handling for delete rulesSo we can handle deleted References

Adds class to the registrySo we can load the data into the correct class

OUT

METACLASSES

TopLevelDocument

Document

python's type

Builds index specificationsInjects queryset managerSets primary key (if needed)

OUT

LESSONS LEARNT

Spend time learning what is being done and why

Don't continually patch:

Rewrote the metaclasses in 0.7

LE S S O N 3 : S T R AY I N G F R O M T H E PAT Hhttp://www.flickr.com/photos/51838104@N02/5841690990

REWRITING THE QUERY LANGUAGE

# In pymongo you pass dictionaries to queryuk_pages = db.page.find({"published": True})# In mongoengineuk_pages = Page.objects(published=True)

# pymongo dot syntax to query sub documentsuk_pages = db.page.find({"author.country": "uk"})# In mongoengine we use dunder insteaduk_pages = Page.objects(author__country='uk')

REWRITING THE QUERY LANGUAGE

#Somethings are nicer - regular expresion searchdb.post.find({'title': re.compile('MongoDB', re.IGNORECASE)})Post.objects(title__icontains='MongoDB') # In mongoengine

# Chaining is nicerdb.post.update({"published": False}, {"$set": {"published": True}}, multi=True)

Post.objects(published=False).update(set__published=True)

LE S S O N 4 : N OT A LL I D E A S A R E G O O Dhttp://www.flickr.com/photos/abiding_silence/6951229015

CHANGING SAVE# In pymongo save replaces the whole documentdb.post.save({'_id': 'my_id', 'title': 'MongoDB', 'published': True})

# In mongoengine we track changespost = Post.objects(_id='my_id').first()post.published = Truepost.save()

# Results in:db.post.update({'_id': 'my_id'}, {'$set': {'published': True}})

CHANGING SAVE

Any field changes are noted

How to monitor lists and dicts?

Custom List and Dict classes

Observes changes and marks as dirty

HOW IT WORKS

class Post(Document): title = StringField(max_length=120, required=True) author = ReferenceField('User') tags = ListField(StringField(max_length=30)) comments = ListField(EmbeddedDocumentField('Comment'))

class User(Document): email = StringField(required=True) first_name = StringField(max_length=50) last_name = StringField(max_length=50)

class Comment(EmbeddedDocument): content = StringField() name = StringField(max_length=120)

Post

HOW IT WORKS

- comments

comment

comment

comment

post = Post.objects.first()post.comments[1].name = 'Fred'post.save()

Post

HOW IT WORKS

- comments

comment

1.Convert the comments data to a BaseList

BaseList Stores the instance and name / location

comment

comment

post.comments[1].name = 'Fred'

Post

HOW IT WORKS

- comments

comment

2.Convert the comment data to BaseDict

sets name as: 'comments.1'

comment

comment

post.comments[1].name = 'Fred'

Post

HOW IT WORKS

- comments

comment

3.Change name to "Fred"

4. Tell Post 'comments.1.name' has changed

comment

comment

post.comments[1].name = 'Fred'

Post

HOW IT WORKS

- comments

comment

5.On save()Iterate all the changes on post and run $set / $unset queries

comment

comment

post.save()

db.post.update( {'_id': 'my_id'}, {'$set': { 'comments.1.name': 'Fred'}})

A GOOD IDEA?

+ Makes it easier to use

+ save acts how people think it should

- Its expensive

- Doesn't help people understand MongoDB

LE S S O N 5 : M A N A G I N G A C O M M U N I T Yhttp://kingscross.the-hub.net/

Github effect

>10 django mongoengineprojectsNone active on pypiLittle cross project communication

CODERS JUST WANT TO CODE *

* Side effect of being stalled?

Flask-mongoengine on pypi There were 2 different projectsNow has extra maintainers from the flask-mongorest

Django-mongoengine*Spoke to authors of 7 projects and merged their work together to form a single library

* Coming soon!

REACH OUT

THE COMMUNITY

Not all ideas are good!

Vocal people don't always have great ideas

Travis is great* * but you still have to read the pull request

Communities have to be managedI've only just started to learn how to herding cats

LE S S O N 6 : D O N ' T B E A F R A I D TO A S Khttp://www.flickr.com/photos/kandyjaxx/2012468692

WebsiteDocumentationTutorialsFramework supportCore mongoengine

http://twitter.com/RossC0

I NEED HELP ;)

http://github.com/MongoEngine

Q U E S T I O N S ?http://www.flickr.com/photos/9550033@N04/5020799468