133
Kristina Chodorow MapReduce, Geospatial Indexes, and Other Cool Features

Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

  • Upload
    mongosf

  • View
    16.485

  • Download
    2

Embed Size (px)

Citation preview

Page 1: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

Kristina Chodorow

MapReduce, Geospatial Indexes, and Other Cool Features

Page 2: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 3: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

connecting to your mongod

Page 4: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

connecting to your mongod

you

Page 5: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

connecting to your mongod

you

mongod

Page 6: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

mongod

Page 7: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

mongod

Page 8: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

mongodupdate query insertinsert query

Page 9: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

mongod

driver

Page 10: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

driver

insert

query

Page 11: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

mongod

driver

insert

query

Page 12: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

mongod

driver

insertgetlasterror

Page 13: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

mongod

driver

insertgetlasterror

Page 14: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 15: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

Dwight @ 3:15Farallon Room

replication

Page 16: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 17: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 18: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

insert this

Page 19: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

all set

Page 20: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 21: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 22: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 23: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

insert this

Page 24: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

all set

Page 25: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 26: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

make sure two slaves have this

Page 27: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 28: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 29: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 30: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

all set

Page 31: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 32: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

make sure two slaves have this

Page 33: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 34: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 35: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 36: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 37: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 38: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.runCommand(... getlasterror : 1... )

Page 39: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.runCommand(... getlasterror : 1,... w : 3... )

Page 40: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.runCommand(... getlasterror : 1,... w : 500... )

Page 41: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.runCommand(... getlasterror : 1,... w : 500,... wtimeout : 3000... )

Page 42: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.runCommand(... getlasterror : 1,... fsync : true... )

Page 43: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

some internals

Page 44: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

insertupdateremovequeryget more

Operations:

Page 45: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

insertupdateremovequeryget more

Operations:

Page 46: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

insertupdateremovequeryget more

Operations:

Page 47: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

insertupdateremovequeryget more

Operations:

Page 48: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

insertupdateremovequeryget more

Operations:

Page 49: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

insertupdateremovequeryget more

Operations:

Page 50: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

insertupdateremovequeryget more

Operations:

Page 51: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

get help

Page 52: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

$ mongoMongoDB shell version 1.5.1url: testconnecting to: testtype "help" for help>

Page 53: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

$ mongoMongoDB shell version 1.5.1url: testconnecting to: testtype "help" for help>

Page 54: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.help()DB methods:

db.addUser()db.auth()db.cloneDatabase()db.commandHelp()db.copyDatabase()db.createCollection()db.currentOp()...

Page 55: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.foo.help()DBCollection help

db.foo.count()db.foo.dataSize()db.foo.distinct()db.foo.drop()db.foo.dropIndex()db.foo.dropIndexes()...

Page 56: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> var cursor = db.foo.find()> cursor.help()DBQuery help

.sort()

.limit()

.skip()

.count()

.size()

...

Page 57: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

http://api.mongodb.org/js

Page 58: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.foo.count()40>

Page 59: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.foo.count()40> db.foo.count

Page 60: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.foo.count()40> db.foo.countfunction(x) return this.find(x).count();;

>

Page 61: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.foo.count()40> db.foo.countfunction(x)

return this.find(x).count();;> var cursor = db.foo.find()>

Page 62: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.foo.count()40> db.foo.countfunction(x)

return this.find(x).count();;> var cursor = db.foo.find()> cursor.count

Page 63: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> cursor.countfunction(applySkipLimit) var cmd = count : this._collection.getName();;if (this._query) if (this._special) cmd.query = this._query.query;;

else cmd.query = this._query;;

cmd.fields = this._fields || ;;if (applySkipLimit)

if (this._limit) cmd.limit = this._limit;;

...

Page 64: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

function(x) // prep cmd...

// run commandvar res = this._db.runCommand(cmd);;

// return result...

Page 65: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.runCommand(count : "foo")

"n" : 40,"ok" : 1

>

Page 66: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.runCommand(listCommands : 1)

Page 67: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.runCommand(listCommands : 1)

"commands" : "$eval" : ...,"assertinfo : ...,"authenticate" : ...,"buildinfo" : ...,"clean" : ...,"clone" : ...,"cloneCollection" : ...,"cloneCollectionAsCapped" : ...,"closeAllDatabases" : ...,...

Page 68: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.runCommand(... datasize : collectionName)

Page 69: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.runCommand(... datasize : collectionName)

> db.runCommand(... dbstats : 1)

Page 70: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.commandHelp("distinct")help for: distinct:

distinct : 'collection name',key : 'a.b'

>

Page 71: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.commandHelp("distinct")help for: distinct:

distinct : 'collection name',key : 'a.b'

> db.commandHelpfunction(name)

var c = ;;c[name] = 1;;c.help = true;;return this.runCommand(c).help;;

>

Page 72: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.runCommand("count" : 1, ... "help" : true)

Page 73: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.runCommand("count" : 1, ... "help" : true)help for: count: no help defined>

Page 74: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

$

Page 75: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.blog.posts.find(... _id : 123,... comments : 1)

Page 76: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.blog.posts.find(... _id : 123,... comments : $slice : 10)

Page 77: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.blog.posts.find(... _id : 123,... comments : $slice : [20, 10])

Page 78: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.blog.posts.find(... _id : 123,... comments : $slice : -­10)

Page 79: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.blog.posts.find(... _id : 123,... comments : $slice : [-­20, 10])

Page 80: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.blog.posts.find(... comments : ... date : $gt : today,... votes : $gte : 10... )

Page 81: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.blog.posts.find(... 'comments.date' : $gt : today,... 'comments.votes' : $gte : 10... )

Page 82: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.blog.posts.find(... comments : $elemMatch : ... date : $gt : today,... votes : $gte : 10... )

Page 83: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

Types

Page 84: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

minKey

maxKey

Page 85: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

MapReduce

Page 86: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 87: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 88: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

_id : 123,name : "Fred",email : "[email protected]"

_id : 456,name : "Paul",phone : "555-­5555",picture : BinData

_id : 789,email : "[email protected]",

Page 89: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

_id 40name 32email 20phone 2picture 17pet 3friends 40hobbies 6favorites 12notes 14

Page 90: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 91: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

_id : 123,name : "Fred",email : "[email protected]"

Page 92: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

1

11

Page 93: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

1

1

1

_id

name

email

Page 94: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

email

Page 95: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

email

Page 96: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

12

Page 97: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

12

cluck cluck

Page 98: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

email

12

Page 99: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

map = function() for (var key in this) emit(key, 1);;

Page 100: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

reduce = function(prev, current) var count = 0;;for (var egg in current) count += current[egg];;

return count;;

Page 101: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 102: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 103: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

EB D FA C

Page 104: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

EB D FA C

Page 105: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

_id : C, value :

_id : F, value :

_id : E, value :_id : A, value :

EB D FA C

_id : B, value : _id : D, value :

Page 106: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

GridFS

Page 107: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 108: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

Indexes

Page 109: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 110: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 111: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.users.find(... date : new Date("10/3/2010"))

> db.users.ensureIndex(... date : 1)

Page 112: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

Dates, timestamps, ObjectIds

Page 113: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

Aaron @ 3:15Niantic Room

indexing

Page 114: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

Geospatial Indexes

Page 115: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 116: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.map.ensureIndex(location : "2d")>

Page 117: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.map.insert(location : [-­40, 78])>

Page 118: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.map.insert(location : [-­40, 78])> db.map.insert(location : ... x : -­40, ... y : 78)>

Page 119: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.map.insert(location : [-­40, 78])> db.map.insert(location : ... x : -­40, ... y : 78)> db.map.insert(location : ... longitude : -­40, ... latitude : 78)>

Page 120: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.map.find(location : ... $near : [-­40, 77])

Page 121: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.map.find(location : ... $near : [-­40, 77])-­>limit(10)

Page 122: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

> db.map.ensureIndex(... location : "2d",... user : 1);;

Page 123: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 124: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 125: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

1 degree of longitude at the equator:

Page 126: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

1 degree of longitude at the equator:

1 degree of longitude near the poles:

Page 127: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 128: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 129: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Page 130: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

Eliot @ 10:15Gateway Room

geospatial

Page 131: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

please help document stuff!

http://cookbook.mongodb.org

Page 132: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

want to work on mongodb?

http://www.10gen.com/jobs

Page 133: Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)

thank you!

@kchodorow

http://www.snailinaturtleneck.com