158

Dropping ACID with MongoDB

Embed Size (px)

DESCRIPTION

http://yapc2010.com/yn2010/talk/2578

Citation preview

Page 1: Dropping ACID with MongoDB
Page 2: Dropping ACID with MongoDB

Who am I?

Page 3: Dropping ACID with MongoDB

what is ?

Page 4: Dropping ACID with MongoDB

Application

Perl

Linux

Apache

Page 5: Dropping ACID with MongoDB

Application

Perl

Linux

DatabaseApache

Page 6: Dropping ACID with MongoDB

Application

Perl

Windows

Apache

Page 7: Dropping ACID with MongoDB

Application

Perl

Linux

Apache

Page 8: Dropping ACID with MongoDB

Application

Perl

Linux

Apache

Page 9: Dropping ACID with MongoDB

Application

Perl

Linux

Apache

Page 10: Dropping ACID with MongoDB

Application

Perl

Linux

Apache

The world's mostpopular open source

database

Page 11: Dropping ACID with MongoDB

Application

Perl

Linux

Apache

The world's mostadvanced open source

database

Page 12: Dropping ACID with MongoDB
Page 13: Dropping ACID with MongoDB

databases

contain tables

contain rows

Page 14: Dropping ACID with MongoDB

databases

contain tables

contain rows

schema

Page 15: Dropping ACID with MongoDB

databases

contain tables

contain rows

joins

schema

Page 16: Dropping ACID with MongoDB

databases

contain tables

contain rows

joins

schema

Page 17: Dropping ACID with MongoDB
Page 18: Dropping ACID with MongoDB
Page 19: Dropping ACID with MongoDB
Page 20: Dropping ACID with MongoDB
Page 21: Dropping ACID with MongoDB

Application

Perl

Linux

Apache

Page 22: Dropping ACID with MongoDB

getting started

Page 23: Dropping ACID with MongoDB

www.mongodb.org

Page 24: Dropping ACID with MongoDB

cpan -i MongoDB

Page 25: Dropping ACID with MongoDB

$conn = MongoDB::Connection->new;

Page 26: Dropping ACID with MongoDB

$conn = MongoDB::Connection->new;$db = $conn->foo;

Page 27: Dropping ACID with MongoDB

$conn = MongoDB::Connection->new;$db = $conn->foo;

NO ADMINISTRATION

Page 28: Dropping ACID with MongoDB

$conn = MongoDB::Connection->new;$db = $conn->foo;$table = $db->bar;

Page 29: Dropping ACID with MongoDB

$conn = MongoDB::Connection->new;$db = $conn->foo;$table = $db->bar;

Page 30: Dropping ACID with MongoDB

server

Page 31: Dropping ACID with MongoDB

databases

server

Page 32: Dropping ACID with MongoDB

databases

contain tables

server

Page 33: Dropping ACID with MongoDB

databases

contain tables

server

Page 34: Dropping ACID with MongoDB

databases

contain tables

server

collections

Page 35: Dropping ACID with MongoDB

databases

contain rows

server

contain tables collections

Page 36: Dropping ACID with MongoDB

databases

contain rows

server

contain tables collections

Page 37: Dropping ACID with MongoDB

databases

contain rows

server

contain tables collections

documents

Page 38: Dropping ACID with MongoDB

databases

contain rows

server

contain tables collections

documents

schema

Page 39: Dropping ACID with MongoDB

databases

contain rows

server

contain tables collections

documents

schema

joins

Page 40: Dropping ACID with MongoDB

databases

contain collections

contain documents

server

Page 41: Dropping ACID with MongoDB

$conn = MongoDB::Connection->new;$db = $conn->foo;$collection = $db->bar;

Page 42: Dropping ACID with MongoDB

$conn = MongoDB::Connection->new;$db = $conn->foo;$collection = $db->bar;

NO ADMINISTRATION

Page 43: Dropping ACID with MongoDB

$conn = MongoDB::Connection->new;$db = $conn->foo$collection = $db->bar;

$doc = {"name" => "kristina", "contact info" => { "twitter" => "@kchodorow", "email" => "[email protected]" }, "friends" => 400232, "pic" => \$file_str, "member since" => DateTime->now};

Page 44: Dropping ACID with MongoDB

$conn = MongoDB::Connection->new;$db = $conn->foo$collection = $db->bar;

$doc = {"name" => "kristina", "contact info" => { "twitter" => "@kchodorow", "email" => "[email protected]" }, "friends" => 400232, "pic" => \$file_str, "member since" => DateTime->now};

$collection->insert($doc);

Page 45: Dropping ACID with MongoDB

mongo is type-sensitive and type-rich

Page 46: Dropping ACID with MongoDB

•null•boolean•integer•double•string•array•hash

•qr/fobar/i•\$bindata•DateTime•MongoDB::OID•MongoDB::Code•MongoDB::MinKey•MongoDB::MaxKey

Page 47: Dropping ACID with MongoDB

$collection->find_one({ "name" => "kristina"});

Page 48: Dropping ACID with MongoDB

$collection->find_one({ "name" => "kristina"});

$collection->find_one({ "contact.twitter" => "@kchodorow"});

Page 49: Dropping ACID with MongoDB

$collection->find_one({ "name" => "kristina"});

$collection->find_one({ "contact.twitter" => "@kchodorow"});

$collection->find({ "member since" => { '$gt' => $yesterday, '$lt' => $now}});

Page 50: Dropping ACID with MongoDB

$collection->find_one({ "name" => "kristina"});

$collection->find_one({ "contact.twitter" => "@kchodorow"});

$collection->find({ "member since" => { '$gt' => $yesterday, '$lt' => $now}})->sort({"friends" => 1});

Page 51: Dropping ACID with MongoDB

$collection->find_one({ "name" => "kristina"});

$collection->find_one({ "contact.twitter" => "@kchodorow"});

$collection->find({ "member since" => { '$gt' => $yesterday, '$lt' => $now}})->sort({"friends" => 1})->limit(10);

Page 52: Dropping ACID with MongoDB

$collection->find_one({ "name" => "kristina"});

$collection->find_one({ "contact.twitter" => "@kchodorow"});

$collection->find({ "member since" => { '$gt' => $yesterday, '$lt' => $now}})->sort({"friends" => 1})->limit(10) ->skip(100);

Page 53: Dropping ACID with MongoDB

$collection->find_one({ "name" => "kristina"});

$collection->find_one({ "contact.twitter" => "@kchodorow"});

$cursor = $collection->find({ "member since" => { '$gt' => $yesterday, '$lt' => $now}})->sort({"friends" => 1})->limit(10) ->skip(100);

Page 54: Dropping ACID with MongoDB

while (my $doc = $cursor->next) { # ... }

my @results = $cursor->all;

Page 55: Dropping ACID with MongoDB

x < 3

{"x" => {'$lt' => 3}}

Page 56: Dropping ACID with MongoDB

$gt$lte$ne$mod

$exists$type...

Page 57: Dropping ACID with MongoDB

$collection->update( {"name" => "kristina"},

Page 58: Dropping ACID with MongoDB

$collection->update( {"name" => "kristina"}, {'$set' =>

Page 59: Dropping ACID with MongoDB

$collection->update( {"name" => "kristina"}, {'$set' => {"contact.website" => "snailinaturtleneck.com"}});

Page 60: Dropping ACID with MongoDB

updates are...•atomic•in-place (when possible)•able to create and delete fields•able to create documents

Page 61: Dropping ACID with MongoDB

$collection->update( {"url" => "example.com"}, {'$inc' => {"pageviews" => 1}}, {"upsert" => 1});

Page 62: Dropping ACID with MongoDB
Page 63: Dropping ACID with MongoDB
Page 64: Dropping ACID with MongoDB

$set$inc$push$pop$pull...

Page 65: Dropping ACID with MongoDB

$set$inc$push$pop$pull...

Page 66: Dropping ACID with MongoDB

$set$inc$push$pop$pull...

Page 67: Dropping ACID with MongoDB

$set$inc$push$pop$pull...

Page 68: Dropping ACID with MongoDB

"$gte"

Page 69: Dropping ACID with MongoDB

"<null>"

Page 70: Dropping ACID with MongoDB

""

Page 71: Dropping ACID with MongoDB

'$gte' or "\$gte"

Page 72: Dropping ACID with MongoDB

Or define your own!

":gte""=gte""?gte"

Page 73: Dropping ACID with MongoDB

Or define your own!

" gte"

Page 74: Dropping ACID with MongoDB

Or define your own!

" gte"" ♕ gte"

Page 75: Dropping ACID with MongoDB

Or define your own!

" gte"" ♕ gte""xgte"

Page 76: Dropping ACID with MongoDB

features

Page 77: Dropping ACID with MongoDB

I suddenly have a deep love for stored procedures

Page 78: Dropping ACID with MongoDB

I suddenly have a deep love for stored procedures

now with stored

procedures!

Page 79: Dropping ACID with MongoDB

$db->system->js->insert({ "_id" => "x", "value" => 3});

$db->system->js->insert({ "_id" => "y", "value" => 4});

$db->eval("return x+y");

Page 80: Dropping ACID with MongoDB

my $log = <<<LOGfunction(msg, level) { var date = "[" + new Date() + "] "; var lvl = level ? level+" " : ""; print(date + lvl + msg);}LOG;

$db->system->js->insert({ "_id" => "log", "value" => $log});

Page 81: Dropping ACID with MongoDB

$db->eval("log('all your base are belong to us', 'FATAL');");

Page 82: Dropping ACID with MongoDB

$db->eval("log('all your base are belong to us', 'FATAL');");

[Fri May 19 18:34:57] FATAL all your base are belong to us

Page 83: Dropping ACID with MongoDB
Page 84: Dropping ACID with MongoDB
Page 85: Dropping ACID with MongoDB

J J J

J J J

J J J

Page 86: Dropping ACID with MongoDB
Page 87: Dropping ACID with MongoDB

$metadata = { "_id" => " " "date" => DateTime->now "downloads" => 0, "user" => $userId, "filename" => "rickroll.flv"}

J

Page 88: Dropping ACID with MongoDB

$grid = $db->get_gridfs;

$grid->put($fh, $metadata);

Page 89: Dropping ACID with MongoDB

$grid = $db->get_gridfs;

$grid->put($fh, $metadata);

$file = $grid->find_one({ "filename" => qr/rickroll/i});

Page 90: Dropping ACID with MongoDB

$grid = $db->get_gridfs;

$grid->put($fh, $metadata);

$file = $grid->find_one({ "filename" => qr/rickroll/i});

$file->print($fh2);

Page 91: Dropping ACID with MongoDB

Benefits:• Automatic backups• Unlimited metadata• Millions of files per directory• Random access• Better disk locality

Page 92: Dropping ACID with MongoDB

Geospatial Indexes

Page 93: Dropping ACID with MongoDB
Page 94: Dropping ACID with MongoDB

$coll->insert({location => (-40, 78)});

Page 95: Dropping ACID with MongoDB

$coll->insert({location => (-40, 78)});

$coll->find({location => { '$near' => (-40, 77)}})

Page 96: Dropping ACID with MongoDB

$coll->insert({location => (-40, 78)});

$coll->find({location => { '$near' => (-40, 77)}})->limit(10)

Page 97: Dropping ACID with MongoDB

what doesn't it have?

Page 98: Dropping ACID with MongoDB

transactions

Page 99: Dropping ACID with MongoDB

are you practicing safedata storage?

Page 100: Dropping ACID with MongoDB
Page 101: Dropping ACID with MongoDB

Insert this.

Okay, got it.

Phew, my data's safe.

Page 102: Dropping ACID with MongoDB

Write this to disk

All over it!

I know better than he does, I'll just let this sit in a buffer for a while.

Page 103: Dropping ACID with MongoDB
Page 104: Dropping ACID with MongoDB

? I have no idea what you're talking about.

Page 105: Dropping ACID with MongoDB

trust no one!

...trust a bunch of ones. Mostly.

Page 106: Dropping ACID with MongoDB

prod1.example.com

prod2.example.com

Page 107: Dropping ACID with MongoDB

prod1.example.com

prod2.example.com

Page 108: Dropping ACID with MongoDB

prod1.example.com

prod2.example.com

Page 109: Dropping ACID with MongoDB

prod1.example.com

prod2.example.com

Page 110: Dropping ACID with MongoDB
Page 111: Dropping ACID with MongoDB

prod1.example.com

prod2.example.com

Page 112: Dropping ACID with MongoDB

prod1.example.com

prod2.example.com

Page 113: Dropping ACID with MongoDB

prod1.example.com ?

prod2.example.com

Page 114: Dropping ACID with MongoDB

prod2.example.com

prod1.example.com

Page 115: Dropping ACID with MongoDB

prod2.example.com

prod1.example.com

?

Page 116: Dropping ACID with MongoDB

prod2.example.com

prod1.example.com

Page 117: Dropping ACID with MongoDB

prod2.example.com

prod1.example.com

Page 118: Dropping ACID with MongoDB

primary

secondary

passive

secondary

Page 119: Dropping ACID with MongoDB
Page 120: Dropping ACID with MongoDB
Page 121: Dropping ACID with MongoDB
Page 122: Dropping ACID with MongoDB
Page 123: Dropping ACID with MongoDB

okay

Page 124: Dropping ACID with MongoDB

?

Page 125: Dropping ACID with MongoDB

make sure two slaves have this

Page 126: Dropping ACID with MongoDB
Page 127: Dropping ACID with MongoDB
Page 128: Dropping ACID with MongoDB
Page 129: Dropping ACID with MongoDB

all set

Page 130: Dropping ACID with MongoDB
Page 131: Dropping ACID with MongoDB
Page 132: Dropping ACID with MongoDB

EXCEPTION

Page 133: Dropping ACID with MongoDB

EXCEPTION

Page 134: Dropping ACID with MongoDB

EXCE

PTIO

N

Page 135: Dropping ACID with MongoDB
Page 136: Dropping ACID with MongoDB

$conn->w(3);

$collection->insert({"x" => 1}, {"safe" => 1});

Page 137: Dropping ACID with MongoDB

scaling

Page 138: Dropping ACID with MongoDB
Page 139: Dropping ACID with MongoDB

sometimes read-only

Page 140: Dropping ACID with MongoDB

sometimes read-onlysometimes inconsistent

Page 141: Dropping ACID with MongoDB

U - ZK - OA - E F - J P - T

Page 142: Dropping ACID with MongoDB

U - ZK - OA - E F - J P - T

Page 143: Dropping ACID with MongoDB

U - ZK - OA - E F - J P - T

Page 144: Dropping ACID with MongoDB
Page 145: Dropping ACID with MongoDB

config

mongod mongod mongod

mongos

Page 146: Dropping ACID with MongoDB

mongos

mongod mongod mongod

I want posts from April 2009 - June 2009

Page 147: Dropping ACID with MongoDB

mongos

mongod mongod mongod

April and May is on #2. June is on #3.

Page 148: Dropping ACID with MongoDB

mongos

mongod mongod mongod

Page 149: Dropping ACID with MongoDB

mongos

mongod mongod mongod

Page 150: Dropping ACID with MongoDB

mongos

mongod mongod mongod

Add this post for March 26, 2010.

Page 151: Dropping ACID with MongoDB

mongos

mongod mongod mongod

#3 hold June 2009-present

Page 152: Dropping ACID with MongoDB

config

mongod mongod mongod

mongos

So, what's going on?

Page 153: Dropping ACID with MongoDB

config

mongod mongod mongod

mongos

Page 154: Dropping ACID with MongoDB

$conn = MongoDB::Connection->new;# ...

Page 155: Dropping ACID with MongoDB
Page 156: Dropping ACID with MongoDB

try.mongodb.org

Page 157: Dropping ACID with MongoDB

www.mongodb.org

Page 158: Dropping ACID with MongoDB

thank you!

@kchodorow

http://www.snailinaturtleneck.com