Embed Size (px)
Citation preview
MongoDBAggregationFrameworkin action !
MongoDB User Group - Nantesmardi 20 janvier 2015
@sebprunier
/me
Sébastien Prunier
Développeur chez SERLI
@sebprunier
http://blog.sebprunier.com
Sondage (1/2)
?
Sondage (2/2)
?Aggregation Framework
MongoDB Aggregation Framework
Opérations simples
Map-Reduce
Aggregation Pipeline
Opérations simple
db.collection.count(filter?)
db.collection.distinct(attribute)
db.collection.group(query)
Map-Reduce
db.collection.mapreduce( mapFunction, reduceFunction, options)
Aggregation Pipeline
db.collection.aggregate( [ stage1, stage2, ..., stageN ])
Aggregation Pipeline
json
json
json
json
$match
$unwind
$group
Opérateurs vs. SQLSQL Terms, Functions, and Concepts MongoDB Aggregation Operators
WHERE $match
GROUP BY $group
HAVING $match
SELECT $project
ORDER BY $sort
LIMIT $limit
SUM() $sum
COUNT() $sum
join N/A~ $unwind / arrays
Dataset
campings
marvel geeks
Campings
{"_id" : ObjectId("54bd347880d46d750f7a48c2"),"raking_date" : ISODate("2013-06-06T22:00:00Z"),"publication_date" : ISODate("2013-06-06T22:00:00Z"),"typology" : "CAMPING","ranking" : "2 étoiles","category" : "Tourisme","mention" : "-","commercial_name" : "CAMPING LA BERGERIE","address" : "4231 route de Giens","zip_code" : "83400","city" : "HYÈRES","phone" : "494589175","email" : "[email protected]","website" : "www.camping-de-la-bergerie.com","visit_type" : "-","capacity" : "-","number_of_rooms" : "-","number_of_spaces" : "55","number_of_housing_units" : "-","number_of_chambers" : "-"
}data.gouv.fr
Campings
“ Déterminer le nombre de campings pour chaque niveau de classement
(1 étoile, 2 étoiles, etc…) ”
$group
Campings
db.campings.aggregate([
{$group : {_id : "$ranking", total : {$sum : 1}}}
])
{ "_id" : "5 étoiles", "total" : 177 }
{ "_id" : "4 étoiles", "total" : 940 }
{ "_id" : "3 étoiles", "total" : 2224 }
{ "_id" : "2 étoiles", "total" : 1681 }
{ "_id" : "1 étoile", "total" : 389 }
Campings
“ Top 5 des villes avec le plus de campings ”
$group$sort $limit
$project
Campingsdb.campings.aggregate([
{$group : {_id : "$city", total : {$sum : 1}}},
{$sort : {total : -1}},
{$limit : 5},
{$project: {_id: 0, city : "$_id", total: 1}}
])
{ "city" : "ARGELÈS-SUR-MER" , "total" : 29 }
{ "city" : "AGDE", "total" : 23 }
{ "city" : "VIAS", "total" : 20 }
{ "city" : "SAINT-JEAN-DE-MONTS" , "total" : 20 }
{ "city" : "LES MATHES", "total" : 17 }
Campings
“ Nombre de villes avec seulement un camping ”
$group$match
$project
Campingsdb.campings.aggregate([
{$group : {_id : "$city", total : {$sum : 1}}},
{$match : {total : 1}},
{$group: {_id: null, count: {$sum: 1 }}},
{$project: {_id: 0, count: 1}}
])
{ "count" : 2802 }
Marvel
{"_id" : 4,"title" : "Rogue (2004) #5","description" : "...","format" : "Comic","creators" : {
"available" : 6,"items" : [...]
},"characters" : {
"available" : 1,"items" : [
{"name" : "Rogue"
}]
},...
}
developer.marvel.com
Marvel
“ Top 5 des personnages apparaissant dans le plus de bandes dessinées ”
$match $project$unwind$group
$sort $limit
Marvel
“Deconstructs an array field from the input documents to output a document for each element.”
$unwind
{ "name" : "john doe", "tags" : ["A", "B", "C"]}
{ "name" : "john doe", "tags" : "A"}
{ "name" : "john doe", "tags" : "B"}
{ "name" : "john doe", "tags" : "C"}
$unwind
Marveldb.comics.aggregate([
{$match : {"characters.returned" : {$gt : 0}}},
{$project : {title : 1, characters : 1}},
{$unwind : "$characters.items"},
{$group : {_id : "$characters.items.name", total : {$sum : 1}}},
{$sort : {total : -1}},
{$limit : 5}
])
{"_id": "Spider-Man","total": 2413}
{"_id": "X-Men","total": 2320}
{"_id": "Iron Man","total": 1904}
{"_id": "Wolverine","total": 1594}
{"_id": "Captain America" ,"total": 1367}
Marvel
“ Créez la collection des personnages à partir de la collections des bandes
dessinées ”
$match $project$unwind $group
$out
Marvel
db.comics.aggregate([
{$match : {"characters.returned" : {$gt : 0}}},
{$project : {title : 1, characters : 1}},
{$unwind : "$characters.items"},
{$group : {
_id : "$characters.items.name",
total : {$sum : 1},
comics : {$push : {id : "$_id", title : "$title"}}}
},
{$out : "characters"}
])
Marvel
{
"_id": "Frog-Man",
"total": 2,
"comics": [
{
"id": 38126,
"title": "Spider-Man: New York Stories (Trade Paperback)"
},
{
"id": 39753,
"title": "Spider-Island: Avengers (2011) #1"
}
]
}
Geeks
{"_id" : ObjectId("54bd3f1a84c2c169160a88d5"),"nom" : "Mark Zuckerberg","ville" : "Palo Alto","likes" : [
"Facebook","Tongues","PHP"
],"imageUrl" : "static/GIT_HASH/img/geek5.jpg","location" : {
"type" : "Point","coordinates" : [
-122.1430195,37.4418834
]}
}
code-story.net
Geeks
“ Rechercher ce qui rassemble le plus de geeks autour de Paris ”
$geoNear$project $unwind $group $match
$sort
Geeks
db.geeks.aggregate([ { $geoNear: { near: { type: "Point", coordinates: [2.352241, 48.856638] }, distanceField: "distance", spherical: true, maxDistance: 10000 } }, ...])
Geeks
{ "_id" : "java", "total" : 8, "friends" : [ { "nom" : "...", "distance" : 3.015741215085397 }, { "nom" : "...", "distance" : 9864.484291991206 }, ... ]}
Index
● Index utilisé pour certains opérateurs, en début de pipeline○ $match○ $sort○ $geoNear
● explain○ db.collection.aggregate([...],
{explain: true})
Index
● Optimisations
{ $sort: { age : -1 } },
{ $match: { status: 'A' } }
{ $match: { status: 'A' } },
{ $sort: { age : -1 } }
“ Optimizations are subject to change between releases. “
Sharding
● Découpage du pipeline en deux parties○ 1ère partie (jusqu’au premier $group ou $sort)
exécutée sur chaque shard (certains shards peuvent être exclus grâce à un $match)
○ 2ème partie exécutée sur le shard primaire à partir des résultats consolidés de la première partie
The end
Merci pour votre attention !
● Questions / Réponses● Partage d’expérience
