MongoDB Europe 2016 - Debugging MongoDB Performance

Asya Kamsky, Lead Product Manager MongoDB

Diagnostics and Debugging 3.4

“ I thought of my old teacher Joe Bell, ... of his eerie trick of spotting details. If he were a detective he would surely reduce this ... business to something nearer an exact science. —Arthur Conan Doyle

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.

Understanding The Patient

#MDBW16

Understanding The Patient

Understand the system

Knowledge

Monitor trends over time

Trends

Record all metrics "at rest"

Baseline

Gathering Data “Data! Data! Data! I can't make bricks without clay.”

-- Sherlock Holmes, The Adventure of the Copper Beeches

“There is nothing like first-hand evidence.” � -- Sherlock Holmes, A Study in Scarlet

Available Tools

#MDBW16

Available Tools: Say "AH"

#MDBW16

> db.isMaster( ) > rs.conf( ) > rs.status( ) > sh.status( )

> db.version( ) > db.serverCmdLineOpts( )

#MDBW16

> show dbs > show collections > db.getCollectionNames().forEach(function(c) { printjson(db.getCollection(c).getIndexes()); });

#MDBW16

Available Tools: Compass

#MDBW16

Available Tools: OS

% free % blockdev --report % dmesg % ulimit -a % ifconfig, ip <...>, iptables % iostat % netstat % top; htop % perf % iperf3

#MDBW16

Available Tools: OS for MongoDB % mongostat % mongotop

mongod logs

2014-09-01T15:50:03.184-0004 [conn208] query

data.activity query: { $query: { _updated_at: { $gt: new

Date(1396459946346) }, _to: "_UserA" } }, $orderby:

{ _updated_at: -1 } } planSummary: IXSCAN { _to: 1,

_created_at: -1 } ntoreturn:100 ntoskip:0 nscanned:19692

nscannedObjects:19692 numYields:170 locks(micros) r:

283957 nreturned:65 reslen:24939 342ms

2014-09-01T15:50:03.184-0004 [conn208] query

data.activity query: { $query: { _updated_at: { $gt: new

Date(1396459946346) }, _to: "_UserA" } }, $orderby:

{ _updated_at: -1 } } planSummary: IXSCAN { _to: 1,

_created_at: -1 } ntoreturn:100 ntoskip:0 nscanned:19692

nscannedObjects:19692 numYields:170 locks(micros) r:

283957 nreturned:65 reslen:24939 342ms

mongod logs

2015-05-26T23:33:32.774-0500 I

2015-05-28T12:37:11.440-0500 I

2015-05-28T12:38:35.839-0500 I

D, I, W, E, F

3.0 mongod logs

2015-05-26T23:33:32.774-0500 I COMMAND

2015-05-28T12:37:11.440-0500 I NETWORK

2015-05-28T12:38:35.839-0500 I QUERY

COMMAND, NETWORK, QUERY, REPL, ACCESS, INDEX, JOURNAL, SHARDING, WRITE, -

mongod logs

2015-05-26T23:33:32.774-0500 I COMMAND [conn45] command admin.$cmd command: listDatabases { listDatabases: 1.0 } ntoskip:0 keyUpdates:0 writeConflicts:0 numYields:0 reslen:393 locks:{ Global: { acquireCount: { r: 12 } }, Database: { acquireCount: { r: 6 } } } 321ms

2015-05-28T12:37:11.440-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:48625 #183 (21 connections now open)

2015-05-28T12:38:35.839-0500 I QUERY [conn183] getmore tableau.flights201406 query: { origin_city_market_id: 31703.0 } cursorid:61957110347 ntoreturn:0 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:176 nreturned:22579 reslen:1332181 locks:{ Global: { acquireCount: { r: 354 } }, Database: { acquireCount: { r: 177 } }, Collection: { acquireCount: { r: 177 } } } 114ms

3.0 mongod logs

3.2 mongod logs

2016-06-23T10:34:41.559-0700 I COMMAND [conn26] command test.c

command: find { find: "c", filter: { a: { $elemMatch: { v: { $gte: 3.0, $lt: 4.0 } } } } } planSummary: IXSCAN { a.v: 1.0 }

keysExamined:2 docsExamined:1 fromMultiPlanner:1 cursorExhausted:1

keyUpdates:0 writeConflicts:0 numYields:1 nreturned:1 reslen:187

locks:{ Global: { acquireCount: { r: 4 } }, Database: { acquireCount: { r: 2 } }, Collection: { acquireCount: { r: 2 } } } protocol:op_command 110ms

3.2 mongod logs

2016-06-23T10:34:41.559-0700 I COMMAND [conn26] command test.c

command: find { find: "c", filter: { a: { $elemMatch: { v: { $gte: 3.0, $lt: 4.0 } } } } } planSummary: IXSCAN { a.v: 1.0 }

keysExamined:2 docsExamined:1 fromMultiPlanner:1 cursorExhausted:1

keyUpdates:0 writeConflicts:0 numYields:1 nreturned:1 reslen:187

locks:{ Global: { acquireCount: { r: 4 } }, Database: { acquireCount: { r: 2 } }, Collection: { acquireCount: { r: 2 } } } protocol:op_command 110ms

3.2 mongod logs

3.4 mongod logs

2016-06-01T15:30:04.373-0700 I COMMAND [conn99] command socialite.following command: aggregate { aggregate: "following", pipeline: [ { $match: { _f: "45705" } }, { $group: { _id: null, followees: { $addToSet: "$_t" } } }, { $lookup: { from: "following", localField: "followees", foreignField: "_f", as: "fofollowees" } }, { $project: { fofs: { $setUnion: [ "$followees", "$fofollowees._t" ] } } } ] } planSummary: IXSCAN { _f: 1, _t: 1 } keysExamined:1 docsExamined:0 numYields:11 reslen:316214 locks:{ Global: { acquireCount: { r: 3800 } }, Database: { acquireCount: { r: 1900 } }, Collection: { acquireCount: { r: 1900 } } } protocol:op_query 218ms

3.4 mongod logs

2016-06-25T23:38:27.346-0500 I WRITE [conn128] update ycsb.usertable query: { _id: "user7074965863272626663" } planSummary: IDHACK update: { $set: { field1: BinData(0, 23...) } } keysExamined:1 docsExamined:1 nMatched:1 nModified:1 numYields:1 locks:{ Global: { acquireCount: { r: 3, w: 3 } }, Database: { acquireCount: { w: 3 } }, Collection: { acquireCount: { w: 2 } }, Metadata: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 11ms

> db.getLogComponents()

> db.getLogComponents(){ "verbosity" : 1,

"accessControl" : { "verbosity" : -1},

"command" : { "verbosity" : -1},"control" : { "verbosity" : -1},

"geo" : { "verbosity" : -1},

"index" : { "verbosity" : -1},"network" : { "verbosity" : -1},

"query" : { "verbosity" : -1},

"replication" : { "verbosity" : -1},"sharding" : { "verbosity" : -1},

"storage" : { "verbosity" : -1,

"journal" : { "verbosity" : -1}},"write" : { "verbosity" : -1}

> db.getLogComponents(){ "verbosity" : 1,

"accessControl" : { "verbosity" : -1},

"command" : { "verbosity" : -1},"control" : { "verbosity" : -1},

"executor" : { "verbosity" : -1},

"geo" : { "verbosity" : -1},"index" : { "verbosity" : -1},

"network" : { "verbosity" : -1},

"query" : { "verbosity" : -1},"replication" : { "verbosity" : -1},

"sharding" : { "verbosity" : -1},

"storage" : { "verbosity" : -1,"journal" : { "verbosity" : -1}},

"write" : { "verbosity" : -1},

"ftdc" : { "verbosity" : -1}}

> db.setLogLevel( logLevel, component )

> db.setLogLevel( 1, "sharding" )

> db.setLogLevel( 2, "query" )

"It is of the highest importance ... to be able to recognize, �out of a number of facts, which are incidental and which vital. " �� Sherlock Holmes, The Reigate Puzzle

https://github.com/rueckstiess/mtools

by Thomas Rückstieß

https://github.com/rueckstiess/mtools

by Thomas Rückstieß

#MDBW16

mplotqueries $ mplotqueries --help usage: mplotqueries [OPTIONS] [logfile [logfile ...]] A script to plot various information from logfiles. ...

#MDBW16

optional arguments:

--type {nscanned/n,rsstate,connchurn,durline,histogram,range,scatter,event} type of plot (default=scatter with --yaxis duration)

#MDBW16

optional arguments:

--type {nscanned/n,rsstate,connchurn,durline,histogram,range,scatter,event} type of plot (default=scatter with --yaxis duration) --group GROUP specify value to group on. All basic plot types can group on 'namespace', 'operation', 'thread', 'pattern' ...

"... what is out of the common is usually a guide� rather than a hindrance." � — Sherlock Holmes, A Study in Scarlet

mplotqueries

% mplotqueries firstmongo.log --type nscanned/n

% mplotqueries firstmongo.log --type nscanned/n 2014-06-17T17:13:34.235 [conn1569841] query db1.coll query: { time: { $lt: "2014-06-17 17:13:31", $gte: "2014-06-17 17:04:31" } } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:5169727 nscannedObjects:5169727 keyUpdates:0 numYields:12492 locks(micros) r:37736571 nreturned:72 reslen:32707 30129ms

mplotqueries

% mplotqueries firstmongo.log --type nscanned/n 2014-06-17T17:13:34.235 [conn1569841] query db1.coll query: { time: { $lt: "2014-06-17 17:13:31", $gte: "2014-06-17 17:04:31" } } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:5169727 nscannedObjects:5169727 keyUpdates:0 numYields:12492 locks(micros) r:37736571 nreturned:72 reslen:32707 30129ms 2014-06-17T17:13:47.607 [conn1569990] query db2.coll query: { time: { $lt: "2014-06-17 17:14:05", $gte: "2014-06-17 17:05:05" }, status: 8 } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:2057564 nscannedObjects:2057564 keyUpdates:0 numYields:5008 locks(micros) r:11557172 nreturned:56 reslen:18745 13086ms

mplotqueries

% mplotqueries updates?.log

"Eliminate all other factors, and the one which remains must be the truth." Sherlock Holmes -The Sign of Four

#MDBW16

Available Tools

% mongostat

#MDBW16

mongostat

#MDBW16

--discover

mongostat

#MDBW16

mongostat insert query update delete getmore command % dirty % used flushes vsize res faults qr|qw ar|aw netIn netOut conn ReplSetName role ts

#MDBW16

Trends

#MDBW16

Available Tools

#MDBW16

Available Tools: in the Cloud

#MDBW16

% mplotqueries –type connchurn

#MDBW16

db.currentOp()

#MDBW16

db.currentOp()

> db.currentOp({secs_running:{$gt:10}})

{ "desc" : "conn3482",

"threadId" : "140512575297280",

"connectionId" : 3482,

"client_s" : "10.0.149.179:55295",

"active" : true,

"opid" : "SocialiteAtlas-shard-2:55547103",

"secs_running" : 13,

"microsecs_running" : NumberLong(13483812),

"op" : "query",

"ns" : "db1.coll1",

"numYields" : 48815,

"query" : {

"field1" : 5,

#MDBW16

db.currentOp()

> db.currentOp({desc:/^conn/,secs_running:{$gt:0}}).inprog.forEach(function(op) { print(op.opid, op.op, op.ns, op.secs_running); });

SocialiteAtlas-shard-0:68479014 query socialite.content 3

db.killOp ( opid )

#MDBW16

Title Slide Option 1 Click here to add speaker name and title

Title Slide Option 2 Click here to add speaker name and title

This is divider slide option 2

#MDBW16

Click to add title: keep it to 56 characters w/spaces

This is a typical content slide with full width body.

•  First level bullet list •  Second level bullet list

•  Third level bullet list

#MDBW16

Two content

Click to add text.

#MDBW16

Left content

Click to add text.

#MDBW16

Left content

Click to add text.

#MDBW16

Pie Chart

1st Qtr 2nd Qtr 3rd Qtr

1st Quarter Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique .

2nd Quarter Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique .

3rd Quarter Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique .

#MDBW16

Bar Graph

Category 1 Category 2 Category 3 Category 4

Chart Title

Series 1 Series 2 Series 3

#MDBW16

Column Header 1 Column Header 2 Column Header 3 Column Header 4 Column Header 5

Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet

#MDBW16

Title only

#MDBW16

Coding Example – Light Background

// Retrieve var MongoClient = require('mongodb').MongoClient;

// Connect to the db

MongoClient.connect("mongodb://localhost:27017/exampleDb", function(err, db) {

if(err) { return console.dir(err); }

db.collection('test', function(err, collection) {});

db.collection('test', {w:1}, function(err, collection) {});

db.createCollection('test', function(err, collection) {});

db.createCollection('test', {w:1}, function(err, collection) {});

#MDBW16

Coding Example – Dark Background

// Retrieve var MongoClient = require('mongodb').MongoClient;

// Connect to the db

MongoClient.connect("mongodb://localhost:27017/exampleDb", function(err, db) {

if(err) { return console.dir(err); }

db.collection('test', function(err, collection) {});

db.collection('test', {w:1}, function(err, collection) {});

db.createCollection('test', function(err, collection) {});

db.createCollection('test', {w:1}, function(err, collection) {});

#MDBW16

List slide – can also be used for agenda

Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique sit elit.

Lorem Ipsum 01 Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique sit elit.

Lorem Ipsum 02

Lorem Ipsum 05

#MDBW16

Columns and icons with copy (option 1)

Lorem ipsum dolor sit amet, onsectetur

adipiscing elit. Praesent sodales odio sit amet odio tristique.

Linked Lorem ipsum dolor sit

amet, onsectetur adipiscing elit.

Praesent sodales odio sit amet odio tristique.

Planning Lorem ipsum dolor sit

Writing Lorem ipsum dolor sit

Research

#MDBW16

Columns and icons with copy (option 2)

Lorem ipsum dolor sit amet, onsectetur

adipiscing elit. Praesent sodales odio

sit amet odio tristique .

Linked Lorem ipsum dolor sit

Praesent sodales odio sit amet odio tristique .

Planning Lorem ipsum dolor sit

Writing Lorem ipsum dolor sit

Research

#MDBW16

Timeline or progress

“ Quote sample. Lorem ipsum dolor sit amet, onsectetur adipiscing elit amet sodales. Praesent sodales odio sit amet odio tristique. Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique. Lorem ipsum dolor sit amet, onsectetur adipiscing elit.”

MongoDB Europe 2016 - Debugging MongoDB Performance

Data & Analytics

Android Platform Debugging and Development · 1 Android Platform Debugging and Development Embedded Linux Conference Europe 2013 Karim Yaghmour @karimyaghmour karim.yaghmour@opersys.com

Mongodb debugging-performance-problems

Efficient JTAG-based Linux kernel debugging JTAG-based Linux kernel debugging Embedded Linux Conference Europe - 2011 . ADCS#: ##### Revised: 23 September, 2010 Rationale Embedded

MongoDB Europe 2016 - Using MongoDB to Build a Fast and Scalable Content Repository

MongoDB: What, why, when. Solutions Architect, MongoDB Inc. Massimo Brignoli #mongodb

MongoDB Europe 2016 - Advanced MongoDB Aggregation Pipelines

Debugging Ruby with MongoDB

MongoDB Europe 2016 - ETL for Pros – Getting Data Into MongoDB The Right Way

MongoDB World 2016: MongoDB + Google Cloud

MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company

Debugging of Embedded IoTSystems with a Multi …...Debugging of Embedded IoTSystems with a Multi-Domain Oscilloscope Leonardo Nanetti, BDM Oscilloscopes – South Europe Guido Schulze,

Powering Microservices with MongoDB, Docker, Kubernetes & Kafka – MongoDB Europe 2016

MongoDB Europe 2016 - The Rise of the Data Lake

Debugging Effectively - DrupalCon Europe 2016

MongoDB in use(김인범, mongodb korea)

Linux Kernel Debugging - Univerzita Karlova · Linux Kernel Debugging. Advanced Operating Systems 2018/2019 Debugging 2 Agenda – Debugging Scenarios Debugging during individual

MongoDB Performance Debugging

MongoDB Europe 2016 - Choosing Between 100 Billion Travel Options – Instant Search From Amadeus

MongoDB Europe 2016 - MongoDB Atlas

Chris (I’m not a ghost) Woods. What went right (with Mongo) Application design Application development and system migration Application debugging MongoDB