mTools - A must have for MongoDB

Maintain an application using MongoDB can be painful, especially when you either have a large amount of data, or a lot of writing operations. This is even more true if, as a developer, you don't have access to production servers, oftenly allowed to few.

Here is the problem : how to find in your huge amount of data or in your daily requests, which one has a negative impact on your performances and much more, find bottlenecks.

Giving the help of MongoDB Inc. on Numeric Editions subjects at FranceTV, we automated the use of tools, in order to study behaviours of our production servers without incidences on our applications.

The tool, or rather the toolbox we most oftenly use is MTools. This project was started and is still maintained by Thomas Rückstieß, who worked for MongoDB.

MTools : 6 tools in one

Mloginfo

Mloginfo reads logs generated by MongoDB and returns informations about the usage of the database. In our case, we wanted to monitor and profile our request to define the ones which will be take a long amount of time and the ones which will be used the most.

Here is an example of use:

mloginfo --queries logs.mongo/mongod.log

namespace              operation        pattern                              count     min (ms)    max (ms)    mean (ms)    95%-ile (ms)    sum (ms)

myCol.$cmd               findandmodify    {"ID": 1}                          916493         101       10486          277           453.0    254423325
myCol.User               count            {"activeSeg": 1, "updatedAt": 1}   68           30135     1413998       419353       1350776.4    28516024
myCol.InactiveUser       count            {"inactiveSeg": 1}                 32          625038     1019698       813384        984999.6    26028315

About columns:

• namespace: which database and collection is used;
• operation: the operation being executed (find, update, remove, etc.);
• pattern: the filter used on the request. On the first row for example, ID is the field used to filter the findAndModify operation, and could have any values(3, 42, etc...);
• count: the total amount of similar requests in log file;
• min, max, mean, 95%-ile: the time it takes to execute a request;
• sum: cumulated time for this request (in milliseconds).

In this example, we can see that findandmodify operation is extremly used in an acceptable amount of time. However, simple count operations are taking a huge amount and time, which can be due to a missing index.

More info here.

Mlogfilter

Its name says all, Mlogfilter allows you to filter logs. You can easily apply multiple filters and maybe combine it with mloginfo.

cat logs.mongo/mongod.log | mlogfilter --human --slow --from start +1day

Fri Sep 16 08:01:58.883 I COMMAND [conn195591] command dbm_rcu_v2.Client command:
aggregate { aggregate: "Client", pipeline: [ { $match: { somedate:
{ $lte: new Date(1474005604000), $gte: new Date(1469951542000) } } }, { $sort:
{ somedate: -1 } },{ $unwind: "$someArray" }, { $skip: 165000 },
{ $limit: 5000 } ], allowDiskUse: true }
ntoskip:0 keyUpdates:0 writeConflicts:0 numYields:549 reslen:862557
locks:{ Global: { acquireCount: { r: 1120 } },
Database: { acquireCount: { r: 560 } }, Collection: { acquireCount: { r: 560 } } }
protocol:op_query (0hr 0min 3secs 99ms) 3,099ms

This command for example show only the slower operations within a day from the beginning of the log file. As we can see, the output gives us an aggregate of commands which took more than three seconds to execute.

More info here.

Mplotqueries & Mlogvis

Those tools allows you to generate graphics to visualize data in a human-readable format. We can see : calls distribution, type of commands, etc :

More info on Mlogvis & Mplotqueries.

Mgenerate

Mgenerate is a tool that can randomly fill a MongoDB database given a JSON model. It is a must have when it comes to test functions or requests behavior with a large set of data.

This is an example of JSON file used to generate an user collection :

{
    "user": {
        "name": {
            "first": {"$choose": ["Liam", "Noah", "Ethan", "Mason", "Logan", "Jacob", "Lucas", "Jackson", "Aiden", "Jack", "James", "Elijah", "Luke", "William", "Michael", "Alexander", "Oliver", "Owen", "Daniel", "Gabriel", "Henry", "Matthew", "Carter", "Ryan", "Wyatt", "Andrew", "Connor", "Caleb", "Jayden", "Nathan", "Dylan", "Isaac", "Hunter", "Joshua", "Landon", "Samuel", "David", "Sebastian", "Olivia", "Emma", "Sophia", "Ava", "Isabella", "Mia", "Charlotte", "Emily", "Abigail", "Avery", "Harper", "Ella", "Madison", "Amelie", "Lily", "Chloe", "Sofia", "Evelyn", "Hannah", "Addison", "Grace", "Aubrey", "Zoey", "Aria", "Ellie", "Natalie", "Zoe", "Audrey", "Elizabeth", "Scarlett", "Layla", "Victoria", "Brooklyn", "Lucy", "Lillian", "Claire", "Nora", "Riley", "Leah"] },
            "last": {"$choose": ["Smith", "Jones", "Williams", "Brown", "Taylor", "Davies", "Wilson", "Evans", "Thomas", "Johnson", "Roberts", "Walker", "Wright", "Robinson", "Thompson", "White", "Hughes", "Edwards", "Green", "Hall", "Wood", "Harris", "Lewis", "Martin", "Jackson", "Clarke", "Clark", "Turner", "Hill", "Scott", "Cooper", "Morris", "Ward", "Moore", "King", "Watson", "Baker" , "Harrison", "Morgan", "Patel", "Young", "Allen", "Mitchell", "James", "Anderson", "Phillips", "Lee", "Bell", "Parker", "Davis"] }
        },
        "gender": {"$choose": ["female", "male"]},
        "age": "$number",
        "address": {
            "street": {"$string": {"length": 10}},
            "house_no": "$number",
            "zip_code": {"$number": [10000, 99999]},
            "city": {"$choose": ["Manhattan", "Brooklyn", "New Jersey", "Queens", "Bronx"]}
        },
        "phone_no": { "$missing" : { "percent" : 30, "ifnot" : {"$number": [1000000000, 9999999999]} } },
        "created_at": {"$date": ["2010-01-01", "2014-07-24"] },
        "is_active": {"$choose": [true, false]}
    },
    "tags": {"$array": {"of": {"label": "$string", "id": "$oid", "subtags":
        {"$missing": {"percent": 80, "ifnot": {"$array": ["$string", {"$number": [2, 5]}]}}}}, "number": {"$number": [0, 10] }}}
}

More info on Mgenerate.

Mlaunch

Mlaunch on the other hand let you create quickly a local workable environment for MongoDB. It can provide a configuration for a standalone usage or for replicas or/and shards. Combined with Mgenerate, it's an efficient and a quick way to setup test environments in order to test apps using MongoDB.

Example :

mlaunch --replicaset --nodes 5

This command is creating a MongoDB server with 5 replicas.

Bonus: MongoDB Compass

Compass is a desktop client allowing you to browse and analyze data from a MongoDB database. Its goal is to allow a person to manipulate data without real knowledges on MongoDB querying. However, it's only available on Windows and MacOS yet.