Archive

Archive for the ‘mongodb’ Category

MongoDB: reduce file size

August 23, 2016 Leave a comment

Problem
I have a small Digital Ocean VPS with MongoDB with 20 GB storage. The database dump is 1.7 GB, but the directory /var/lib/mongodb occupies about 7 GB disk space. If it goes on like that, I will run out of free space.

Solution
MongoDB 3 uses the WiredTiger storage engine, but before that the mmapv1 engine was the default. The mmapv1 has two options to reduce file size but WiredTiger doesn’t support these options :(

So I switched back to mmapv1. Here is my /etc/mongod.conf file that enables file size reduction:

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
  engine: mmapv1
  mmapv1:
    preallocDataFiles: false
    smallFiles: true
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1

Notice preallocDataFiles and smallFiles. Keep in mind that they may have negative impact on the performance. In my case I was just collecting a lot of data and not running out of free space was more important.

My new /var/lib/mongodb is 4 GB now, not 7 GB.

To figure out if Mongo uses WiredTiger or mmapv1, execute this in the mongo shell:

> db.serverStatus().storageEngine

MongoDB: connect remotely

August 8, 2016 Leave a comment

Problem
I have a Digital Ocean VPS running MongoDB. There is a web application on this machine that is on port 80. MongoDB is hidden from the outside world and can only be accessed internally. There is also an SSH port where I can log in.

How to connect to my MongoDB server from home? Say I want to use a graphical client, e.g. MongoChef. The client runs on my home machine and I want to connect to MongoDB with it on my DO VPS. How to do that?

Solution
I found the solution here. In short: we connect securely to our database through an SSH tunnel.

Make sure that:

  • you can SSH into your Mongo droplet
  • your MongoDB is bound to localhost

For connecting, I use this script:

REMOTE_SSH_PORT=1234
LOCAL_PORT=2345
REMOTE_MONGO_PORT=27017
cmd="ssh -p ${REMOTE_SSH_PORT} -L ${LOCAL_PORT}:localhost:${REMOTE_MONGO_PORT} user@your.remote.ip"
echo "#" $cmd
echo "# connect on your home machine to port ${LOCAL_PORT}"
echo "# example:    mongo --port ${LOCAL_PORT}"
$cmd

The default SSH port is 22, but it’s a good idea to change it. With the command “ssh -p ${REMOTE_SSH_PORT} user@your.remote.ip” I could log in to my VPS. However, MongoDB was not accessible from outside, thus executing “mongo --host your.remote.ip --port ${REMOTE_MONGO_PORT}” failed.

The SSH tunneling above works as follows. On your home machine you open the port ${LOCAL_PORT} that is connected to your remote machine via the SSH port ${REMOTE_SSH_PORT}, and the connection is tunneled to localhost:${REMOTE_MONGO_PORT}, where localhost means the remote machine where we logged in with SSH.

So, when you execute the script above, you’ll have to log in to your remote machine via SSH. Then open a new terminal and type “mongo --port 2345” and voilá, you are connected to MongoDB on your remote machine!

If you use a Mongo client (e.g. MongoChef), then simply create a new connection and specify localhost with port 2345. Connect, and you are in.

It works as long as you are logged in in a terminal via SSH. When you log out, the local port closes that is tunneled to your remote machine.

Upgrade from Ubuntu 14.04 to 16.04

On my laptop I upgraded my Ubuntu to 16.04 but after the restart I didn’t get the graphical interface. The booting stopped and I found myself in a maintenance shell. I also got messages about some hard disk errors.

Well, I also have Windows 8.1 on this laptop and the Windows didn’t do a complete shutdown. Windows 8 does some tricky shutdown to boost the booting process, so it does some kind of half-hibernation. When I used Ubuntu 14.04, I got a message that one partition couldn’t be mounted, “press S to skip”. And I upgraded to 16.04 like this. And this unmountable partition caused this trouble.

So the solution was the following: start Windows 8.1, disable that shitty “fast startup” (see this post for more info), and restart Ubuntu. This time Ubuntu 16.04 booted normally.

Update (20160526)
I upgraded two more machines, there were no real problems. Libreoffice didn’t want to start on one of them, it dropped a “General I/O” error. Here is the cure:

$ sudo apt-get remove --purge libreoffice*
$ sudo apt install libreoffice

It’s a good idea to use “apt” instead of “apt-get”. It does the same thing but in addition you get a nice progress bar.

MongoDB stopped too. Here is how to start it with systemctl:

$ sudo systemctl enable mongodb
$ sudo systemctl start mongodb
# verification:
$ systemctl status mongodb
$ mongo

[mongodb] make MongoDB available on localhost only

March 15, 2016 Leave a comment

If you want to hide MongoDB from the world and only want to allow connections to it from localhost, then here is a basic config file:

# /etc/mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1

In MongoDB 3.2 WiredTiger is the default storage engine. The interesting part here is the “net” configuration.

I have a Digital Ocean VPS with MongoDB preinstalled and this config file comes from there.

Links

Categories: mongodb Tags: ,

[mongodb] drop all indexes of a database

March 12, 2016 Leave a comment

Problem
You have a database with several collections in it. You want to drop all the indexes of these collections.

Solution

use name_of_your_database;

db.getCollectionNames().forEach(function(collName) {
    db.runCommand({dropIndexes: collName, index: "*"});
});

Tip from here.

Categories: mongodb Tags: , ,

[mongodb] move docs from one collection to another one

February 24, 2016 Leave a comment
var documentsToMove = db.collectionA.find({});
documentsToMove.forEach(function(doc) { 
    db.collectionB.insertOne(doc); 
    db.collectionA.remove(doc);
});

Tip from here.

Categories: mongodb Tags:

MongoDB: upgrade to the WiredTiger storage engine

February 7, 2016 Leave a comment

Problem
I started to use the MongoDB 2.x series and since then I’ve upgraded to MongoDB 3.2. MongoDB 3.0 introduced pluggable storage engines. In 3.0 the default was MMAPv1 but in 3.2 the default became the WiredTiger storage engine that has some advantages: document level concurrency, compression on data and indexes, etc. So naively, I checked my configuration:

$ mongo
MongoDB shell version: 3.2.0
connecting to: test
> db.serverStatus().storageEngine
{ "name" : "mmapv1", "supportsCommittedReads" : false }

What???

Solution
Well, since I had some MMAPv1 databases when I upgraded MongoDB, it continued to use the old storage engine, so it’s understandable why I didn’t have WiredTiger. When I got over the shock, I decided to upgrade to the new engine. Here are the steps (under Manjaro):

Make a full backup of your databases:

$ mongodump --out <exportDirectory>

Stop the mongod process:

$ systemctl stop mongodb

Try to start the mongo shell with the command “mongo”. If it fails, then the server is correctly stopped.

Rename your database folder and create a new one. The “#” prompt means a root prompt.

# cd /var/lib
# mv mongodb mongodb.mmapv1
# mkdir mongodb
# chown mongodb mongodb
# chgrp daemon mongodb

Make a copy of your log file:

# cd /var/log/mongodb
# mv mongod.log mongod.log.mmapv1

Edit the config file /etc/mongodb.conf . I still had the old format (the new format uses YAML), but if it works then I don’t replace it yet. Add the following line to its end:

storageEngine=wiredTiger

Now restart the server:

$ systemctl start mongodb

And check if it uses the new engine:

$ mongo
MongoDB shell version: 3.2.0
connecting to: test
> db.serverStatus().storageEngine
{ "name" : "wiredTiger", "supportsCommittedReads" : true }

Great! Now restore the databases that we backed up in the first step.

$ mongorestore <exportDirectory>

It will take a while. Let’s do a final check. Print the stats of a collection (sample):

$ mongo
MongoDB shell version: 3.2.0
connecting to: test
> show dbs
mydb
> use mydb
switched to db mydb
> show collections
mycoll
> db.mycoll.stats()
{
        "ns" : "crunchbase.companies",
        "count" : 18801,
        "size" : 72236994,
        "avgObjSize" : 3842,
        "storageSize" : 36052992,
        "capped" : false,
        "wiredTiger" : {
                "metadata" : {
                        "formatVersion" : 1
                },
        ...

It’s in WiredTiger format. Awesome.

If you want, you can delete the folder /var/lib/mongodb.mmapv1 with the old databases. But first make sure that everything works as before the migration.

Links