Update, ElementMatch and the $ Positional Operator on MongoDB/Mongoose

One of the biggest advantages of mongodb is the option of defining documents inside documents and by doing so creating powerful and at the same time flexible data structures.
In most cases, the deepest you will go is one level, even though mongo doesn’t set any limit on how many levels you can go inside a document, for example: A BlogPost can have an array of Comments, or an Image can have an array of Tags, and there is plenty of documentation online on how to manipulate single dimension arrays in documents. However, what if you have a multi dimensional array in a document?
Take for example the following model:

/**
* Line Item schema
*/
exports.EstimateLineItem = (function () {
 schemas.lineItem = new Schema({
    'name' : String,
    'description' : String,
    'quantity' : Number,
    'cost' : Number
  });

  return db.model('EstimateLineItem', schemas.lineItem);
})();

/**
* Estimate schema
*/
exports.Estimate = (function () {
 schemas.estimate = new Schema({
    'name' : String,
    'quoteID' : Number,
    'subTotal' : Number,
    'finalTotal' : Number,
    'creationDate' : { type: Date, default: Date.now },
    'status' : { type: String, default: "Active" },
    'lineItemSet' : [schemas.lineItem]
  });

  return db.model('Estimate', schemas.estimate);
})();

/**
* Job schema
*/
exports.Job = (function () {
 schemas.job = new Schema({
    'name' : String,
    'description' : String,
    'creationDate' : { type: Date, default: Date.now },
    'status' : { type: String, default: "Active" },
    'scheduledDates' : [Date],
    'customerID' : ObjectId,
    'estimateSet' : [schemas.estimate]
  });

  return db.model('Job', schemas.job);
})();

So a Job has an array of Estimates and an Estimate has an array of LineItems.
The problem is, how to add a new LineItem to an Estimate?
One option would be to query for a job, then loop through its estimateSet and find the selected estimate, then push a new line item to the lineItemSet and finally save back the job. However, there is a simpler way:

Job.update(
    {estimateSet: {"$elemMatch": {_id: estimateID}}},
    {$push:
      {
        "estimateSet.$.lineItemSet":
        {
          'name' : lineItem.name,
          'description' : lineItem.description,
          'quantity' : parseInt(lineItem.quantity),
          'cost' : parseInt(lineItem.cost),
          '_id': lineItem._id
        }
      }
    },
    {upsert:false,safe:true},
    function (err) {
      console.log("err: ", err);
      if (err) {
        res.send({
          "err": true,
        });
      }
      else {
        res.send({
          "err": false,
        })
      }
    }
  );

Breaking down the query:
It finds a job that has an Estimate with the specified ID:

{estimateSet: {"$elemMatch": {_id: estimateID}}}

Once the job is found, it pushes a new line item to the lineItemSet:

{$push:
      {
        "estimateSet.$.lineItemSet":
        {
          'name' : lineItem.name,
          'description' : lineItem.description,
          'quantity' : parseInt(lineItem.quantity),
          'cost' : parseInt(lineItem.cost),
          '_id': lineItem._id
        }
      }
    },

For more info:

Advertisements

MongooseJS Validators – Contributing to an open source project

Today I was able to put in practice all the tools I learned last semester in the DPS 909 – Topics in Open Source Development class

The Problem

In a project for a class I’m taking this semester, I was working on writing the validation portion for the mongoose schemas for one of the collections being used

  User = new Schema({
    'username': {
      type: String,
      validate: [validateUsername, 'username not valid'],
    },
  });

It’s defening a validator for username, so when saving a User object:

var user = new User();
user.save(function(err) {

}

It will call validateUsername on username and if the validation fails the object won’t be saved and the err will have the information about the error.

{ message: 'Validation failed',
  name: 'ValidationError',
  errors: 
   { username: 
      { message: 'Validator "username not valid" failed for path username',
        name: 'ValidatorError',
        path: 'username',
        type: 'username not valid' 
      } 
   } 
}

So my problem was that I wanted to add more than one validator to a single field

  User = new Schema({
    'username': {
      type: String,
      validate: [validateUsername, 'username not valid'], [validator2, 'second validator'],
    },
  });

However that didn’t work.
I went back to the mongoose documentation but couldn’t find a way to attach two validators to a single field.

So I was faced with two options, accept the facts and move on, or try to modify the library
Mongoose is an open source library, so I started reading the source code trying to find a way to accomplish my goal.

Because the source code is very organized and easy to read it didn’t take me long to find where a shchema field was being created.

A schema field is defined in the schematype.js

function SchemaType (path, options, instance) {
  this.path = path;
  this.instance = instance;
  this.validators = [];
  this.setters = [];
  this.getters = [];
  this.options = options;
  this._index = null;

  for (var i in options) if (this[i] && 'function' == typeof this[i]) {
    var opts = Array.isArray(options[i])
      ? options[i]
      : [options[i]];

    this[i].apply(this, opts);
  }
};

path is the name of the field
options is an object cointaining all the options for the field
instance is the type of the field
for example:

path: username
instance: String
options: { 
  type: [Function: String],
  validate: [ [Function: validateUsername], 'username not valid' ],
}

The for loop iterates through all the options and calls the appropiate function depending on the property
So using the example above, ‘i’ would be equal to ‘validate’.
So calling this[i].apply(this, opts)
would be the same as calling
this.validate([Function: validateUsername], ‘username not valid’)

Here is the part where the validator gets added to the field

SchemaType.prototype.validate = function (obj, error) {
  this.validators.push([obj, error]);
  return this;
};

Pretty straight forward, it pushes the function and the error to the validators array.
But I wanted to pass more than just one function and error.

The Solution

SchemaType.prototype.validate = function (obj, error) {
  if ('function' == typeof obj && 'string' == typeof error) {
    this.validators.push([obj, error]);  
  }
  else {
    for (var i in arguments) {
      this.validators.push([arguments[i].func, arguments[i].error]);
    }
  }
  return this;
};

So if I defined more than one validator for a single field in the schema, the arguments var for the validate method would look something like this:

arguments:  { '0': { func: [Function: trim], error: 'trim error' },
  '1': { func: [Function: validateEmail], error: 'email error' } }

However I couldn’t just break the existing code, so I added a check to see if there was more than one validator before pushing to the validator array

In the end, the solution worked.
So I created a patch and opened a ticket on the mongoose repo to discuss the issue.
I’m not sure if the change will be accepted in the project, but it was nice to see that I can actually modify the library if nedded

**After going over one more time through the documentation I found a different way to add multiple validators

User.path('username').validate(function (v) {
  return false;
  }, 'my error type'); 
  User.path('username').validate(function (v) {
    return true;
  }, 'another error');

It calls the validate function explicity on the field, allowing multiple validators to be added


Installing MongoDB on Ubuntu

This tutorial will cover the basics to get MongoDB running on Ubuntu

I’ll break down the tutorial in 6 parts:

  • 1 – Setting up the environment
  • 2 – Adding repo key
  • 3 – Adding repo source
  • 4 – Installing mongo
  • 5 – Running Mongo
  • 6 – Tips

1 – Setting up the environment

If you tried to install mongo before and wasn’t successful, the best option is to uninstall all the existing mongo packages,. To do that you can run:

diogogmt@diogogmt-ID54-Series:~$ dpkg -l | grep mongo

If you see mongodb-10gen installed, then you have the right version, if you see mongodb-server, then you’ve installed from Ubuntu’s repository.
10gen repo is always up to date, and contains all mongo’s updates. So its better to install mongo using their repo.

If mongodb-server is installed, to remove the package run:

dpkg mongodb-server -P

A small description of dpkg:

dpkg is a tool to install, build, remove and manage Debian packages. The primary and more user-friendly front-end for dpkg is aptitude. dpkg itself is controlled entirely via command line parameters, which consist of exactly one action and zero or more options. The action-parameter tells dpkg what to do and options control the behavior of the action in some way.

**Some extra info, on how Ubuntu handles deb packages:
There are several tools to install a deb package on Ubuntu. The base tool that actually do the installation is the dpkg command.
Before the dpkg, is the apt system, which serves as a front end for dpkg. The synapitc, aptitute are a front end for the apt system, which is contained in the apt (Debian package). From apt that all the commands, apt-get, apt-update, apt-key comes from.

This blog has some very good information on how deb packages are handle :
http://algebraicthunk.net/~dburrows/blog/

2- Adding repo key

On this tutorial we’ll install mongo using 10gen official repo.

To be able to download mongo with apititude from 10gen repo, a key must be added first. That will verify if the repository is trusted.
The key can be added using apt-key
A quick description for the command:

apt-key is used to manage the list of keys used by apt to authenticate packages. Packages which have been authenticated using these keys will be considered trusted.

Here is the command to add the key:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10

Breaking down the command:
Another command used in the authentication of the key is gpg, because apt-key is called passing adv as an option, gpg will be invoked
GDP quick description:

gpg is the OpenPGP part of the GNU Privacy Guard (GnuPG). It is a tool to provide digital encryption and signing services using the OpenPGP standard. gpg features complete key management and all bells and whistles you can expect from a decent OpenPGP implementation.

More information on GnuPG : http://www.gnupg.org/
More info on OpenGP : http://www.openpgp.org/

3- Adding repo source

After you added the key, you can go and add the repository to your list.

On mongo’s website, it says to add

deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen

as an apt source.
If you add the source manually by editing the /etc/apt/sources.list like they recommend on the website it will work. However if you go to the Ubuntu Software Centre GUI and add the repo there. Two entries will be made to the /etc/apt/sources.list one as deb repourl and the other as deb-src repo url
For some reason, having the db-src will fail to get the updates.

Solution:

Manually enter

deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen

to

/etc/apt/sources.list

or use the Ubuntu Software Centre GUI, and after
deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen
is added, uncheck deb-src.

**Sysvinit and upstart.

On mongo’s website, there is the option of choosing the upstart and sysvinit repos. If you are using a recent version of Ubuntu(6 >) you can select the upstart.
Sysvinit used to be the startup boot program for ubuntu. Since version 6 Ubuntu has been using upstart.
If you notice, on the /etc/init.d/ dir, a lot of files are links to an upstart job

More info on boot management: https://help.ubuntu.com/community/UbuntuBootupHowto

4- Installing mongo

With the repository, and key added to your system, now is time to install mongo.

apt-get install mongodb-10gen

Congratualitions, you have MongoDB installed in your system.

5- Running mongo

If you installed mongo in a new version of Ubuntu it will be possible to start and stop the it as a service. However, if you run the command start mongodb you’ll and get this message:

diogogmt@diogogmt-ID54-Series:~$ start mongodb
start: Rejected send message, 1 matched rules; type="method_call", sender=":1.62" (uid=1000 pid=6540 comm="start mongodb ") interface="com.ubuntu.Upstart0_6.Job" member="Start" error name="(unset)" requested_reply=0 destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init"))

don’t be afraid! Even though the message is not very user friendly, what happens is that you must be root to start/stop a service, so if you run:

diogogmt@diogogmt-ID54-Series:~$ sudo start mongodb
mongodb start/running, process 6482

It will work

To run mongo there are a couple of options.
You can start as a service
Or you can simple run the program

Both options have their benefits, some times you just want to create an instance for some project that you are testing

Others you want to have mongo running consistently on the background

If you start mongo as a service, you cannot pass any arguments in the command, example:

diogogmt@diogogmt-ID54-Series:~$ sudo start mongodb --port 27001
start: invalid option: --port
Try `start --help' for more information.

All the configuration for mongo will be in the /etc/mongodb.conf
So every time you start mongo as a service it will have the configuration specified on the mongo.conf file.

Now comparing to running an instance of mongo, every time you start that instance it will have the default configuration.
To change its configuration, you can then pass the options in the star up, for example:

diogogmt@diogogmt-ID54-Series:~$ mongod --port 27001 --dbpath /home/diogogmt/data
Mon Oct 24 01:19:03 [initandlisten] MongoDB starting : pid=6576 port=27001 dbpath=/home/diogogmt/data 64-bit host=diogogmt-ID54-Series
Mon Oct 24 01:19:03 [initandlisten] db version v2.0.1, pdfile version 4.5
Mon Oct 24 01:19:03 [initandlisten] git version: 3a5cf0e2134a830d38d2d1aae7e88cac31bdd684
Mon Oct 24 01:19:03 [initandlisten] build info: Linux bs-linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Mon Oct 24 01:19:03 [initandlisten] options: { dbpath: "/home/diogogmt/data", port: 27001 }
Mon Oct 24 01:19:03 [initandlisten] journal dir=/home/diogogmt/data/journal
Mon Oct 24 01:19:03 [initandlisten] recover : no journal files present, no recovery needed

or if you want you can load a configuration file passing as an argument:

sudo mongod --config /etc/mongodb.conf

it will create an instance of mongo with the same configuration settings as starting mongo as a service.

6- Tips

Here are just a few tips, that maybe helpful if you’re getting started with mongo:

As you can see mongo gives you a lot of flexibility on how to run an configure your servers.

Like I said before, if you are testing a new project, you can create a new instance of mongo and give a different port and dbpath, so all the changes you make it wont effect the one running as a service.

Another difference, is that when you start mongo as a service, it won’t sit on your terminal listing all the interaction, to see the details of the server you can access http://localhost:28017/ or whatever port you decided to run it.

If you click on the listDatabases tab, it will say that REST is not enable, and you must start mongo with –rest option. However, you can’t pass arguments when you start mongo as a service, and if you check the /etc/mongodb.conf it doesn’t have any REST option.
To fix this is very simple, just add “rest = true” to the conf file.
For a list of all the posible configuration for mongo check their official website: http://www.mongodb.org/display/DOCS/File+Based+Configuration

**You can also just download mongo from their website: http://www.mongodb.org/downloads
After you unzip, you will see a bin folder, there are all the commands that you need to run mongo.
This way doesn’t give you a lot of flexibility, but if you just want to give it a quick and fast try, it is an option.

In the end, there are several ways to download, install, and run mongo. Choose the one it suits you better.

Good references:
http://www.javahotchocolate.com/tutorials/mongodb.html
http://www.mongodb.org/display/DOCS/Ubuntu+and+Debian+packages