Jackson overview

Recently I found the need to dig deeper and start getting a better grip on how jackson handles data parsing and manipulation. I always had some problems converting Jpa -> Date -> DateTime -> Json and back and forth, however, by plugin in some custom serializes/deserializers I always hacked my way around it.

Now that I’m starting to use Jongo I had to look at some features jackson provides to customize the mapping between mongo BasicDBObjects and POJOs

In the next few weeks I plan to write some blog posts show casing some cool features jackson offers.

 

Jackson overview

One thing to make clear is that the official project is now being maintained under FasterXML, not Codehaus.
Codehaus is a collaborative environment for building opensource projects, jackson moved away from codehaus is now under the FasterXML umbrella

The last released made under the codehaus banner was in July 14 2013, version 1.9.13,
Some differences between the old and new jackson are:

  • Maven build instead of Ant
  • Annotations carved out to a separate package (that this package depends on)
  • Java package is now com.fasterxml.jackson.core (instead of org.codehaus.jackson)

Main components

jackson-core – defines low-level streaming api, and includes JSON-specific implementations
The core package is where all the low level parser implementation is kept, some core classes that handle the raw json object creation are JsonReadContext and JsonWriteContext

More info can be found at their javadocs

jackson-annoations – contains standard jackson annotations
The annoations packages contains the definition of all the annoations used by jackson, 31 in total.
Some annoations worth noting are:
@JsonProperty
@JsonIgnore
@JsonInclude
@JsonView
@JacksonInject
@JsonSubTypes

jackson-databind – implements data-binding and object serialization support on streaming packages

This is the package the handles most of the jackson parsing logic, classes like the ObjectMapper and SimpleModule are present in this packaged

The databind package bootstraps the annotations defined in the jackson-annotations package, one reason to separate the annoations from the databind package is to allow third party libraries to extend the utilize the annoations without having to include the whole databind package in its build.

Besides the main modules jackson also provides support for third party libraries, some of them are:

  • joda
  • hibernate
  • guava
  • hppc
  • jax-rs

Overall jackson is a stable library that provides solid data manipulation support for different data types in java. The way it is architecture it allows new types to be easily implemented and it also provides a rich feature api that developers can extend to fit different application needs.

One thing I would say is that the documentation about jackson is very fragmented, I usually like to go to a single page and get all the info I need about a project, with jackson I always find myself hoping around between sites to find the info I need. It might be fragmented due to the change from codehaus to fasterxml, but in any case I would really like to see some effort into making the library more presentable.

I know for sure that several applications use jackson and the library is rock solid, its web presence should reflect the same image.

I’ll give one example: Jongo

Jongo documentation is pretty well done, plus other useful links regarding the project are well organized. Jongo uses jackson as its based parser, maybe jackson could use jongo presentation as an inspiration?

Advertisements

Installing GitLab 5 on CentOS 6.2

You now have no excuses to not be using git for your projects.
Beside great FREE git hosting services and paid self hosted solutions like github and bitbucket there is also the option to go self hosted for FREE

You can have full control over your servers and environment and only allow access to your projects to the people involved in it.

GitLabs is a full fledged open source self hosted git management solution.

The installation of gitlabs is very straight forward.
Here are the steps that I followed:

1. Download

The folks at gitlabs published a script to install gitlabs on Ubuntu but not on CentOS.
However, thanks to Mattias Ohlsson, that compiled all the notes about how to install gitlabs on cento, now there is a script that installs from top to bottom gitlabs on centos.
You can find the script here:
Or try the steps manually following these notes: https://github.com/gitlabhq/gitlab-recipes/tree/master/install

So the commands you need to run are:

wget https://raw.github.com/mattias-ohlsson/gitlab-installer/master/gitlab-install-el6.sh

*I edit the file and set the mysql password manually instead of leaving up to the script to create one. If you don’t care about the mysql pass just leave the script the way it is that later it will spit out which password it chose for the mysql db

2. Install

chmod +x gitlab-install-el6.sh
HOSTNAME=yourhostnamehere ./gitlab-install-el6.sh

It will take a few mins and after it is done you will have a working version of gitlabs on your machine \o/

3. Fix issues

The only problem I had was with the path to ruby.
The script completed fine, without any errors, however when I tried to push a repo to the new server I got this error message:

/usr/bin/env: ruby: No such file or directory

The server was not finding the path to ruby.
To solve it was pretty simple.
I just added ruby to the $PATH.

cat > /etc/profile.d/root.sh << EOF
export PATH=/usr/local/rvm/src/ruby-1.9.3-p392:$PATH
export PATH=/usr/local/rvm/src/ruby-1.9.3-p392/bin:$PATH
EOF

source /etc/profile.d/root.sh

After running the commands above you should be able to push without any trouble to your gitlab server.


Dependency injection with Node.js

In the last project I was working on I had the chance to apply some dependency injection patterns on a node.js application.
Before I get into the details of the implementation it is important to understand how using dependency injection could benefit your project.

Wikipedia’s definition

Dependency injection is a software design pattern that allows removing hard-coded dependencies and making it possible to change them, whether at run-time or compile-time.[1]

This can be used, for example, as a simple way to load plugins dynamically or to choose stubs or mock objects in test environments vs. real objects in production environments. This software design pattern injects the depended-on element (object or value etc) to the destination automatically by knowing the requirement of the destination. Another pattern, called dependency lookup, is a regular process and reverse process to dependency injection.

Basically, dependency injection gives you the flexibility to separate the module’s functionality from it’s dependencies.
This decoupling can come in handy during testing or even when you find yourself in the need to modify some dependencies of a module later on.

Creating the module

Lets look at how you would be able to implement some dependency injection patterns with node.

I’m going to use the WebVirt project to show some examples in action.

The code blow represents a single controller that manages some express routes:

var VirtController = function (di) {

};

VirtController.prototype.actions = function (req, res) {

};

VirtController.prototype.hostStats = function (req, res) {

}

VirtController.prototype.list = function (req, res) {

};

module.exports.inject = function(di) {
   if (!_virtController) {
    virt = di.virtModel
    Step = di.Step;
    _ = di._;
    logger = di.logger;
    _virtController = new VirtController(di.config.logger);
  }

  return _virtController;
}

The controller has three basic methods:

  • actions
  • hostStats
  • list

However, only the inject method is exported.
That’s the only entry point of the module, you can perform some validation, initialization procedures, anything that needs to be done before you instantiate the module.

In the example above we only check if an instance was already created so we don’t create two equal objects, applying the Singleton pattern.

Injecting dependencies

To use the module all we need to do is to “inject” the dependencies and receive back the initialized instance:

// Load dependencies
var _ = di._ = require("underscore");
di.Step = require('../../external/step/lib/step.js');
di.exec = require('child_process').exec;
di.config = config = require('../../config/config.js');
di.logger = logger = require('../../utils/logger.js');

exports.virtModel = di.virtModel = require("./models/virt-model.js").inject(di);

exports.virtController = virtController = require("./controllers/virt-controller").inject(di);

One of the major benefits we gained by applying dependency injection into our project was that gave us the flexibility to quickly identify what the module needed to operate on, and if any changes were needed we could quickly patch them.
For example;
The WebVirt project is composed of two different pieces, the WebVirt-Manager and the WebVirt-Node.
They are separate modules that share the same code base but are designed to run on different hosts. Each one of them have specific dependencies.
The WebVirt-Manager requires Redis to store the users of the system as well other bits of data.
However the WebVirt-Node does not need Redis.
That posed a huge problem since both apps were sharing the same code base and we were using a Logger module that was saving the logs to a Redis db.
And only the WebVirt-Manager host had a Redis db running.

To fix this problem we passed a “Custom Logger” to the WebVirt-Node.
Instead of requiring the Logger that was talking with the Redis db, we passed a Logger that only logged stuff to the console.

// Load dependencies
var _ = di._ = require("underscore");
di.Step = require('../../external/step/lib/step.js');
di.exec = require('child_process').exec;
di.config = config = require('../../config/config.js');
var logger = {
  error: function (err, metadata) {
    console.log("err: ", err);
    console.log("medatata: ", metadata);
  }
}
di.logger = logger;

exports.virtModel = di.virtModel = require("./models/virt-model.js").inject(di);

exports.virtController = virtController = require("./controllers/virt-controller").inject(di);

And by just changing a few lines of code we were able to modify the module’s dependencies without altering it’s functionality.


DPS915 Workshop 1 – Initial Profile

Int the first workshop for the DPS915 course(Parallel Programming Fundamentals) we had to profile a simple application.
I wrote a previous blog post listing the steps to profile an application on osx.

The application we had to profile was:

// Profile a Serial Application - Workshop 1
 // w1.cpp

 #include <iostream>
 #include <iomanip>
 #include <cstdlib>
 #include <ctime>
 using namespace std;

 void init(float** a, int n) {
     float f = 1.0f / RAND_MAX;
     for (int i = 0; i < n; i++)
         for (int j = 0; j < n; j++)
             a[i][j] = rand() * f;
 }

 void add(float** a, float** b, float** c, int n) {
     for (int i = 0; i < n; i++)
         for (int j = 0; j < n; j++)
             c[i][j] = a[i][j] + 3.0f * b[i][j];
 }

 void multiply(float** a, float** b, float** c, int n) {
     for (int i = 0; i < n; i++)
         for (int j = 0; j < n; j++) {
             float sum = 0.0f;
             for (int k = 0; k < n; k++)
                 sum += a[i][k] * b[k][j];
             c[i][j] = sum;
         }
 }

 int main(int argc, char* argv[]) {
     // start timing
     time_t ts, te;
     ts = time(nullptr);

     // interpret command-line arguments
     if (argc != 3) {
         cerr << "**invalid number of arguments**" << endl;
         return 1;
     }
     int n  = atoi(argv[1]);   // size of matrices
     int nr = atoi(argv[2]);   // number of runs

     float** a = new float*[n];
     for (int i = 0; i < n; i++)
        a[i] = new float[n];
     float** b = new float*[n];
     for (int i = 0; i < n; i++)
        b[i] = new float[n];
     float** c = new float*[n];
     for (int i = 0; i < n; i++)
        c[i] = new float[n];
     srand(time(nullptr));
     init(a, n);
     init(b, n);

     for (int i = 0; i < nr; i++) {
         add(a, b, c, n);
         multiply(a, b, c, n);
     }

     for (int i = 0; i < n; i++)
        delete [] a[i];
     delete [] a;
     for (int i = 0; i < n; i++)
        delete [] b[i];
     delete [] b;
     for (int i = 0; i < n; i++)
        delete [] c[i];
     delete [] c;

     // elapsed time
     te = time(nullptr);
     cout << setprecision(0);
     cout << "Elapsed time : " << difftime(te, ts) << endl;
 }

We had to run the application with 12 different combinations to see how much time the program spent executing the “add” and “multiply” functions.

Here is the profile results:

To easy the process of generating the profile data, I create a bash script to automate the runs:

#!/bin/bash

# First Set
N[0]=80
NR[0]=50

N[1]=160
NR[1]=50

N[2]=320
NR[2]=50


# Second Set
N[3]=80
NR[3]=100

N[4]=160
NR[4]=100

N[5]=320
NR[5]=100


# Third Set
N[6]=80
NR[6]=200

N[7]=160
NR[7]=200

N[8]=320
NR[8]=200


# Fourth Set
N[9]=80
NR[9]=400

N[10]=160
NR[10]=400

N[11]=320
NR[11]=400


if [ $(uname) = "Darwin" ]
then
OS="mac"
  CC="g++-4.7"
else
OS="linux"
  CC="g++"
fi

echo "OS $OS"

OPTIONS="-std=c++0x -O2 -g -pg"
OBJ="w1"
SRC="w1.cpp"

INSTRUMENT_TEMPLATE="/Applications/Xcode.app/Contents/Applications/Instruments.app/Contents/Resources/templates/Time Profiler.tracetemplate"
#compile workshop
$CC $OPTIONS -o $OBJ $SRC

#generate profile info
for i in {0..11}
do
echo "Running ${i}th set"
  if [ $OS = "mac" ]
  then
echo "Running on MacOS"
    instruments -t "$INSTRUMENT_TEMPLATE" -D results/mac/"${N[$i]}x${NR[$i]}.log" $OBJ ${N[$i]} ${NR[$i]}
  else
echo "Running some linux distro."
    ./$OBJ ${N[$i]} ${NR[$i]}
    gprof -p $OBJ > "results/linux/${N[$i]}x${NR[$i]}.log"
  fi
done

The script works both on mac and linux.
If it’s running on a mac, it uses the Instruments Time Profiler, on a linux distro it uses gprof.

I’m committing all my course work to github

Any suggestions are more than welcome 🙂


Using Instruments Time Profiler

Gprof problem

On OSX 10.8.1 (Mountain Lion) the gnu profiling tool wasn’t working.
I’ve looked it up online and there was very little documentation about the problem.
I read in a couple of places saying that gprof in fact didn’t work but I couldn’t find any final answers.
Basically what happened is that when the program was compiled with the “pg” option, the gmon.out file was not created, thus not being able to run gprof to gather profile information for a specific program.

At first I thought the problem could be related to the fact that I was running gcc 4.2.1(the one that comes by default with XCode) so I tried to compile the latest version of gcc from source to check if it solved the problem.
I compiled gcc version 4.7.1. However it didn’t fix the problem.

I even try linking the profiling lib manually, but the gmon.out file was not being created.

**I’m still trying to find why the gmon.out file wasn’t being created, if anybody knows the reason or have any suggestions please leave a comment below.
My next step will be to compile the libc from source to add some profile symbols.
I’m following these references:

A couple of resources that are not related to gprof but nevertheless very useful:

 

Time Profiler

With all that being said, I needed to profile a c++ program on the mac, so I went looking for alternatives.

Luckly, I found that XCode comes with some extra tools called Instruments
A few tools included in the Instruments toolset are:

  • Allocations
  • Leaks
  • Activity Monitor
  • Time Profiler
  • System Trace
  • Automation
  • Energy Diagnotics

To get started with the Time Profiler is very simple, you first need to create a Xcode project.

Select the Profile option under Product (Command + I)

Select the Time Profiler template

Finally it will display the profile of your application

So far so good, I managed to generate profile information for my application. However, what if I wanted to get the information via the command line?
In my case I had to run the same application several times with different arguments to inspect how some functions behaved in certain situations and if they needed some optimizations.
With that in mind, running the time profiler via XCode was out of the question since I would need to manually modify the arguments and run the profiler each single time.
Instead I created a bash script to automate the runs.

Now I needed to find how to run the Instruments Time Profiler via the command line.
It wasn’t easy, there is very few documentation online and the manual has some outdated information.
Instead of [-d document] the correct is [-D document]
Anyway, to run Instruments from the command line:

instruments -t PathToTemplate -D ProfileResults YourApplication [list of arguments]

To see a list with all the available templates:

instruments -s

The result is a trace file that will contain the information regarding the profiling of the application.


Building Firefox on Mountain Lion 10.8

All the work that I’ve done on Firefox so far has been on a linux box.
I bought a mac recently so I’m in the process of switching all my dev tools.
To build Firefox on a mac is almost as straight forward as building on a linux distro.

Here are the steps:

1.

First you’ll need to install macports.
Download the pkg installer for Mountain Lion or whatever version you are running and install macports

After the installation you’ll need to restart your shell so the $PATH gets updated.
You can find more details here

Once macports is installed:

$ sudo port selfupdate
$ sudo port sync
$ sudo port install libidl autoconf213 yasm mercurial ccache

The commands above will install all the dependencies you need to build firefox.

**More info on how to configure ccache here

2.

Next it’s time to checkout the source code.

hg clone http://hg.mozilla.org/mozilla-central

It might take a while to clone the whole repo.

3.

Now that you have both the dev dependencies and the source code the last thing missing is a .mozconfig file.
Below is a default configuration:

ac_add_options --enable-debug
ac_add_options --enable-trace-malloc
ac_add_options --enable-accessibility
ac_add_options --enable-signmar

# Enable parallel compiling
mk_add_options MOZ_MAKE_FLAGS="-j12"

# Treat warnings as errors in directories with FAIL_ON_WARNINGS.
ac_add_options --enable-warnings-as-errors
ac_add_options --with-ccache

# Package js shell.
export MOZ_PACKAGE_JSSHELL=1

You can find more info about .mozconfig here

4.

Now it is time to start building.

First run:

make -f client.mk configure

That will make sure everything is setup properly, if you don’t see any error messages then you can start the build:

make -f client.mk build > build.out

A trick is to redirect the output of make to a file, it not only makes it easier to spot errors but it also decreases the build time.

Depending on your computer the build might take some time, don’t expect the build to finish before 15min, it will probably take something between 30min to 2h

5.

Once the build is done, you can run Firefox by going to dir obj-dir/dist/NightlyDebug.app/Contents/MacOS and launch the firefox executable.

References:
Simple Firefox build
Mac OS X Build Prerequisites


Getting started with CUDA on OSX 10.8 – Driver Problems

To install all the dev dependencies for CUDA enabled GPUs is not that bad, I faced a few issues but overall the documentation is pretty good.

You can find more information about how to get started here, it has all the links for the download of the driver + toolkit + SDK for windows, linux and mac

They also posted a PDF giving detail instructions about how to install everything.

 

Road Blocks

I’m running a MacBook Pro 2012 that comes with a GeForce GTM 650M.
On their website, they have the driver version 4.2 for download. However, I can update the CUDA driver to version 5.0.24 through the CUDA Preferences window under the System Preferences tab.

So after following the instructions they have posted on the Get Started pdf, I would get the message “Driver not supported” when running the deviceQuery test script.
I looked up online and found that this problem usually happened when the driver had a lower version than the SDK, I thought it was weird since I had downloaded all files they had instructed on the website.

I started browsing on the System Preferences when I saw the CUDA preferences tab.
On the tab it had the option to update the driver.
After the update, my driver was on version 5.0.24, and the deviceQuery test would work. \o/

After running the deviceQuery test, they suggested to run the bandwithTest to make sure the communication with the GPU was working properly.
To my surprise, when I ran the bandwithTest the computer crashed, some weird noises came from the case and a kernel panic messaged appeared.


Interval Since Last Panic Report:  75 sec
Panics Since Last Report:          2
Anonymous UUID:                    CD3F065C-4392-433E-8B7B-9D466743EE14

Tue Sep 11 23:16:23 2012
panic(cpu 4 caller 0xffffff802e8b7b95): Kernel trap at 0xffffff7faef9d18e, type 14=page fault, registers:
CR0: 0x0000000080010033, CR2: 0xffffff8191902000, CR3: 0x000000006b34b06c, CR4: 0x00000000001606e0
RAX: 0xffffff815123d000, RBX: 0x00000000406c5000, RCX: 0x00000000101b1400, RDX: 0xffffff8043302374
RSP: 0xffffff815117b650, RBP: 0xffffff815117b650, RSI: 0xffffff8043302004, RDI: 0xffffff80432ff804
R8:  0x00000000003f6a01, R9:  0xffffff815117b664, R10: 0x0000000000ffffff, R11: 0xffffff8100d10004
R12: 0xffffff80432ff804, R13: 0xffffff8043302374, R14: 0x0000000000000000, R15: 0xffffff8043302004
RFL: 0x0000000000010206, RIP: 0xffffff7faef9d18e, CS:  0x0000000000000008, SS:  0x0000000000000010
Fault CR2: 0xffffff8191902000, Error code: 0x0000000000000002, Fault CPU: 0x4

I wasn’t sure if the kernel panic was connected with the driver update, so I went back and ran some other scripts that come with the CUDA SDK, I ran the particles, simpleGL, volumeRender and a few others, then to my surprise again, when I ran the mergeSort another kernel panic was generated.
By now I was starting to get worried, I went back to the scripts dir and run a few others to make sure my GPU was still functioning properly, I ran the particles, simpleGL, volumeRender and the clock script, and again, after starting the clock script another kernel panic.

Now I knew for sure something was wrong, that shouldn’t be happening.

It was almost 12pm and I was getting tired and frustrated.
I did the only logical thing left to do… google it.

I entered the search: “mac 2012 crash with cuda driver 5”

 

Solution

To my relief it appeared that the kernel panics were in fact a known problem with the CUDA driver version 5 for the MacBook pro 2012.
I found this post on Adobe’s blog explaining the issue.
Apparently having the “Automatic Graphics Switching” option enable causes some CUDA applications to crash.
Turning the option off solves the problem.

Without the automatic graphics switching ON I ran the bandwithTest, mergeSort and clock apps and they worked just fine.

That Adobe’s blog post was created on August 29, so I believe that a fix for this problem should be coming out very soon.
Only Mountain Lion (Mac OSX v10.8) and Lion (Mac OSX v10.7) are affected by this bug.