PointerLock API Updates

A quick update on the Firefox PointerLock API implementation

Lets start with mochitests. While writing mochitests for pointerlock we stumbled on two problems

  1. Not being able to specify how many tests should run (different platforms were running different number of tests)
  2. Mochitest iframe not allowed to go fullscreen, making us run all the tests on a different window

David Humphrey came up with a solution for our first problem and added an “expect” functionality to the mochitest framework.
So now we can specify how many tests should occur when making asynchronous tests, for example:
SimpleTest.waitForExplicitFinish(3)
Bug 724578

For our second problem, I added the attribute mozallowfullscreen=true to the mochitest iframe that runs all tests.
I’m not sure if there was a specific reason for not allowing fullscreen on the mochitest iframe, but if it wasn’t it will simplify a lot writing tests for pointerlock
Bug 728893

Spec Updates

The spec had two major changes

  1. Switching from callbacks to events
  2. Moving functionality to the Document and Element

For example:
Everytime the pointer is locked/unlocked a mozpointerchange event will be dispatched to the document
A mozpointererror event will be dispatched if there are any errors while locking the pointer
Now It’s possible to access the element with the pointer locked via the document

var div = document.createElement("div");

document.addEventListener("mozpointerlockchange", function (e) {
  if (document.mozPointerLockElement === div) {
    // Pointer is locked
  }
}, false);

document.addEventListener("mozpointerlockerror", function (e) {

}, false);

document.addEventListener("mozfullscreenchange", function (e) {
  if (document.mozFullScreen &&
      document.mozFullScreenElement === div) {
      div.mozRequestPointerLock();
  }
}, false);

div.mozRequestFullScreen();

Instead of something like this:

var div = document.createElement("div");

div.addEventListener("mozpointerlocklost", function (e) {
  // Dispatched when pointer is unlocked
}, false);

document.addEventListener("mozfullscreenchange", function (e) {
  if (document.mozFullScreen &&
      document.mozFullScreenElement === div) {
      navigator.mozPointer.lock(
        div, // Element
        function () {
          // Success callback
        },
        function () {
          // Failure callback
        }
      );
  } 
}, false);

div.mozRequestFullScreen();

Advertisements

Updating PointerLock API – Callbacks, Events and Threads

The PointerLock implementation of Firefox is going great, we are close to having the patch ready to land, maybe Firefox 13.

The work being done now is mainly some final touches, specially on the mochitests and on the API.

Recently the W3C PointerLock spec has been updated, the changes are the following:

  • When locking the mouse, dispatch pointerlockchange/pointerlockerror events instead of firing callbacks
  • Locking the pointer by requesting pointer lock on the target element
  • Adding a reference to the locked element in the Document
  • Exiting pointerlock by calling exitPointerLock on the Document

Those were significant changes, since it affected a big chunk of the code we had it implemented. However, I believe these updates to the API are beneficial, since with them developers will have an API similar to the fullscreen to work with.

The first bit I started working on was to dispatch the pointerlockchange/pointerlockerror instead of callbacks.

To Dispatch the events, the nsAsyncDOMEvent object was used:

static void
DispatchPointerLockChange(nsINode* aTarget)
{
  nsRefPtr e =
    new nsAsyncDOMEvent(aTarget,
                        NS_LITERAL_STRING("mozpointerlockchange"),
                        true,
                        false);
  e->PostDOMEvent();
}

Same logic to dispatch the pointerlockerror and pointerlocklost

One of the good things about having to go back and rewrite some code, is the fact that opens the possibility to analyse some of the decisions made before.
Specifically in this case, the use of different threads when locking the pointer.
At first, the callbacks were being fired on a different thread so the execution wouldn’t hang, and the Lock method would be able to return as soon as possible and not make the user wait for a result.

Before, the logic for callbacks was mainly based off the nsGeoLocation implementation. However, now with the pointerlock API looking more like the fullscreen api I went and looked how they handle setting the element into fullscreen.
I had written a blog post a while back inspecting the fullscreen API, so even with the API receiving some changes it was easy to locate the code path for requesting fullscreen on an element.

Here is a simple diagram I drew

The diagram shows that once mozRequestFullScreen is called on an element, the method returns really fast and all the heavy processing happens on a separate thread.

On the other hand, this is how PointerLock does it:

On PointerLock, different from the FullScreen, the heavy processing happens on the main thread, and the new thread only handles the callback firing. Now switching to events, even less processing happens on the new thread, so that made me rethink the logic for locking the pointer.

I remember hearing that all the code that involves changing the presentation, it needs to happen on the main thread, so maybe that’s why we’re not spinning the pointerlock check/validation to another thread, since it involves changing the UI presentation by hiding the pointer if the lock is successful.

Another thing that caught my attention was the fact that on the fullscreen code, the nsCallRequestFullScreen object was dispatched to a new thread using NS_DispatchToCurrentThread and on PointerLock we are using NS_DispatchToMainThread


NS_DispatchToCurrentThread
NS_GetCurrentThread
nsThreadManager::GetCurrentThread

NS_DispatchToMainThread
mMainThread


Bug 713608 Update

One of the bugs I’m currently working on is: Bug 713608 – HTML5 Video controls are missing in Fullscreen
A quick overview of the bug:
This bug involves working with html5 video controls.
The goal of the bug is to enable the stock controls when the video enters in fullscreen via the context menu. For example: the controls are hidden when the video isn’t in fullscreen, then the user right clicks on the video and selects enter fullscreen. When the video switches to fullscreen mode, the controls will be displayed, even though they weren’t before toggling fullscreen, and after exiting fullscreen mode, if the controls were disabled in the first place, disable them again.

After working on Bug 714071 – The Show Statistics setting is not preserved when toggling the full screen that also involved working on the videocontrols.xml file, I thought would be good to keep digging how XBL bindings work on firefox.
So far Bug 713608 has proven to be more complicated than I thought it would be.
At the beginning, I followed the same logic used on Bug 714071 and tried to find the method that shows/hides the controls. However, different from the Statistics menu option, it seems that hiding the controls uses a different approach.
This is the code that displays the statistics of a video

showStatistics : function(shouldShow) {
                    if (this.statsInterval) {
                        clearInterval(this.statsInterval);
                        this.statsInterval = null;
                    }

                    if (shouldShow) {
                        this.video.mozMediaStatisticsShowing = true;

                        this.statsOverlay.hidden = false;
                        this.statsInterval = setInterval(this.updateStats.bind(this), this.STATS_INTERVAL_MS);
                        this.updateStats();
                    } else {
                        delete this.video.mozMediaStatisticsShowing;
                        this.statsOverlay.hidden = true;
                    }
                },

But I couldn’t find the equivalent method for the video controls.

Browsing through the code, I was able to find the methods triggered by the event listeners for mousemove/over/out used by the video element:

MouseMove

onMouseMove : function (event) {
                    // If the controls are static, don't change anything.
                    if (!this.dynamicControls)
                        return;

                    clearTimeout(this._hideControlsTimeout);

                    // Suppress fading out the controls until the video has rendered
                    // its first frame. But since autoplay videos start off with no
                    // controls, let them fade-out so the controls don't get stuck on.
                    if (!this.firstFrameShown &&
                        !(this.video.autoplay && this.video.mozAutoplayEnabled))
                        return;

                    this.startFade(this.controlBar, true);
                    // Hide the controls if the mouse cursor is left on top of the video
                    // but above the control bar and if the click-to-play overlay is hidden.
                    if (event.clientY < this.controlBar.getBoundingClientRect().top && this.clickToPlay.hidden) {
                        this._hideControlsTimeout = setTimeout(this._hideControlsFn, this.HIDE_CONTROLS_TIMEOUT_MS);
                    }
                },

MouseInOut

onMouseInOut : function (event) {
                    // If the controls are static, don't change anything.
                    if (!this.dynamicControls)
                        return;

                    clearTimeout(this._hideControlsTimeout);

                    // Ignore events caused by transitions between child nodes.
                    // Note that the videocontrols element is the same
                    // size as the *content area* of the video element,
                    // but this is not the same as the video element's
                    // border area if the video has border or padding.
                    if (this.isEventWithin(event, this.videocontrols))
                        return;

                    var isMouseOver = (event.type == "mouseover");

                    // Suppress fading out the controls until the video has rendered
                    // its first frame. But since autoplay videos start off with no
                    // controls, let them fade-out so the controls don't get stuck on.
                    if (!this.firstFrameShown && !isMouseOver &&
                        !(this.video.autoplay && this.video.mozAutoplayEnabled))
                        return;

                    if (!isMouseOver) {
                        this.adjustControlSize();
                        
                        // Keep the controls visible if the click-to-play is visible.
                        if (!this.clickToPlay.hidden)
                            return;

                        // Setting a timer here to handle the case where the mouse leaves
                        // the video from hovering over the controls.
                        this._hideControlsTimeout = setTimeout(this._hideControlsFn, this.HIDE_CONTROLS_TIMEOUT_MS);
                    }
                },

My plan was to find the event listeners that trigger the display of the video controls, then find how those events are added/removed to element.

I found where the listeners were defined

<handlers>
        <handler event="mouseover">
            if (!this.isTouchControl)
                this.Utils.onMouseInOut(event);
        </handler>
        <handler event="mouseout">
            if (!this.isTouchControl)
                this.Utils.onMouseInOut(event);
        </handler>
        <handler event="mousemove">
            if (!this.isTouchControl)
                this.Utils.onMouseMove(event);
        </handler>
    </handlers>

However, I couldn’t find where they are added/removed from the video element.

I feel that I’m getting close to finding a solution.

Big thanks to Jared Wein (jaws) for helping me with this and other bugs 🙂


MongooseJS Validators – Contributing to an open source project

Today I was able to put in practice all the tools I learned last semester in the DPS 909 – Topics in Open Source Development class

The Problem

In a project for a class I’m taking this semester, I was working on writing the validation portion for the mongoose schemas for one of the collections being used

  User = new Schema({
    'username': {
      type: String,
      validate: [validateUsername, 'username not valid'],
    },
  });

It’s defening a validator for username, so when saving a User object:

var user = new User();
user.save(function(err) {

}

It will call validateUsername on username and if the validation fails the object won’t be saved and the err will have the information about the error.

{ message: 'Validation failed',
  name: 'ValidationError',
  errors: 
   { username: 
      { message: 'Validator "username not valid" failed for path username',
        name: 'ValidatorError',
        path: 'username',
        type: 'username not valid' 
      } 
   } 
}

So my problem was that I wanted to add more than one validator to a single field

  User = new Schema({
    'username': {
      type: String,
      validate: [validateUsername, 'username not valid'], [validator2, 'second validator'],
    },
  });

However that didn’t work.
I went back to the mongoose documentation but couldn’t find a way to attach two validators to a single field.

So I was faced with two options, accept the facts and move on, or try to modify the library
Mongoose is an open source library, so I started reading the source code trying to find a way to accomplish my goal.

Because the source code is very organized and easy to read it didn’t take me long to find where a shchema field was being created.

A schema field is defined in the schematype.js

function SchemaType (path, options, instance) {
  this.path = path;
  this.instance = instance;
  this.validators = [];
  this.setters = [];
  this.getters = [];
  this.options = options;
  this._index = null;

  for (var i in options) if (this[i] && 'function' == typeof this[i]) {
    var opts = Array.isArray(options[i])
      ? options[i]
      : [options[i]];

    this[i].apply(this, opts);
  }
};

path is the name of the field
options is an object cointaining all the options for the field
instance is the type of the field
for example:

path: username
instance: String
options: { 
  type: [Function: String],
  validate: [ [Function: validateUsername], 'username not valid' ],
}

The for loop iterates through all the options and calls the appropiate function depending on the property
So using the example above, ‘i’ would be equal to ‘validate’.
So calling this[i].apply(this, opts)
would be the same as calling
this.validate([Function: validateUsername], ‘username not valid’)

Here is the part where the validator gets added to the field

SchemaType.prototype.validate = function (obj, error) {
  this.validators.push([obj, error]);
  return this;
};

Pretty straight forward, it pushes the function and the error to the validators array.
But I wanted to pass more than just one function and error.

The Solution

SchemaType.prototype.validate = function (obj, error) {
  if ('function' == typeof obj && 'string' == typeof error) {
    this.validators.push([obj, error]);  
  }
  else {
    for (var i in arguments) {
      this.validators.push([arguments[i].func, arguments[i].error]);
    }
  }
  return this;
};

So if I defined more than one validator for a single field in the schema, the arguments var for the validate method would look something like this:

arguments:  { '0': { func: [Function: trim], error: 'trim error' },
  '1': { func: [Function: validateEmail], error: 'email error' } }

However I couldn’t just break the existing code, so I added a check to see if there was more than one validator before pushing to the validator array

In the end, the solution worked.
So I created a patch and opened a ticket on the mongoose repo to discuss the issue.
I’m not sure if the change will be accepted in the project, but it was nice to see that I can actually modify the library if nedded

**After going over one more time through the documentation I found a different way to add multiple validators

User.path('username').validate(function (v) {
  return false;
  }, 'my error type'); 
  User.path('username').validate(function (v) {
    return true;
  }, 'another error');

It calls the validate function explicity on the field, allowing multiple validators to be added


Firefox Bug 714071

Working on Bug 714071 introduced me to another layer of Firefox.
So far all I’ve been doing was working with c++ code, specifically related to the MouseLock API.
Bug 714071 on the other hand was focused on the js layer.

A brief summary of the bug:

Fix a problem with the Statistics video control


When the showing statistics option of a video was on, and the fullscreen was toggled the video would stop displaying the statistics but the menu would not be updated.
The goal for the bug was to keep showing the statistics when toggling between fullscreen.

The code that displays the statistics on a video is the following:

                showStatistics : function(shouldShow) {
                    if (this.statsInterval) {
                        clearInterval(this.statsInterval);
                        this.statsInterval = null;
                    }

                    if (shouldShow) {
                        this.video.mozMediaStatisticsShowing = true;

                        this.statsOverlay.hidden = false;
                        this.statsInterval = setInterval(this.updateStats.bind(this), this.STATS_INTERVAL_MS);
                        this.updateStats();
                    } else {
                        delete this.video.mozMediaStatisticsShowing;
                        this.statsOverlay.hidden = true;
                    }
                },

Everytime the video was loaded on the page, or toggled in fullscreen, the init method would be called and an event listener would be attached to listen for the “Show Statistics” option click. However, the init method would initialize the video with a fresh config, so if the statistics were being displayed it would be hidden after switching between fullscreen mode.

Solution

The solution was to add a check on the setUpInitialState method to activate the statistics on the video if they were being displayed before toggling the fullscreen.

if (this.video.mozMediaStatisticsShowing) {
  this.showStatistics(true);
}

That apparently solved the problem and the statistics are preserved even when toggling fullscreen.

Working on this bug made me very curious. How could some javascript code interact with c++ at run time.
I knew about the XPCOM object model used in Firefox and lately I’ve started to read more about XUL and the Gecko engine . This was the perfect time to start digging more deep and learn more about the Firefox foundations.
Using the videocontrols as a starting point I went to mxr and started searching some code.
From my initial search I think I found where the controls were loaded in c++. Now I just need to figure it out how all that happens 🙂

videoscontrols.xml – define the video controls

nsCOMPtr mVideoControls;
nsVideoFrame::CreateAnonymousContent – where the menu gets attached to the video element

nsNodeInfoManager – apparently used to load the video controls as well the poster image
nsNodeInfoMangager::GetNodeInfo – looks like the loading happens here

Manage XPCOM objects?
PL_HasTableLookup
PLHasEntry
PL_HasTableAdd

NS_TrustedNewXULElement – where the XUL element gets created and casted to an nsIContent
NS_TrustedNewXULElement declaration
NS_TrustedNewXULElement definition

TODO:
Keep digging and find how the pieces fit together!


DPS911 Update

After being introduced to Firefox development last semester in the DPS909 – Topics in Open Source Development and now being enrolled in the DPS911 – Open Source Project it’s time to get some bugs fixed
Besides continuing the work started last semester in the implemention of the MouseLock Spec on Firefox, I picked a few other bugs to work on.

My first Firefox patch originated from Bug 715141. The bug was to add a keyboard shorcut to resize the zoom level of an image to a 1:1 scale. When Firefox serves an image file and the image is bigger than the boundaries of the browser, the image gets resize to fully fit in the browser window. A magnifying glass with a +/- sign appears when the mouse hover overs an image if the image can be resized.
So the goal of the bug was to add support to manipulate the zoom level with only the keyboard.
In one of the comments of the bug, Jared suggested that the changes would probably happen on the ImageDocument.cpp file
At first I thought to look for the code that changed the browser zoom level, the Ctrl +/- shortcut. I found what I thought was the code for switching zoom levels in the HandleEvent method. However, it turned out to be the code that was resizing an mage to a 1:1 scale! The shortcut already existed, was Shift +/- I then spoke with Jared and he talked to gavin and in the end was decided that since noboby ever used, or even knew about the shortcut, it would be better to remove it due code maintance in the future, so Bug 715141 was closed and a new bug to remove the code was created, Bug 718133.
On Bug 718133, knowing where the code for the shortcut was located, all I had to do was remove it, and that was my first contribution to Firefox 🙂

My second contribution was Bug 713383. On Bug 376997, when the browser serves an image file instead of the image being displayed on the top left corner in a white background it began being displayed in the center of the page with a greyish background. However, if somedoy wanted to print an image, most likely they don’t want the greyish background or the image to be centered on the page. After some suggestions, was agreeded that the best solution would be to wrap all the new styles introduced in Bug 376997 on a

@media not print

rule and preserving the new styles for displaying an image but not applying them when the image is being printed.

Quick Summary:

Bug 633602 – Implement Pointer Lock (Mouse Lock) API
Bug 715141 – Using keyboard to switch zoom level when viewing images directly
Bug 718133 – Remove keyboard shortcuts for zooming to 1:1 scale when viewing standalone images
Bug 713383 – Directly viewed images should be printed with a transparent background on the body
Bug 714071 – The Show Statistics setting is not preserved when toggling the full screen mode
Bug 712225 – When element is in fullscreen mode, window.innerHeight is bigger than window.outerHeight
Bug 581866 – outerWidth / outerHeight features in window.openDialog do not include window manager decorations
Bug 581863 – gtk2/nsWindow::GetScreenBounds returns client window size instead of outer (frame) size