Ted’s Mozilla Blog

Recently we implemented a manifest format for xpcshell unit tests (with Joel Maher doing the lion’s share of the work). After that work landed, we realized there were some things missing from our initial design, so we set out to revamp a few things to make it easier to write useful manifests.

We decided to make the manifests support boolean expressions, similar to what reftest manifests allow, except with a restricted grammar, and not “all of JavaScript”. To make this useful, we had to offer a set of values to test, so Jeff Hammel buckled down and wrote a Python module called mozinfo that had been in discussion for a long time. I wrote a few bits to hook this all up between the build system and the xpcshell test harness, and it all landed on mozilla-central this morning.

I’ll update the MDN documentation later today, but for a preview, a sample manifest entry might look like:

[test_foo.js]
skip-if = os == 'win' || os == 'linux'

If you look in your object directory in a build containing these patches (or in the xpcshell directory of a test package from a tinderbox build), you’ll find a mozinfo.json, which is where most of the values you can use in these expressions come from. For example, the mozinfo.json for my 64-bit Linux build looks like:

{'os': 'linux', 'toolkit': 'gtk2', 'crashreporter': false, 'debug': false, 'bits': 64, 'processor': 'x86_64'}

You can annotate tests with “skip-if, run-if and fail-if” conditions currently. “skip-if” indicates that a test should not be run if the condition evaluates to true, “run-if” indicates that a test should only be run if the condition evaluates to true and “fail-if” indicates that the test is known to fail if the condition is true. Tests marked fail-if will produce TEST-KNOWN-FAIL output if they fail, and TEST-UNEXPECTED-PASS (which is treated as a failure) if they pass.

Hopefully this work will enable developers to more easily work with xpcshell tests. We’d appreciate any feedback you have on these changes!

One of the goals for the Firefox team is to ensure that the user interface remains responsive to input at all times. Clearly a responsive interface is incredibly important to making the browser a useful application, but how do we measure “responsiveness”?

Dietrich has done some work on this, writing an add-on that measures the time that various UI actions take. This covers the direct case, where a user initiates an action and expects a response in a reasonable amount of time. Clearly we want to make sure that individual actions don’t take an extraordinary amount of time.

I took the opposite tack, with an eye on being able to detect when the application was not responsive to user input regardless of what actions the user was taking. Building on some work by Chris Jones and Alon Zakai, I wrote some code that instruments the main thread event loop to find out how long it takes to respond to events, which ought to be a reasonable proxy for measuring responsiveness. When the instrumentation detects that the event loop takes too long to respond (more than 50 milliseconds, currently) it writes a data point to a log giving the current timestamp and the amount of time the event loop was not responsive.

When I implemented this I had my eye on Talos integration, where we could run the browser through some automated UI tests with this instrumentation enabled, and then correlate “UI actions” with “unresponsive periods” and ensure that the browser did not become unresponsive during those actions. Talos integration has been shifted off as a longer-term goal, with the more immediate goal being “find UI actions that are the worst offenders of unresponsiveness”. To that end we’ve filed some other bugs about correlating this unresponsiveness data with JavaScript execution, and correlating the data with C++ execution. If you’ve got any ideas please feel free to contribute to those bugs!

If you’d like to try out the responsiveness instrumentation I implemented, it landed on mozilla-central a while ago, and there’s some reasonably complete documentation in the source code. There are implementations for Windows, Linux/GTK2 and OS X currently. (And a patch for an Android implementation in a bug.)

I was just looking at some data produced from our crash reporting system, and I continue to be amazed at the amount of third-party code that gets loaded into Firefox on Windows. That data file contains a list of all unique binary files (EXE or DLL) that were listed in Windows crash reports in a single day. A quick look at it shows:

$ cut -f1 -d, 20110613-modulelist.txt  | sort -u | wc -l
10385

There are over 10,000 unique filenames in a single day’s worth of crash reports. That sure seems like a lot! Now, certainly, a lot of these modules look like they’ve been randomly named, which probably indicates that they’re some kind of virus (like 0eYZf0QFDSGEAbTRWD3F.dll, for example), so those are likely to inflate the number. There’s a bug on file asking that we collect MD5 hashes of every DLL in our crash reports so we could more easily detect malware/virus DLLs that use these tactics, as well as integrate with lists of known malware and viruses from antivirus vendors.

In the past, we have had problems with plugins and extensions causing crashes for many Firefox users. We have ways of mitigating those through blacklisting. We can also blacklist specific DLLs from loading in the Firefox process, which is not used as often because it’s harder to get right and provides little feedback to users about what’s been disabled. However, given the sheer number of possible things that can be loaded in our process, it’s unlikely that we’ll ever be able to block all software that causes crashes for users. This is unfortunate, because any one of these pieces of software can cause a crash in Firefox, and all the user sees is “Firefox crashed“. I suppose we now know how Microsoft feels when users blame Windows for crashes caused by faulty drivers.

I’ve been doing some work on Firefox Mobile lately, and I keep winding up in situations where I have a URL that I want to load on my mobile device. It’s a pain to type URLs in with a virtual keyboard (or a tiny keyboard), so people have taken to using QR codes to encode them, and there are lots of mobile apps that can read the codes and load the URL contained within. Conveniently, the Google Charts API supports generating QR codes, so I threw together a tiny bookmarklet that generates a QR code of the page you’re currently viewing. Go to this page and drag it to your bookmarks toolbar to use it, then just click it whenever you’re viewing a URL that you need to view on your device. I’ve been using Barcode Scanner on Android to read them, and it works pretty well.

Last week I managed to find myself a little bit of time for a project I had been meaning to get to: bzexport. bzexport is a Mercurial extension that allows you to attach patches from your Mercurial patch queue to bugzilla from the Mercurial command line. It’s the obvious companion to the qimportbz extension. You can read the README for details on installing and using the extension, but the short form is:

hg bzexport [REV] [BUG]

where REV is the name of a currently applied patch from your queue, and BUG is a bug number to attach it to. Of course, the extension is smarter than just that, and if you leave off those parameters it will default to working on the topmost patch in your queue, and it will attempt to deduce the bug number from the commit message in the patch. (Standard notation used at Mozilla should work, i.e. “bug 12345 – some text”, “b=12345 some text” etc.)

bzexport attempts to be clever and borrow your Bugzilla login cookies from your default Firefox profile so that you don’t have to provide authentication details. This does not currently work on the Mercurial shipped with MozillaBuild, so you’ll have to provide your username and password in your .hgrc as described in the README if you intend to use it under MozillaBuild’s hg.

bzexport relies on Gerv‘s excellent BzAPI, so hats off to Gerv for that! Also, patches for additional functionality are welcome.

On a personal project of mine, I have a need to generate thumbnails of web pages. Until recently my solution was to use Firefox running in Xvfb, a virtual X server. This is not an ideal solution, as it requires you to have lots of X client libraries installed on your server. Additionally, Firefox is not intended for this purpose, so there are lots of ways for things to go wrong.

Because of this, I’d been following Chris Lord‘s work on the offscreen branch of Mozilla for some time, but never tried it out until recently. The offscreen branch provides a widget backend for Mozilla that can render web content to an offscreen buffer. Chris wrote it in support of Clutter, which is a pretty neat use case. Conveniently, he also provided a sample embedding client application called moz-headless-screenshot. This is a simple command line tool that takes a URL, image size, and output filename and generates a PNG screenshot of the webpage. This being exactly what I wanted, and having my poorly-written Firefox+Xvfb solution fall apart due to a server migration, I decided to give his solution a shot.

I hit a few speed bumps on the way, since there wasn’t much documentation to be found on actually building and using moz-headless-screenshot. I’ve attempted to fix this my providing detailed steps and a Makefile in my own moz-headless-screenshot repository. I’ve also modified the code slightly such that it’s easier to run (at least in my use case). I have heard from others over the years that have this same need, so hopefully someone else finds it useful!

I have been remiss in blogging, but as of a few weeks ago, we have two new Build Config peers:

• Mitchell Field <mitchell.field@live.com.au>
• Kyle Huey <me@kylehuey.com>

Both are available for reviewing build system patches.

MozillaBuild 1.5 has been released:
http://ftp.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/MozillaBuildSetup-1.5.exe

The major highlights include:

• A newer Mercurial (1.5.4)
• Support for Visual C++ 2010
• A newer Python (2.6.5)

You can see the full list of dependent bugs,  as well as the full list of committed changes.

As usual, bugs can be filed at bugzilla.mozilla.org.

I’d like to announce the birth of our first child, Isabella Grace-Ann Mielczarek:

She was born at 1:12 PM on Saturday, April 10th, weighing in at 7 lbs 8 oz. Mom and baby are doing great, and we just brought her home this morning. There are more photos available on my website.

I’ll be taking paternity leave starting today and returning April 26th. In the mean time, if you need help with:

• Build system/Breakpad symbols or processing issues—See Jim Blandy
• Breakpad client issues—See Chris Jones
• Unit testing issues—See Clint Talbert

I often find myself landing patches on our 1.9.2 and 1.9.1 branches, both of which are in Mercurial repositories. This generally involves getting a changeset from the mozilla-central repository into one of these repositories, and also amending the changeset message to include the name of the person who gave me approval to land. There was some discussion recently on how it’s kind of a pain to do that. I’ve cooked up an easy solution for my own needs, perhaps it will serve yours as well.

I use “hg transplant” to get changesets from one repository to another. This assumes that you have local clones of both the source repository (mozilla-central in this case) and the destination repository (mozilla-1.9.2 or 1.9.1, usually). Assuming you had both clones side-by-side in a directory, you could run the transplant command in the destination repository’s working directory like so (where xxx is the changeset identifier of the changeset you want transplanted, you can also specify more than one changeset):

hg transplant -s ../mozilla-central xxx

The transplant command conveniently includes a “–filter” option that will let you alter the commit message or patch while transplanting. This requires you to have some sort of script for transplant to run. Here’s what I’m using (on Linux):

#!/bin/sh

if test -n "$APPEND"; then
 echo " $APPEND" >> "$1";
else
 if test -n "$EDITOR"; then
 $EDITOR "$1";
 else
 editor "$1";
 fi
fi

Save this as “transplant.sh” somewhere (and ensure that it’s executable), then in your ~/.hgrc, add a section:

[transplant]
filter = /path/to/transplant.sh

Now, when you run “hg transplant”, by default it will open an editor to edit the commit message for each changeset, allowing you to add approval information. But, even better, transplant.sh will append the contents of the “APPEND” variable if set, so you can run transplant like so to quickly append approval information:

APPEND="a=someone" hg transplant -s ../mozilla-central xxx

I find that this saves me a bunch of time, so hopefully it’s useful to someone else!