Rob Sayre's Mozilla Blog

I’ve been using an evaluation license of Rational PurifyPlus for the past few days. That’s not as much work as it sounds like, because it’s really slow to run, so I just kick off a test run and go away for two or five hours. This software is hard to like, initially. It’s proprietary. It’s expensive. It doesn’t work on Vista. Last, but not least, Firefox 2.0 is not supported by all Rational products.

We had a debate about whether these tools were worth the money. My guess is that we wasted time and money having that debate, because this stuff really really works. But that’s only a guess, because my manager did the right thing and shielded me from it, and I’ll have a license in a few days. It is catching many errors that valgrind doesn’t. It’s also catching errors that only appear on Windows, where some large percentage of our users are found.

It also generates pretty good coverage data. Right now, it’s plodding through mochitest, checking exactly which lines of code in the Mozilla tree are executed by our test suite. It’s not as bad you might expect.

For example, it turns out that reftest covers about 60% of libgklayout.dll, which is really pretty good. Mochitest covers a bunch of things that reftest wouldn’t, like obscure corners of the W3C DOM API. When my analysis is done, I expect that our coverage will actually be decent, but probably not adequate. The good news is that it looks like it will be easy to add things to test lots of things we’re missing.Here’s a summary of the test coverage we get from reftest on HTML table code:

Some files clearly need some help, and mochitest will hit some of these gaps, but it would be a waste to aim for 100%. In the example below, we should clearly be hitting GetContinuousBCBorderWidth (whatever that is!). The NS_ERROR code in the function above it is a lot less interesting. In fact, it seems to me that not hitting it is actually good.

I’m pretty sure this data will give our hackers the information they need to increase testing coverage where we really need it.

The tool will give us this data in a tab-separated text file. Once we figure out how that works, it should be pretty easy for someone to put robcee’s buildbot work to use with Purify, and output a gigantic set of static HTML files after the runs complete. We’ll need a few extra switches for the harnesses, but nothing too complicated.

We should probably get Brian Marick’s How to Misuse Code Coverage PDF converted to HTML, so we can easily refer to it on the Web.

To make this work, we’ll need

1. A fast Windows XP machine to run the tests.
2. Someone to write code that can parse the tab-separated files and produce highlighted HTML files, or make bonsai do it.
3. Someone to call IBM/Rational and check that we’re allowed to publish that HTML publicly. The license agreement is really complicated, so I’m not going to pretend to understand it.
4. buildbot handholding from someone who understands it
5. Some command line flags that tell mochitest, make check, and reftest to execute under Purify.