Benjamin Smedberg has been hard at work making it possible to get stacks in this situation, using the same Breakpad utilities we use on our Socorro server, but locally on the machine running the tests. Practically all of the pieces were in place this afternoon when #developers cornered Alice and closed the tree while she landed the final patch to make Talos produce stack traces. Boris then committed a test crash, and as a result we were able to see crash stacks in Mochitest (OS X, Linux) as well as Talos (OS X, Linux).

Thanks to Benjamin for doing most of the heavy lifting here, and for
Alice for taking the Talos part across the finish line. The Talos work
was mostly in bug 480577, and the unit test work was bug 481732. Note
that currently this only works in Mochitest (all 4 varieties), it will
work in Reftest/Crashtest after bug 479225 is fixed (which should be soon).

(Cross posted in dev.tree-management, but posting here for a wider audience.)

About five months ago, I read an article about how Fedora wanted to standardize on NSS as the cryptography solution for their distro in order to be able to leverage a common certificate database, among other things. The article went into detail on how they wrote an OpenSSL wrapper around NSS so they could easily port applications that only supported OpenSSL to use NSS instead. As a concrete example, they showed a ported version of stunnel using NSS. This gave me pause, as one of the things we were lacking in our Mochitest harness was SSL support and stunnel would do exactly what we needed in this case. Considering we already build and ship NSS with every copy of Firefox, and it was clearly possible to implement the functionality we needed using NSS, I set out to figure out how to implement a bare-bones version of stunnel from scratch. After a bit of poking through the online NSPR and NSS documentation, I had a proof of concept application which I called “ssltunnel.” After some insightful review comments from NSS developers I committed it to CVS.

Unfortunately, that wasn’t the end. We still needed to hook this program up to the test harness, and I just didn’t have the motivation to do so. I filed the bug, and hoped someone else would do the work. (as I often do!) Thankfully, that someone appeared in the person of Honza Bambas, whom I can only describe as a “programming rockstar.” He not only integrated ssltunnel into Mochitest, but he rewrote large sections of it to make it work robustly and made it work as an HTTP proxy while he was at it. After some reviews, and a couple of landings and backouts due to unrelated test failures, and some time spent languishing in bugzilla, we finally made his patch stick.

Of course, now that we have this capability, we need tests to use it! Honza has written some great documentation on what is currently available via Mochitest, and how to add custom servers and certificates other things you might want. If you get motivated to write some tests and hit a rough spot, feel free as always to track me down on IRC and ask me about it.

Ideally at some point I’d like to add a CGI backend to this so you could specify a directory, and have it generate a patch against current CVS to include your test in that directory. That would lower the bar even further for getting new tests into the tree. Another cool addition would be to integrate this with my regression search buildbot (currently offline), so that you could write a mochitest and then with one click submit it to find out when something regressed. That shouldn’t be hard to do, but my buildbot needs to find a more permanent home first.

I think there’s still a lot more we can (and must) do to lower the bar for writing tests. We need all the tests we can get!