The best time to catch bugs is before they make it into a nightly build, so we’ve been working hard on infrastructure to help us detect defects as early as possible. Our most-used tool, tinderbox, has been augmented with much-improved regression testing functionality.

We now execute extensive tests after every checkin, usually more than 50 thousand or so. We’re being very inclusive about it as well, so we’ve added tests from JavaScript libraries like prototype.js. This is really handy when we’re working on touchy functions like setTimeout—the script.aculo.us tests caught a bug in one of my patches today. We’ve always had the most complete JavaScript test suite, and we’re getting closer to that ideal in the rest of the browser. Problems with bidi diacritics? Never again.

Testing can cover more than correctness, though. We now have more performance coverage than ever. We can see that our switch to Quartz on Mac OS X has improved our SVG performance quite a bit. We can see areas where our JS implementation is pretty darn fast, and we can also see areas that we need to improve.

One of the best parts of our new performance testing infrastructure is the Talos framework. We use it for a variety of synthetic benchmarks, and we also test against reference copies of popular websites. It turns out that Asian pages that look sort of like yahoo.com comprise a large proportion of those, at least accoding to the big ratings agencies. We’ve also been able to pinpoint systemic performance issues in our infrastructure with these tools. Some of them have short term fixes, some long term, but it’s only going to get better.

Leaks

One of the trickiest problems in browser engineering is integrating memory management schemes across languages. Usually, there is a host language, such as C++, C, or Objective-C, and JavaScript. In many circumstances, the host language is reference counted, while the JavaScript implementation is garbage collected (GC).

For Firefox 4, we’re moving towards GC for all but the most performance-sensitive pieces of code, using high performance code donated to the Mozilla project by Adobe/Macromedia. For Firefox 3, we’ll use a cycle collector to improve our memory management in comparison to Firefox 2.

We’re also increasing our level of automated defect detection in this area. Jesse Ruderman has been developing ever more advanced fuzzers for leak and security testing. His stuff is getting really scary. We’re also parallelizing leak testing by running leak check scripts on Amazon EC2. EC2 lets us run many parallel leak checking machines against a randomly generated selection of websites. This is great, because we find problems in pages that testers and engineers probably wouldn’t look at. For instance, www.golfballoutlet.com no longer leaks.

Whodunnit

The best part about this project is that so many people have helped. To be sure, there are contributors that dwarf the rest of us, but I think everyone who’s deeply involved in the project knows who they are. I’m more interested in the long tail of bugs assigned to people I’ve never met. Just today, Malcom Parsons fixed a table damage area bug. I don’t know Malcom, and I only have the foggiest idea of what a table damage area is (though if I need to know more, I can jump on irc and find out).

Epilogue

Lastly, I’m positive you’re interested in what I’ve been up to (yeah, yeah). Mostly I’ve been putting a lot of time into busting leaks and other performance issues. But, I do have a pet bug that’s a performance issue for a few pieces of our code, and probably lots of websites: native JSON support. This will help with the security issues that have faced JSON parsing in JS, and speed up object serialization considerably.

This feature really speeds up debugging. Overall, I’m impressed with Pylons. It provides just the right amount support. It provides support for deployment, dependencies, unit tests, and other drudgery, but lets the author choose which components to use otherwise. I’m using the Pylons controllers and routers with SQLAlchemy, Genshi, and AuthKit.

I find this setup about as productive as Rails or Django, but with reduced Kool-Aid consumption.