Ben's Blog

Buildbot 0.7.9 was released a couple weeks ago. Today I will be importing it into our CVS tree, upgrading the aging 0.7.7 code. Before doing an import I will branch the current tip of trunk as BUILDBOT_0_7_7_BRANCH. This will let us easily checkout 0.7.7 code and if necessary, land 0.7.7 specific fixes.

Any checkouts of mozilla/tools/buildbot on HEAD will update to the new code with ‘cvs up’. If you don’t want this to happen you should delete your checkout and get the BUILDBOT_0_7_7_BRANCH (‘cvs co -r BUILDBOT_0_7_7_BRANCH’).

Here’s some of the more exciting fixes and enhancements:

• The /buildslaves page now highlights disconnected slaves, making it easier to see that information at-a-glance
• Build properties can now be passed via a Scheduler
• Build properties can be set with ShellCommand’s through the new ‘SetProperty’ BuildStep

Don’t panic. If all goes well nothing will change from a nightly-build-user perspective. We are going to be moving to a more sane system for the generation of nightly .mar files and AUS2 snippets. Details below, but first, some background.

Rob Helmer talked a lot about AUS and updates, mostly regarding releases. What he did not mention the silly path that our nightly updates take.

All of our nightly updates currently take the following path:

1. Nightly build happens – this includes a complete MAR and complete AUS snippet
2. Cronjob on a specific build machine performs some magic and generates a partial MAR and partial AUS snippet

Once these changes land our 3.1 builds will take the following path:

1. Nightly build happens – this includes complete AND partial MAR, complete AND partial AUS snippet

(We could probably support Tinderbox driven builds (2.x, 3.x) pretty easily too, if someone wants to write the patch.)

I think it’s pretty obvious that this is a more sensible way to do things. As it stands now if we lose the VM that generates partials we lose all nightly updates. The hidden benefit here is that updates for releases and nightlies will have fewer differences between them. This means that problems with that system will be caught in the nightlies and *not* during a live release. (NB: We will still need to do snippets for older than n-1 builds, eg. 3.0->3.0.2 during the release process).

I’m still doing testing of this but I hope to land these changes in mozilla-central/tools/update-packaging by the end of next week.

Special note for build folks from SeaMonkey and Thunderbird: I will be replacing the CreateCompleteUpdateSnippet and snippet uploading ShellCommands with a couple Makefile targets – you may want to do the same.

Some of you may have noticed bug 448616. It turns out that we have not had proper breakpad symbols for OS X builds *ever* on 3.1 builds. This caused our OS X crash reports to be sigificantly less useful We’ve had a heck of a time isolating the problem. It took quite a few manual builds before we could even reproduce the problem. There were a few rounds of diagnosis involved before the problem became clear: Running ‘make package’ before ‘make buildsymbols’ breaks OS X symbols. It seems that ‘make package’ on Mac strips objdir/ppc/dist/universal, which is where we generate our symbols from!

With that knowledge in hand I deployed a fix….and unwittingly broke Linux nightlies. As part of my fix, I had changed around the order in which we do symbols, updates, and packaging. Doing so had tripped another weirdness in our build system: On Linux, ‘make -C tools/update-packaging’ (creating a full mar) *must* be done after ‘make package’. On Mac and Windows we’re able to create a mar at any point, however.

So, the correct order to do post-build processing in is thus: Symbols, Packaging, Updates.

This fix is now deployed in production and nightly builds have been re-spun. The latest Mac nightly of 3.1 should have proper breakdpad symbols (and in turn, make our crash reports useful). (Unfortunately, due to bug 454198 line numbers are only available in the raw dump, but still, this is a big improvement.) Big thanks to Lukas and Ted for all their help here. Any regressions due to this should be filed in mozilla.org:Release Engineering. \

As a side note, we ended up upgrading to Xcode 3.1 in a failed attempt to fix this bug. Included with it is GCC 4.2, which sadly bails out very early in our build process.

During this time we will be landing two things:
* Test reporting to graphs.mozilla.org for codesighs and leak tests on mozilla-central (
bug 433710)
* Support for pushing try server builds to an hg.m.o repository (details later) (bug 448014)

No downtime is expected.

If there is any reason why we shouldn’t go ahead with this please e-mail release@mozilla.com

Not much else to say – bug 437143 (try servers need to support building from a mercurial repository with a patch) has been fixed. You can now submit patches against HG repositories with a patch.

Many thanks to Vlad, who did most of the work on this.

Not much else to say – bug 437143 (try servers need to support building from a mercurial repository with a patch) has been fixed. You can now submit patches against HG repositories with a patch.

Many thanks to Vlad, who did most of the work on this.

During this time we will be moving around tinderboxes (as discussed in the “Moving Tinderboxes” thread). After this maintenance is done, the following changes will have been made:

* All Firefox 3.0.x/Gecko 1.9.0.x/CVS trunk tinderboxes will report to the soon-to-be-created “Firefox3.0″ tree. (http://tinderbox.mozilla.org/Firefox3.0/)

* All Firefox 3.1/Gecko 1.9.1/mozilla-central tinderboxes will report to the “Firefox” tree.
(http://tinderbox.mozilla.org/Firefox/)

* The ‘Mozilla2′ tree will be deleted.

As has been previous documented, we have a *lot* of VMs for Release Engineering purposes. We often need to start, stop, edit, or do other management things to them. For a long time this meant using the Virtual Infrastructure client, which is…quite buggy, and only runs on Windows. Awhile ago I discovered the VMware Web Interface, which let me start and stop VMs. Just now, I realized that I can actually edit settings, too. This means that I no longer have to start up a Windows VM to do 99% of the things I need to.

VMware, you’re web interface kicks ass. Now give me an MKS plugin for Firefox on Mac and I’m all set!

Sometime today (May 12, 2008) the Mozilla2 Buildbot will be down to enable debug+leaktest slaves.. For the curious, details of the work are in bug 422296.

If there’s any reason we can’t go ahead with this please e-mail release@mozilla.org.

On May 5, 2008 between 5am and 7am PDT the Mozilla2 and Mobile Buildbots will be down for maintenance. For the curious, details of the work are in bug 429001.

If there’s any reason we can’t go ahead with this please e-mail release@mozilla.org.