Ben’s Blog

(Please direct responses to mozilla.dev.planning)

Hi All,

To follow-up on my post yesterday, here’s then when and how of branching:
* Tag mozilla-central and l10n repositories with GECKO_1_9_1_BASE so we know where we branched going forward
* File IT bugs to have mozilla-central, l10n repositories cloned and a Firefox 3.1 Tinderbox created.
* While this is happening RelEng will do as much setup of infrastructure as possible.
* Once the clones are finished we will do some tests in our staging environment.
* When were satisfied with those results we will start push those infrastructure changes to production.

ETA on being fully up and running (that is to say: parity with mozilla-central) is early in the PST day Monday, November 24th. We do expect to have infrastructure up this week but some baking time, especially for Talos, is important.

Other important information:
* mozilla-central will NOT be used for trunk development until after we tag for Firefox 3.1b2. Once that happens we will bump mozilla-central versions to 1.9.2a1pre/3.2a1pre.
* The new mozilla-1.9.1 repository will be rebranded to Shiretoko to avoid confusion

https://bugzilla.mozilla.org/show_bug.cgi?id=464640 is the tracking bug for branching, for those interested in tracking the blow-by-blow.

During this time we will be upgrading both the Firefox 3 and Firefox 3.1 Unit test Buildbots to a newer version (0.7.9). In order to avoid interrupting running builds we will be closing the tree at 4am PDT and stopping any new builds from being scheduled. Once all builds have finished we will perform the upgrade and open the tree again. Depending on the timing this could take anywhere from 20 minutes to a few hours. The tree should be open again no later than 7am PDT.

If there is any reason why we shouldn’t go ahead with this please e-mail release@mozilla.com

Buildbot 0.7.9 was released a couple weeks ago. Today I will be importing it into our CVS tree, upgrading the aging 0.7.7 code. Before doing an import I will branch the current tip of trunk as BUILDBOT_0_7_7_BRANCH. This will let us easily checkout 0.7.7 code and if necessary, land 0.7.7 specific fixes.

Any checkouts of mozilla/tools/buildbot on HEAD will update to the new code with ‘cvs up’. If you don’t want this to happen you should delete your checkout and get the BUILDBOT_0_7_7_BRANCH (’cvs co -r BUILDBOT_0_7_7_BRANCH’).

Here’s some of the more exciting fixes and enhancements:

• The /buildslaves page now highlights disconnected slaves, making it easier to see that information at-a-glance
• Build properties can now be passed via a Scheduler
• Build properties can be set with ShellCommand’s through the new ‘SetProperty’ BuildStep

1. Take a picture of yourself right now.
2. Don’t change your clothes, don’t fix your hair…just take a picture.
3. Post that picture with NO editing.
4. Post these instructions with your picture.

Via Rob.

Don’t panic. If all goes well nothing will change from a nightly-build-user perspective. We are going to be moving to a more sane system for the generation of nightly .mar files and AUS2 snippets. Details below, but first, some background.

Rob Helmer talked a lot about AUS and updates, mostly regarding releases. What he did not mention the silly path that our nightly updates take.

All of our nightly updates currently take the following path:

1. Nightly build happens – this includes a complete MAR and complete AUS snippet
2. Cronjob on a specific build machine performs some magic and generates a partial MAR and partial AUS snippet

Once these changes land our 3.1 builds will take the following path:

1. Nightly build happens – this includes complete AND partial MAR, complete AND partial AUS snippet

(We could probably support Tinderbox driven builds (2.x, 3.x) pretty easily too, if someone wants to write the patch.)

I think it’s pretty obvious that this is a more sensible way to do things. As it stands now if we lose the VM that generates partials we lose all nightly updates. The hidden benefit here is that updates for releases and nightlies will have fewer differences between them. This means that problems with that system will be caught in the nightlies and *not* during a live release. (NB: We will still need to do snippets for older than n-1 builds, eg. 3.0->3.0.2 during the release process).

I’m still doing testing of this but I hope to land these changes in mozilla-central/tools/update-packaging by the end of next week.

Special note for build folks from SeaMonkey and Thunderbird: I will be replacing the CreateCompleteUpdateSnippet and snippet uploading ShellCommands with a couple Makefile targets – you may want to do the same.

Some of you may have noticed bug 448616. It turns out that we have not had proper breakpad symbols for OS X builds *ever* on 3.1 builds. This caused our OS X crash reports to be sigificantly less useful We’ve had a heck of a time isolating the problem. It took quite a few manual builds before we could even reproduce the problem. There were a few rounds of diagnosis involved before the problem became clear: Running ‘make package’ before ‘make buildsymbols’ breaks OS X symbols. It seems that ‘make package’ on Mac strips objdir/ppc/dist/universal, which is where we generate our symbols from!

With that knowledge in hand I deployed a fix….and unwittingly broke Linux nightlies. As part of my fix, I had changed around the order in which we do symbols, updates, and packaging. Doing so had tripped another weirdness in our build system: On Linux, ‘make -C tools/update-packaging’ (creating a full mar) *must* be done after ‘make package’. On Mac and Windows we’re able to create a mar at any point, however.

So, the correct order to do post-build processing in is thus: Symbols, Packaging, Updates.

This fix is now deployed in production and nightly builds have been re-spun. The latest Mac nightly of 3.1 should have proper breakdpad symbols (and in turn, make our crash reports useful). (Unfortunately, due to bug 454198 line numbers are only available in the raw dump, but still, this is a big improvement.) Big thanks to Lukas and Ted for all their help here. Any regressions due to this should be filed in mozilla.org:Release Engineering. \

As a side note, we ended up upgrading to Xcode 3.1 in a failed attempt to fix this bug. Included with it is GCC 4.2, which sadly bails out very early in our build process.

During this time we will be landing two things:
* Test reporting to graphs.mozilla.org for codesighs and leak tests on mozilla-central (
bug 433710)
* Support for pushing try server builds to an hg.m.o repository (details later) (bug 448014)

No downtime is expected.

If there is any reason why we shouldn’t go ahead with this please e-mail release@mozilla.com

Not much else to say – bug 437143 (try servers need to support building from a mercurial repository with a patch) has been fixed. You can now submit patches against HG repositories with a patch.

Many thanks to Vlad, who did most of the work on this.

Not much else to say – bug 437143 (try servers need to support building from a mercurial repository with a patch) has been fixed. You can now submit patches against HG repositories with a patch.

Many thanks to Vlad, who did most of the work on this.

During this time we will be moving around tinderboxes (as discussed in the “Moving Tinderboxes” thread). After this maintenance is done, the following changes will have been made:

* All Firefox 3.0.x/Gecko 1.9.0.x/CVS trunk tinderboxes will report to the soon-to-be-created “Firefox3.0″ tree. (http://tinderbox.mozilla.org/Firefox3.0/)

* All Firefox 3.1/Gecko 1.9.1/mozilla-central tinderboxes will report to the “Firefox” tree.
(http://tinderbox.mozilla.org/Firefox/)

* The ‘Mozilla2′ tree will be deleted.