Ben's Blog

Today is a great day. bug 421534 has landed (thanks Ted!). From here on in Mac symbols will be generated in DWARF format rather than stabs. This may sound pretty lame and unworthy of a blog post, but our Mac build machines sure are happy about it. Due to a bug in Xcode 3.0/3.1 building Mozilla results in a GCC crash when passing ‘-gstabs’. The fix? Pass ‘–save-temps’. Why’s that so bad, you may ask? Well, when you save all temporary files from a Mozilla build it adds up to quite a bit (20GB for a universal build). There’s also a possibility that this slows us a non-trivial amount.

Disk space has been getting pretty tight with all of these active codelines. Things are looking a little better going forward.

Hi All,

Due to a few reasons we were unable to migrate our Buildbot master to
faster disks yesterday. Instead, we will be doing it tomorrow morning
PST. We will be closing all trees at 5am tomorrow to shut down and
migrate this machine. We expect it to take a couple of hours to complete.

This affects the following trees:
* Firefox
* Firefox3.0
* Firefox3.1

- Ben

a – http://lxr.mozilla.org/mozilla/source/browser/locales/all-locales
b – http://hg.mozilla.org/build/tools
c – http://hg.mozilla.org/build/buildbot-configs/index.cgi/file/tip/mozilla2/config.py
d – https://download.mozilla.org/admin/login.php
e – https://intranet.mozilla.org/index.php?title=Build:Meetings:Latest&action=edit
f – http://tinderbox.mozilla.org/Firefox/
g – https://mail.google.com/mail/
h – http://hrpassport.com/
i – https://build.inventory.mozilla.org/build/
j – http://people.mozilla.org/~johnath/pdb/
k – http://www.cbc.ca/ (WTF?)
l – http://mxr.mozilla.org/mozilla/source/tools/buildbotcustom/l10n/l10n.py
m – http://hg.mozilla.org/mozilla-central/
n – https://nagios.mozilla.org/nagios/cgi-bin/status.cgi?host=all
o – http://osnews.com/
p – http://production-1.9-master.build.mozilla.org:8810/waterfall
q – http://qm-rhel02:2006/
r – http://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla2-staging/release_config.py
s – http://slashdot.org/
t – http://mxr.mozilla.org/mozilla/source/tools/buildbot/buildbot/status/tinderbox.py
u – https://10.2.72.6/ui/ (vmware ESX web UI)
v – http://hg.mozilla.org/build/tools/index.cgi/file/tip/release/version-bump.pl
w – http://wiki.mozilla.org/WeeklyUpdates
x – http://xkcd.com/
y – http://youtube.com/
z – http://www.zipcar.com/reservations

(Please direct responses to mozilla.dev.planning)

Hi All,

To follow-up on my post yesterday, here’s then when and how of branching:
* Tag mozilla-central and l10n repositories with GECKO_1_9_1_BASE so we know where we branched going forward
* File IT bugs to have mozilla-central, l10n repositories cloned and a Firefox 3.1 Tinderbox created.
* While this is happening RelEng will do as much setup of infrastructure as possible.
* Once the clones are finished we will do some tests in our staging environment.
* When were satisfied with those results we will start push those infrastructure changes to production.

ETA on being fully up and running (that is to say: parity with mozilla-central) is early in the PST day Monday, November 24th. We do expect to have infrastructure up this week but some baking time, especially for Talos, is important.

Other important information:
* mozilla-central will NOT be used for trunk development until after we tag for Firefox 3.1b2. Once that happens we will bump mozilla-central versions to 1.9.2a1pre/3.2a1pre.
* The new mozilla-1.9.1 repository will be rebranded to Shiretoko to avoid confusion

https://bugzilla.mozilla.org/show_bug.cgi?id=464640 is the tracking bug for branching, for those interested in tracking the blow-by-blow.

During this time we will be upgrading both the Firefox 3 and Firefox 3.1 Unit test Buildbots to a newer version (0.7.9). In order to avoid interrupting running builds we will be closing the tree at 4am PDT and stopping any new builds from being scheduled. Once all builds have finished we will perform the upgrade and open the tree again. Depending on the timing this could take anywhere from 20 minutes to a few hours. The tree should be open again no later than 7am PDT.

If there is any reason why we shouldn’t go ahead with this please e-mail release@mozilla.com

Buildbot 0.7.9 was released a couple weeks ago. Today I will be importing it into our CVS tree, upgrading the aging 0.7.7 code. Before doing an import I will branch the current tip of trunk as BUILDBOT_0_7_7_BRANCH. This will let us easily checkout 0.7.7 code and if necessary, land 0.7.7 specific fixes.

Any checkouts of mozilla/tools/buildbot on HEAD will update to the new code with ‘cvs up’. If you don’t want this to happen you should delete your checkout and get the BUILDBOT_0_7_7_BRANCH (‘cvs co -r BUILDBOT_0_7_7_BRANCH’).

Here’s some of the more exciting fixes and enhancements:

• The /buildslaves page now highlights disconnected slaves, making it easier to see that information at-a-glance
• Build properties can now be passed via a Scheduler
• Build properties can be set with ShellCommand’s through the new ‘SetProperty’ BuildStep

1. Take a picture of yourself right now.
2. Don’t change your clothes, don’t fix your hair…just take a picture.
3. Post that picture with NO editing.
4. Post these instructions with your picture.

Via Rob.

Don’t panic. If all goes well nothing will change from a nightly-build-user perspective. We are going to be moving to a more sane system for the generation of nightly .mar files and AUS2 snippets. Details below, but first, some background.

Rob Helmer talked a lot about AUS and updates, mostly regarding releases. What he did not mention the silly path that our nightly updates take.

All of our nightly updates currently take the following path:

1. Nightly build happens – this includes a complete MAR and complete AUS snippet
2. Cronjob on a specific build machine performs some magic and generates a partial MAR and partial AUS snippet

Once these changes land our 3.1 builds will take the following path:

1. Nightly build happens – this includes complete AND partial MAR, complete AND partial AUS snippet

(We could probably support Tinderbox driven builds (2.x, 3.x) pretty easily too, if someone wants to write the patch.)

I think it’s pretty obvious that this is a more sensible way to do things. As it stands now if we lose the VM that generates partials we lose all nightly updates. The hidden benefit here is that updates for releases and nightlies will have fewer differences between them. This means that problems with that system will be caught in the nightlies and *not* during a live release. (NB: We will still need to do snippets for older than n-1 builds, eg. 3.0->3.0.2 during the release process).

I’m still doing testing of this but I hope to land these changes in mozilla-central/tools/update-packaging by the end of next week.

Special note for build folks from SeaMonkey and Thunderbird: I will be replacing the CreateCompleteUpdateSnippet and snippet uploading ShellCommands with a couple Makefile targets – you may want to do the same.

Some of you may have noticed bug 448616. It turns out that we have not had proper breakpad symbols for OS X builds *ever* on 3.1 builds. This caused our OS X crash reports to be sigificantly less useful We’ve had a heck of a time isolating the problem. It took quite a few manual builds before we could even reproduce the problem. There were a few rounds of diagnosis involved before the problem became clear: Running ‘make package’ before ‘make buildsymbols’ breaks OS X symbols. It seems that ‘make package’ on Mac strips objdir/ppc/dist/universal, which is where we generate our symbols from!

With that knowledge in hand I deployed a fix….and unwittingly broke Linux nightlies. As part of my fix, I had changed around the order in which we do symbols, updates, and packaging. Doing so had tripped another weirdness in our build system: On Linux, ‘make -C tools/update-packaging’ (creating a full mar) *must* be done after ‘make package’. On Mac and Windows we’re able to create a mar at any point, however.

So, the correct order to do post-build processing in is thus: Symbols, Packaging, Updates.

This fix is now deployed in production and nightly builds have been re-spun. The latest Mac nightly of 3.1 should have proper breakdpad symbols (and in turn, make our crash reports useful). (Unfortunately, due to bug 454198 line numbers are only available in the raw dump, but still, this is a big improvement.) Big thanks to Lukas and Ted for all their help here. Any regressions due to this should be filed in mozilla.org:Release Engineering. \

As a side note, we ended up upgrading to Xcode 3.1 in a failed attempt to fix this bug. Included with it is GCC 4.2, which sadly bails out very early in our build process.

During this time we will be landing two things:
* Test reporting to graphs.mozilla.org for codesighs and leak tests on mozilla-central (
bug 433710)
* Support for pushing try server builds to an hg.m.o repository (details later) (bug 448014)

No downtime is expected.

If there is any reason why we shouldn’t go ahead with this please e-mail release@mozilla.com