Ben's Blog

Mozilla Scheduled Downtime for Tinderbox Moving - 07/04/2008, 5am - 10am PDT (approx)

bhearsum — Thu, 03 Jul 2008 17:34:57 +0000

During this time we will be moving around tinderboxes (as discussed in the “Moving Tinderboxes” thread). After this maintenance is done, the following changes will have been made:

* All Firefox 3.0.x/Gecko 1.9.0.x/CVS trunk tinderboxes will report to the soon-to-be-created “Firefox3.0″ tree. (http://tinderbox.mozilla.org/Firefox3.0/)

* All Firefox 3.1/Gecko 1.9.1/mozilla-central tinderboxes will report to the “Firefox” tree.
(http://tinderbox.mozilla.org/Firefox/)

* The ‘Mozilla2′ tree will be deleted.

VMware ESX

bhearsum — Fri, 06 Jun 2008 13:45:53 +0000

As has been previous documented, we have a *lot* of VMs for Release Engineering purposes. We often need to start, stop, edit, or do other management things to them. For a long time this meant using the Virtual Infrastructure client, which is…quite buggy, and only runs on Windows. Awhile ago I discovered the VMware Web Interface, which let me start and stop VMs. Just now, I realized that I can actually edit settings, too. This means that I no longer have to start up a Windows VM to do 99% of the things I need to.

VMware, you’re web interface kicks ass. Now give me an MKS plugin for Firefox on Mac and I’m all set!

Mozilla2 Buildbot Downtime - Sometime today (May 12, 2008)

bhearsum — Mon, 12 May 2008 12:25:22 +0000

Sometime today (May 12, 2008) the Mozilla2 Buildbot will be down to enable debug+leaktest slaves.. For the curious, details of the work are in bug 422296.

If there’s any reason we can’t go ahead with this please e-mail release@mozilla.org.

Mozilla2, Mobile Buildbot Downtime - May 5, 2008, 5am - 7am PDT

bhearsum — Thu, 01 May 2008 12:29:06 +0000

On May 5, 2008 between 5am and 7am PDT the Mozilla2 and Mobile Buildbots will be down for maintenance. For the curious, details of the work are in bug 429001.

If there’s any reason we can’t go ahead with this please e-mail release@mozilla.org.

You! Can help make the Try Server better.

bhearsum — Wed, 30 Apr 2008 21:36:16 +0000

Once or twice a month a bug is filed on the try server. I see all of these go by and typically say something like “this would be cool, I don’t have time to do it”. Someone mentioned to me that they were going to blog about a specific try server to try and get some traction on it and I thought, “Why didn’t I think of that?”. So, the rest of this post is just a list of try server bugs that are relatively simple try server bugs that shouldn’t be too difficult to fix.

Bug 391364 - Tryserver should have the option of building specified platforms

Requires changes to: CGI script/UI, master.cfg/Buildbot steps

Bug 411735 - Client side form validation

Requires changes to: CGI script/UI

Bug 411735 - Support building from an alternate revision/branch/tag in a mercurial based try server build

Requires changes to: CGI script/UI, master.cfg/Buildbot steps

Bug 430942 - If bug # given, automatically add comment to bugzilla when try build finishes

Requires new Buildbot Status plugin

Bug 431375 - Try server needs to be able to handle non-Firefox builds

Requires changes to: CGI script/UI, master.cfg/Buildbot steps

Bug 431400 - Try server should support downloading bugzilla attachments as well as receiving uploaded patches

Requires changes to: CGI script/UI

I am happy to give guidance to and assist anyone who wants to work on any of these bugs. The CGI/UI can be found here: http://lxr.mozilla.org/mozilla/source/webtools/buildbot-try, the buildbot master.cfg is here: http://lxr.mozilla.org/mozilla/source/tools/buildbot-configs/tryserver/, and classes supporting it are here: http://mxr.mozilla.org/mozilla/source/tools/buildbotcustom/tryserver/.

Buildbot + Mozilla

bhearsum — Wed, 30 Apr 2008 00:41:27 +0000

Over the past year or so Mozilla has increasingly relied on Buildbot for our Building, Releasing, Testing, and l10n processes. For awhile, everyone that touched Buildbot did things their own way. As a result we had a ton of code (and bug) duplication. There is now a heavy push within the Release Engineering group to rectify this. We’ve slowly but surely been moving code into our “buildbotcustom”. This module contains many different things that are not appropriate to put in upstream-Buildbot, but are very appropriate and useful to those working with Mozilla. We’ve done our best to keep things as generic as possible. Some things (tryserver and unittest) were moved as-is into buildbotcustom to make it easier to generalize them in the future.

The rest of this post is aimed towards people who want to build Mozilla products with Buildbot and talks about some of the custom classes in more detail.

Change Sources

TinderboxPoller

TinderboxPoller watches a Tinderbox quickparse file (such as http://tinderbox.mozilla.org/Firefox/quickparse.txt) for new builds from a specific builder (each column on the Tinderbox waterfall is a builder). When a new build is detected it kicks off a “build” (which isn’t necessarily a compilation process) on the specified “branch” (which in this case is simply a string that matches the TinderboxPoller to a specific builder or builders. For example, we have a TinderboxPoller that watches fx-linux-tbox for new builds. Whenever it sees one it kicks off Talos runs (”builds”) on a number of different machines. To the best of my knowledge, the only thing we use this for is Talos - but there may be things I’m unaware of.

Build Steps

TinderboxShellCommand

This is a BuildStep that simply executes a regular Buildbot ShellCommand and ignores the result. Those familiar with Tinderbox client will understand the naming of it.

GetBuildID

This step attempts to pull the BuildID from the objdir of a build and set it as a BuildProperty. It does so through printconfigsetting.py and application.ini. If you are using an objdir you must pass the path to it, relative to the builddir. If you are not using one you do not need to pass anything. Once this step is run you can refer to it through WithProperties.

SetMozillaBuildProperties

This step does the job of GetBuildID and more (oh so much more). However, it depends on a slave-side command that is only in Mozilla’s Buildbot tree. To use it, you must be using Buildbot checked out from mozilla/tools/buildbot. (Technically, you could copy and paste the slave command into an upstream Buildbot, but let’s keep it simple). In addition to gathering the BuildID and setting it as a BuildProperty it does the same for the following: appVersion (from application.ini, falling back on platform.ini), the filename, size (in bytes), and sha1 hash of the package, installer, and complete mar are gathered. Please see the in-code documentation for the names of the BuildProperties. Obviously, this step should not be run until the package and/or installer and/or complete mar are created, if you care about their metadata.

As with GetBuildID it needs to know where the objdir is rooted (relative to builddir). Again, if there is no objdir you can simply ignore this parameter.

MozillaStageUpload

(I’m not sure how useful this is to non-Firefox projects as it contains some business logic that may be specific to Fx, but it performs an important function so I figure I’ll include it here.)

This is a bit of a kludgy class as it ends up generating one huge line of bash that gets executed. It was written to be a replacement for the
Tinderbox client ‘pushit()’ function. It’s job is simple: upload packages and mars to a server (for example, stage.mozilla.org). Because it was written to replicate some Tinderbox logic it’s got a lot of different options that I don’t need to go through here. The in-code documentation for it is quite complete and available here: http://mxr.mozilla.org/mozilla/source/tools/buildbotcustom/steps/transfer.py

One note: your BuildSlaves will need passwordless logins to remoteHost with the ssh host key accepted, or else this step will hang. You can test this by doing ’ssh username@remoteHost’ on your slaves.

In the coming weeks I intend to write a few more posts about Mozilla’s Buildbot usage.

[UPDATED] Try Server, Mozilla2, Mobile Downtime - April 21, 2008, 5am - 8am PDT

bhearsum — Fri, 18 Apr 2008 13:30:51 +0000

On April 21, 2008 between 5am and 7am PDT the Try Server will be down for maintenance. For the curious, details of the work are in bug 428308.

The Mozilla2 and Mobile Buildbots will also be down for migration to a new host. Details of this work are in bug 429562.

If there’s any reason we can’t go ahead with this please e-mail release@mozilla.org.

Nightly updates on Mozilla2 builds.

bhearsum — Fri, 18 Apr 2008 12:46:24 +0000

As of today, anyone running a mozilla-central nightly build (all zero of you) will start receiving nightly updates for it. There isn’t a whole lot going on there right now but this is another step is getting 1.9 parity on Mozilla2. Many many thanks to nthomas for teaching me so much about the update system while getting this going.

At some point in the future we plan to enable updates for other branches as well, such as actionmonkey.

Try Server Downtime - April 21, 2008, 5am - 7am PDT

bhearsum — Thu, 17 Apr 2008 14:20:36 +0000

On April 21, 2008 between 5am and 7am PDT the Try Server will be down for maintenance. For the curious, details of the work are in bug 428308. If there’s any reason we can’t go ahead with this please e-mail release@mozilla.org.

Try Server Symbol Server

bhearsum — Tue, 08 Apr 2008 18:28:18 +0000

As of earlier today, the try server began uploading windows debugging symbols to a symbol server. For those of you interested in debugging your Windows try server builds you can follow the same instructions listed on devmo replacing http://symbols.mozilla.org/firefox with http://build.mozilla.org/tryserver-symbols. (For posterity, this information is also recorded on the try server wiki page. Any problems discovered should be filed (as always) in Webtools:Try Server.