Ben's Blog

I’m hoping to tackle bug 505512 (Make infrastructure related problems turn the tree a color other than red) in the next few weeks. Most of the ground work for it is laid, which means that most of what I’ll be doing is parsing logs for infrastructure errors.

So, what errors do you see most from our build infrastructure? Are there other things that you would classify as infrastructure issues? Please add any suggestions you have to this Etherpad: http://etherpad.mozilla.com:9000/build-infra-errors

Do you have a random or permanent orange bug that you want to debug? Are you having trouble reproducing it? If so, please file a bug in mozilla.org:Release Engineering and we’ll get you access to a machine. If the failure is on Linux you can even bypass this, and download a copy of our ref VM here. Anyone with time and effort is welcome to this offer — you do not need to be an employee of Mozilla.

…but only those to RelEng infrastructure.

We in Release Engineering always love it when people take the time and effort to fix a bug, cleanup some code, or otherwise enhance our infrastructure. However, it’s often difficult to take outside patches because they are generally untested. Because of the nature of our code and systems – how tightly it’s tied to infrastructure, limited access to said infrastructure, and how many different systems a single change can touch – it’s nearly impossible for outside contributors to do more than the most basic testing. We don’t have a Try Server that can test patches to RelEng code, and proper testing requires many different machines and can be very time consuming – especially if you’ve never done it.

It’s always unfortunate to see patches sitting for days, weeks, or in rare cases, months, before they get landed. However, we don’t like to land untested patches because it can lead to unnecessary build bustage.

I want to fix this, so over the next few months I’m going to be prioritizing testing your patches every Monday. I will set aside my normal work for a day to help test and get ready to land contributed patches. High priority things such as releases or infrastructure problems will take precedence over this, but that shouldn’t be a common occurrence.

I’ll be keeping an eye out for things, but if you want to me ping directly about a bug please feel free to do so.

Consider this in an experimental state. I’ll be tweaking the process along the way and am very open to improvements here.

Once or twice a month a bug is filed on the try server. I see all of these go by and typically say something like “this would be cool, I don’t have time to do it”. Someone mentioned to me that they were going to blog about a specific try server to try and get some traction on it and I thought, “Why didn’t I think of that?”. So, the rest of this post is just a list of try server bugs that are relatively simple try server bugs that shouldn’t be too difficult to fix.

Bug 391364 – Tryserver should have the option of building specified platforms

Requires changes to: CGI script/UI, master.cfg/Buildbot steps

Bug 411735 – Client side form validation

Requires changes to: CGI script/UI

Bug 411735 – Support building from an alternate revision/branch/tag in a mercurial based try server build

Requires changes to: CGI script/UI, master.cfg/Buildbot steps

Bug 430942 – If bug # given, automatically add comment to bugzilla when try build finishes

Requires new Buildbot Status plugin

Bug 431375 – Try server needs to be able to handle non-Firefox builds

Requires changes to: CGI script/UI, master.cfg/Buildbot steps

Bug 431400 – Try server should support downloading bugzilla attachments as well as receiving uploaded patches

Requires changes to: CGI script/UI

I am happy to give guidance to and assist anyone who wants to work on any of these bugs. The CGI/UI can be found here: http://lxr.mozilla.org/mozilla/source/webtools/buildbot-try, the buildbot master.cfg is here: http://lxr.mozilla.org/mozilla/source/tools/buildbot-configs/tryserver/, and classes supporting it are here: http://mxr.mozilla.org/mozilla/source/tools/buildbotcustom/tryserver/.