Ben's Blog » tinderbox

Which build infrastructure problems do you see the most?

bhearsum — Fri, 13 Aug 2010 19:31:03 +0000

I’m hoping to tackle bug 505512 (Make infrastructure related problems turn the tree a color other than red) in the next few weeks. Most of the ground work for it is laid, which means that most of what I’ll be doing is parsing logs for infrastructure errors.

So, what errors do you see most from our build infrastructure? Are there other things that you would classify as infrastructure issues? Please add any suggestions you have to this Etherpad: http://etherpad.mozilla.com:9000/build-infra-errors

Update on recent Tinderbox issues

bhearsum — Thu, 12 Aug 2010 14:11:52 +0000

My last post talked about the issues we’ve been having with load on the Tinderbox server and some ways we could fix it. I’m happy to report that two things were completed yesterday that should keep the load under control for the foreseeable future.

One of the things mentioned in my previous post, splitting incoming build processing from the rest of Tinderbox (bug 585691), was completed very late last night. Additionally, Nick Thomas discovered that we had lost the cronjob that takes care of cleaning out old builds from Tinderbox’s memory. That script was re-enabled and a one time clean up removed 64GB of old build data. Both of these were completed around 4am PDT this morning and load is looking much better.

Especially because we’re now running cleanup scripts on a regular basis again, I believe that this should get as back to good state.

Everyone should feel free to send justdave their thanks for staying up reaaaaallllly late last night to get us back to a good state.

Recent Tinderbox issues

bhearsum — Tue, 10 Aug 2010 15:36:31 +0000

As many of you know there have been numerous times lately that Tinderbox has become unresponsive, sometimes to the point of going down completely for a period of time. This post will attempt to summarize the issues and what’s being done about them.

The biggest issue is load (surprise!). In a period of a few years we’ve gone from a few active trees with tens of columns between them to tens of active trees with hundreds of columns between them. Unsurprisingly, this has made the Tinderbox server a lot busier. The biggest load items are:

showlog.cgi – Shows a log file for a specific build
showbuilds.cgi – Shows the main page for a tree (like this)
processbuilds.pl – Processes incoming “build complete” mail

A bit of profiling has also been done in bug 585814 to try to find specific hotspots.

We’ve already done a few things to help with Tinderbox load:

Other ways we’re looking at improving the situation:

bug 585691 – Split up Tinderbox data processing from display. This wouldn’t reduce overall load, but it should segregate it enough to keep the Tinderbox display up.
bug 390341 – Pregenerate brief and full logs. This would eliminate the need for showlog.cgi to uncompress logs in most cases.
bug 530318 – Put full logs on FTP server; stop serving them from Tinderbox.

Bring your problems to me, this week

bhearsum — Mon, 23 Feb 2009 15:16:07 +0000

It’s the start of a new week and the start of my first shift as RelEng sheriff. Do you suspect rando-orange? Talk to me. Do you suspect machine problems? Talk to me. I’ll be watching dev.tree-management, the releng triage queue, and #developers for issues.

Mozilla Scheduled Downtime for Tinderbox Moving – 07/04/2008, 5am – 10am PDT (approx)

bhearsum — Thu, 03 Jul 2008 17:34:57 +0000

During this time we will be moving around tinderboxes (as discussed in the “Moving Tinderboxes” thread). After this maintenance is done, the following changes will have been made:

* All Firefox 3.0.x/Gecko 1.9.0.x/CVS trunk tinderboxes will report to the soon-to-be-created “Firefox3.0″ tree. (http://tinderbox.mozilla.org/Firefox3.0/)

* All Firefox 3.1/Gecko 1.9.1/mozilla-central tinderboxes will report to the “Firefox” tree.
(http://tinderbox.mozilla.org/Firefox/)

* The ‘Mozilla2′ tree will be deleted.

Mozilla Tinderbox Downtime – January 8, 2008, 5am – 8am PST

bhearsum — Mon, 07 Jan 2008 18:58:39 +0000

On January 8, 2008 between the hours of 5am and 8am PST the Firefox and l10n Tinderboxes will be down.

The Win32 build machines for these tinderboxes are set to be upgraded to the most recent version. Details are in Bug 402848.If there’s any reason we can’t go ahead with this tomorrow morning, please e-mail build@mozilla.org.