I want to test your patches for you
April 7th, 2009
…but only those to RelEng infrastructure.
We in Release Engineering always love it when people take the time and effort to fix a bug, cleanup some code, or otherwise enhance our infrastructure. However, it’s often difficult to take outside patches because they are generally untested. Because of the nature of our code and systems – how tightly it’s tied to infrastructure, limited access to said infrastructure, and how many different systems a single change can touch – it’s nearly impossible for outside contributors to do more than the most basic testing. We don’t have a Try Server that can test patches to RelEng code, and proper testing requires many different machines and can be very time consuming – especially if you’ve never done it.
It’s always unfortunate to see patches sitting for days, weeks, or in rare cases, months, before they get landed. However, we don’t like to land untested patches because it can lead to unnecessary build bustage.
I want to fix this, so over the next few months I’m going to be prioritizing testing your patches every Monday. I will set aside my normal work for a day to help test and get ready to land contributed patches. High priority things such as releases or infrastructure problems will take precedence over this, but that shouldn’t be a common occurrence.
I’ll be keeping an eye out for things, but if you want to me ping directly about a bug please feel free to do so.
Consider this in an experimental state. I’ll be tweaking the process along the way and am very open to improvements here.
April 7th, 2009 at 2:12 pm
Great to hear, Ben!
I hope this announcement comes with support from management; these are the types of initiatives I had always wanted to undertake, but it was, lamentably, never a viable option.
April 7th, 2009 at 3:59 pm
Awesome, well done.
April 7th, 2009 at 5:37 pm
I’d love to see you spending a bit of those Mondays to make the build infra more accessible.
BuildSlaves.py is one example. You can’t even run buildbot checkconfig locally without trying to make up some file that doesn’t even have a sample in the upstream repo.
The staging dirs in the buildbot-configs repo is another example. We should get to better factorization there. I still think that a try repo for that is better than some dirs with config changes that are mostly stuff that could be factored into BuildSlaves.py. Like, the slaves are all in BuildSlaves, but which slave belong to which build pool is still in master.cfg, and varies from staging to staging2 to production. The pools could be nicely configured inside BuildSlaves.py. I spearheaded that in my l10n master v2, which has the slaves for my builders encoded in http://hg.mozilla.org/users/axel_mozilla.com/l10n-master/file/e6c4bf4e3699/BuildSlaves.py. Trivial sample, yeah, and I really don’t want parallelism in that one, but you get the idea.
If we get the setup data factored out, we might be able to drop the staging dirs alltogether. Like, we don’t have staging for buildbotcustom, too. Which kinda confuses me.
If people wanted to do real testing locally, being able to just focus testing on particular features would be cool. And more tests and testing environments would help. I try (and more often than not fail) to write tests for my code. I just realized today that once again my builds pass, but my tests don’t. But there are ways to set up scenarios which can significantly help in testing off-site and off-stage.
April 7th, 2009 at 5:46 pm
Ben, this feels like an excellent way to get more people up to speed with releng tools, and is probably more scalable than people just delivering requirements for you to implement/test/get r+’d
I was just catching up on newsgroups and shaver mentioned that in the Cairo project the reviewers tend to do things like style fixes, adds tests, etc. instead of r-’ing and bouncing back to contributors, and that it makes for a better experience. Your “test Monday” idea seems very much in the same spirit.
I am very interested in hearing about your experiences with this, and I’ll bet it would encourage others to do this sort of thing as well.
April 8th, 2009 at 1:37 pm
@Axel
That sort of clean is is definitely something I want to do. It’s not a small task though, and isn’t going to happen in the next few months. Despite the state things are in, people still throw patches our way, and I hope what I am doing this quarter will encourage that.
April 8th, 2009 at 1:43 pm
@Everyone
Thanks for the encouragement!