Maggot Brain

L10n ecosystem in a fishbowl

Building the infrastructure for our l10n builds is hard, mostly because it’s consisting of a ton of things that you don’t have control over. We’re building 3 and a half applications, Firefox, Thunderbird, Fennec, and Sunbird for calendar. Firefox is built on three versions, one of which is still coming from CVS. Thunderbird is one version on CVS, one on hg. We’re touching some 170 hg repos, and a single check-in can do anything between no build, one build, or up to almost 200 builds. Rinse, repeat, yeah, 200 builds for a single landing. Worse than that, you don’t have any control over who’s landing what when where. Bottom line, you really can’t test a change in the l10n build automation reliably in our production setup.

You can create a fake ecosystem, though, and I’ll explain a bit how that works. Of course it doesn’t end up being trivial, but it’s contained. It’s not trying to cover the CVS branches, those would require a setup of bonsai, which I chicken away from. Take this post with a grain of salt, I assume there are some errors here as most of it is typed from memory.

As with any recipe, here are the list of ingredients:

• A set of hg repositories, both for a fake application and some fake l10n repos.
• An hg server serving those repositories (make that port 8001).
• Some buildbot infrastructure working on top of these repositories that you’re trying to test.
• Possibly an instance of the l10n dashboard presenting both the build and the l10n data.

The initial chunk is creating the repositories. I created a helper script create-stage that does that, which is part of buildbotcustom/bin/l10n-stage. It’s main purpose is to get the templates, hooks etc that are part of our server-side setup on hg.mozilla.org, create some upstream repos for en-US and l10n, and push some initial content from a set of working clones. You call it like

python l10n-stage -l stagedir

The -l keeps the l10n repositories from pushing their initial content, which yields a scenario that is closer to what we have upstream, i.e., a flock of en-US pushes before the l10n repos start. This command creates a bunch of main repositories in the repos subdir of stagedir, and a bunch of working clones in workdir. It also creates a webdir.conf, that you’ll use to run the local hg http server. Let’s run that now, in stagedir:

hg serve -p 8001 --webdir-conf=webdir.conf -t repos/hg_templates

Now you have a local setup of a application repository called mozilla, and 4 localizations in l10n, ab, de, ja-JP-mac, x-testing. They’re all equipped with the same hooks that we run on hg.m.o, in particular, they support pushlog.

Now on to the buildbot infrastructure. There’s a sibling script to create-stage, create-buildbot, which should create a master setup that is rather close to what we run on releng. It supports various degrees of parallelism for multiple slaves on three platforms, does only dummy builds, though. IIRC. I want to go into more detail on how to set up the new dashboard master, though.

The dashboard master is merely running compare-locales on the actual source repositories. It does come with our bonsai replacement pushes, though. That’s basically a pushlog spanning repositories, including file and branch indexing. Here’s the basic software components you’ll need:

• django 1.0.x (1.1pre might work, too)
• buildbot 0.7.10p1, older versions won’t work

and from hg.m.o, you’ll need compare-locales, locale-inspector, l10n-master, buildbotcustom and django-site.

Firstly, you set up the db. sqlite and mysql should both work, mysql is actively tested. Edit the various settings.py files to reference your db, with an absolute path if sqlite, and create the schema. The main entry point to the django site is l10n_site, go in there to run python manage.py syncdb. Another edit you want to do is to point REPOSITORY_BASE to a dir where you can stage another set of clones of your repos. I suggest to not share the hg master repo dir here.

Next, create a buildbot master and a slave. You do that by running the buildbot create-master command on your local clone of l10n-master. You’ll need to adapt l10nbuilds.ini to the test set up,

[test]
app = browser
type = hg
locales = all
mozilla = mozilla
l10n = l10n
repo = http://localhost:8001/
l10n.ini = browser/locales/l10n.ini
builders = compare

I should put that into the repo somewhere, I guess.

Setting up the slave is trivial, you need to make sure it’s on the same machine, though. It will run on the django clones directly.

Before starting master and slave, make sure that all the deps are in your PYTHONPATH.

Last but not least, you need to get all the pushes from your repo setup into your django setup. First, you need to tell the db which repositories it’s supposed to get from where. I created sample data for the test app, which you can load by

python manage.py loaddata localhost

The repositories I use for the production environment are in hg_mozilla_org, fwiw. You fill the database with the actual push data by running a twisted daemon. Inside django-site, run

twistd get-pushes -n -t 1 -s l10n_site.settings

That will ping one repo per second, not update, with data from l10n_site.settings. Now you have everything set up, and you can start to edit en-US and l10n files in your workingdir, and push, and see how that changes your builds and dashboard.

The buildbot waterfall will be on port 8364, and with python manage.py runserver, the dashboard will show up on port 8000. None of this is setup to be on a production server at this point, but it’s good for testing.

Update: Forgot to mention that you need to bootstrap the repo lists.

The folks we know

Seth has blogged his response to Walt Mossberg’s interview with Mitchell and John, on top of the already great answer the two provided in the interview on

Why wouldn’t it just be better for the consumer to go with the company that’s hired experts to do its translations?

and here’s my take.

The essential point to me is that we do get experts. We get experts in localizing Firefox. And we get experts we know. The people that spend their free time and passion to create Firefox in their language don’t do it because they can lip sync Shakespeare into Portuguese, but because they share the values and mission that make Firefox Firefox. They care about the experience that people have with the web in their language.

The reason why our localizations rock is that our localizers know us, and we know them, and that we actually speak the same language.

Thoughts on killing tinderbox, foundations

I figured it’d be a good idea to just dump my thinkings on killing tinderbox (as in the the web interface to mozilla’s builds). As just the background information grows to a lengthy post, I’ll cut them into pieces.

To point you where I’m heading, the executive summary of my thinking is:

Killing tinderbox is a webdev problem for the most part, with some chunks in IT and build/releng. The latter two should be fine to mostly do what webdev needs to get the job done.

I won’t give my complete rationale for killing tinderbox, for the most part because I’ve been thinking about that too long and have come up with too many reasons to write them down. But the most important fragments would be in …

The Rationale:

Tinderbox knows relatively little about our builds, and displays even less. The front end is hard to hack, and the back end is tied to a build model that doesn’t match that of buildbot. In particular in our move to hg, things have changed considerably in the back.

Listening in to previous discussions, there seems to be a gap between how people talk about our builds, and what our builds really are. Thus I’ll bore the few people that actually hack on our build automation for a few, and dive into …

What buildbot knows:

Buildbot, aka the software we’re using to control and run our builds, knows plenty about our builds. I’ll give a list of things that come up to my mind, with a focus on things that tinderbox wouldn’t.

• Why a build is running.
  • Which changes went into this build, or if it was a periodic or forced build
• The steps of each build, with separate description, results, logs
  • Logs have separate stdout and stderr chunks in order
• A set of build properties, holding slave name, build number, revision
  • The set of build properties can be amended, to hold more data. The data can be basically anything that can be pickled in python, and could be constrained to json values, or just natives thereof.
• Start and end times for both the individual steps and the complete build

There are some shortages in particular when it comes down to our build setup, mostly …

What even buildbot doesn’t know:

• Dependencies between builds. Buildbot has two builtin methods to run builds that depend on prior builds, but it doesn’t keep track of that relationship.

For those into schema, one possible version of that is depicted in this graph.

Then, there are things we keep …

On tinderbox, but not on buildbot:

• Tree rules
  • open/closed (used to be on despot for cvs/bonsai)
  • sheriff
• Build comments
• log parsers

So much for the read-only side of life. On top of this, there are a few important things that buildbot enables us to do, which we don’t empower our community to use (at least not without a releng-sheriff around).

Buildbot can:

• Trigger builds on arbitrary builders, possibly with particular properties set (the latter requires hackery).
• Stop most builds while running.

Exposing these should provide a powerful tool to investigate and clear bustages.

You can get a slightly better idea of how things are looking on buildbot itself if you browse around on Chromium’s waterfall. IMHO, they share the problem of not being able to present the data they have, even though they have less platforms and trees to handle than we do. You can also see the problem of dependent test builds hanging somewhere in the air. You can also nicely see the output per step with the details they have, unconditionally though. Most of the time, you likely don’t care.

Going forward, I’ll try to wrap my head around which problems our web frontend to our builds actually needs to solve, and which routes I see to getting there.

See your django models in SVG

Last week I found django-command-extensions, which together with graphviz allow you to create pretty displays of your django models. Sadly not the other way around, but still better than nothing.

The awkward piece is when you like to get SVG output, which I do. Renders fine on my linux VM, but my gecko on OSX is worried (rightfully, I guess). Which can be fixed with sed, at least in those graphs that I have:

sed -e's/8.00/8px/g;s/ Bold;/ ; font-weight: bold;/g'

Basically, you have to fix the font size and the font weight.

I’ll see if I blog about the actual models pre or post April 1st. I don’t find them funny, so they’re not gonna go live tomorrow ;-).

New hook for l10n feedback

We have a new hook for folks to get feedback on how to expose patches to localizers. We created a bugzilla account community@localization.bugs, which interested localizers are going to follow. I for one do at least.

So when you have a bug and need input on how to localize it, you can now CC :l10n (part of that accounts real name) instead of just me, and get feedback from a variety of different localizers.

If you’re a localizer that’s willing to help out our developers on creating good patches for l10n, please start watching community@localization.bugs by editing your mail preferences on bugzilla.

Fennec added to l10n dashboard

I just added Fennec localization stati to the l10n dashboard. It’s only pretending to do arm linux tgz’s right now, and it’s not uploading them anywhere. I expect that will happen soon on the main build machines.

There were a few funky edges to make fennec build similar to what we have in Firefox now. Most notably, Fennec uses a completely different directory structure, which looks really familiar once on disk. The main difference is that when Fennec developers check in to a file in mobile/locales/en-US/chrome, that file is actually in locales/en-US/chrome in the repository. When a localizer catches up with that change, they’re really landing to ab-CD/mobile/chrome. That requires some special hooks in the code that watches for l10n changes on the build master. The biggest hurdle was surprisingly the all-locales file in the end. That just blew the logic I had. I’m now loading the all-locales file by assuming that it’s right next to l10n.ini, while I’m still using the path to all-locales in l10n.ini to control what the AllLocalesWatcher pays attention to, to update the list of all-locales in the master when that’s changing in the repo.

What I really like is that we’re adding this product to our localization story in good shape. In my little universe, the big winner is that we’re starting with l10n-merge working. Thanks to mfinkle’s timely reviews, I added logic to the build to export both the gecko and the fennec revisions of the en-US build that’s repackaged, so one can run compare-locales (with or without l10n-merge) against the sources that actually went into the build. This is really important to get the missing strings right, and to ensure that the build is working as good as it can.

So, localizers, now is the time to land. Please make sure to touch base with Stas on any questions regarding web services. Using what we have for Firefox is a good base, though. Gandalf can help with general technical issues, as can I. If you’re not part of the fun, and want to join, please file a bug.

We’re well aware that you’ll need a descent infrastructure to test with. What is most important to you guys? Leave a comment on this post?

Triggering l10n builds

Localized builds are easy, you take away en-US, throw in the localized files, and you’re done. Piece of cake.

Throw in 3 platforms, four different builds each, 70 locales, and you have a lot of cake.

Yesterday I wrapped up a little experiment that allows one to set up a testing environment for l10n builds, and I figured I’d share some of the outcome. The trick here is to take away the complexity of the actual build and abstract away things like getting the en-US binary, the right source repos etc, and to just focus on when l10n builds should do what. There are more details on what the actual l10n build is supposed to do on the l10n deliverables page. The files for this experiment are in my temporary fork of buildbotcustom/bin. The setup includes l10n builds, en-US regular builds, and en-US nightlies. As far as the interaction between en-US and l10n builds go, we don’t need unittest and talos boxens here. All builds on this setup are fakes, that is, they take a few seconds (mac being quick, linux a tad slower, windows being slow), en-US builds taking longer than a l10n repack. The individual builds are represented by a yellow “Build 3″ box, followed by a green box for the only step of that build. It displays some relevant information, in particular, which slave it’s run on (there were two per platform, in one pool), which locale it’s building, and what kind of build it is. Nightlies and regular en-US builds are easy, just the regular column names you’re used to, but the l10n builds can use a little legend:

A build as triggered by a localizer, building Firefox for ja-JP-mac, on slave sw1 would look like this. That would be on Slave Windows 1 (starting with 0, this is the second slave).
A build as triggered by a check-in for en-US, building Firefox for ja-JP-mac, on slave sw1. The difference here is that it’s showing compareOnly, we’ll see in the flow below why that is.
A nightly l10n build would again show the locale and the slave, in addition to the fact that it’s a nightly.

All of these builds share the same builder, for all locales. Just the platforms require different builders, as they’re delegating work to different groups of slaves. That’s because in the end, it doesn’t matter who or what caused that build to be made, the interesting point is whether the build is good or not. We’re not using different builders per locale, just because that doesn’t scale. In this test setup, there are 4 locales, but in real life, we have 70-ish. There’s just not enough space in my flat for monitors to display that.

Let’s move on to see some actual builds in action. Easiest first, as that looks most familiar to what we do with our regular builds. A localizer checks in, and in return, we do a build on all three platforms.

You see the three Firefox test builds on each platform, which are the l10n builders. There is nothing on the mozilla build and mozilla nightly columns, which are the en-US builds and nightlies, resp. There’s only one locale built, in this case, ja-JP-mac, which is the locale for which I had pushed content here. Note to the Japanese localizers, this is on my test setup, so the hg repos I pushed to were local, and in fact, didn’t resemble anything close to your repos.

On to the next scenario, slightly more tricky. Nightly builds. Nightly builds for en-US are triggered by a scheduler that runs each day at a particular time. For l10n, we want repacks of those builds. Which implies that those builds are done, so we run them later.

You can see a nightly build for each platform in en-US, linux being quick, mac slightly slower, windows being a bore. After a while, there is a l10n build for each locale (this setup has 4, ab, de, ja-JP-mac, x-testing) for each platform, being spread over the various slaves of the pool. Remember, we had two slaves per platform here. In real life, there are many more.

Now we start to see some pitfalls. The scheduler that triggers the nightly l10n builds needs to wait for the en-US builds, so despite the fact that the linux build is done, it’s not scheduled before we expect the windows build to be done. Right, expect. There’s actually no guarantee that the windows build, or any other platform for that matter, are actually done. We just delay the l10n builds to an extent that this isn’t a problem in practice. Another pitfall are respins. If the nightly build is busted, we go in and trigger new nightly builds. The l10n scheduler doesn’t know about those, so l10n builds are not respun. That’s covered by bug 469364.

Now on to the really fun pieces. Let’s look at what happens if a core developer checks in a patch that changes strings. Of course, that triggers the regular en-US builds. But what do we want to do for l10n? We should use this occasion to notify our l10n community that there is outstanding work, so there’s good reason to actually start something right away. Let’s look at what the builds do.

You can see that the en-US builds kick off at the same time as the l10n builds. Now, what should the l10n builds do? Firstly, notify the localizer about the new strings. Good, let’s do that. But what more? Really, nothing, the next good thing would be to show off the new strings in context, but as you can see from the waterfall (there is some good in waterfalls, lastly), the en-US builds are far from done so far. That’s why the l10n builds have compareOnly set. These builds should merely run compare-locales and report the current string work on the dashboard, and be done with it.

Three kinds of builds for l10n. Why did I mention four initially? I left out the release builds. Those come with different rules and requirements. More details on aforementioned l10n deliverables page.

For the curious ones thinking about how we can use our computing resources, and why sometimes builds take a while without something happening, you can see a nice battle between different builders over the same slave pools, in particular, the slightly cut off l10n builds were waiting on the en-US builds to free their resources.

I hope that this post gives you a more visual idea of what we need to do in l10n builds land, and why.

Not a Bonsai

News from the tooling sink.

I just brought up what I needed to replace bonsai for hg on stage/pushes. My request list was as follows

• Query multiple repositories at once
• Query for landings to particular directories and files

I would love to get graphs and blame and mark, but I’m still hoping on hgweb on that.

The navigation is a bit rough still, it doesn’t have pages, yet, for example. I’m rather happy with the search, though. As it’s not discoverable, I’ll guide you through a bit. Here’s what a sample search would look like:

You get to this dialog by hitting the search button on the top left. First, you see a limit, this is the maximum number of pushes shown. The button switches that on or off in the query. The next two options are a tad more interesting, they allow you to search in a period of time. I use a date-time-picker for both from and until. Again, the buttons switch those on and off. The choice button lets you add more queries for paths and repository names, which are the piece after hg.mozilla.org in this setup. But on to the date-time-pickers…

I actually think those are fancy, and yes, I wrote those from scratch. I shamelessly stole the navigation pattern from Simile’s Timeline, i.e., I have bands with different resolution in time, which you can scroll. As I don’t display any events inside the bands, they’re actually empty, but the topmost band is months, then you have a band for days and one for hours. That lets you select a date and a time in a rather untuitive fashion, once you get used to the fact that sliding to the right is decreasing. Which is right for the UI pattern, it just feels awkward.

I find that UI pretty good, in fact, much better than other datetime pickers I found on the net, or the one that Philipp presented at FOSDEM.

The biggest challenge I have is making it accessible, though. Suggestions welcome. I think that a11y on pushes sucks in general, patches accepted :-).

A note of warning to end this post. We’re adding services at a faster pace than CPUs on the l10n server. Please pound this one with care, i.e., don’t pound it. It’s gonna enforce limits on searches right now to some extent, be nice. Thanks.

The daily morning traffic lights

The tree is OPEN. However, 2 boxes are orange and 2 are burning.

Man, I hate having patches ready to land.

</rant>

7 Things You May (Or May Not) Know About Me

Got tagged by Pascal Chevrel.

The rules:

1. Link to your original tagger(s) and list these rules in your post.
2. Share seven facts about yourself in the post.
3. Tag seven people at the end of your post by leaving their names and the links to their blogs.
4. Let them know they’ve been tagged.

Seven things:

1. The title of my blog “Maggot Brain” is coming from 1971 album by Funkadelic. I picked that up by cover design, date, and length of songs during my days in Stuttgart, early 90s. Yep, I’m that old. The covers on the net neither show the maggot on the inside nor the skull on the backside of the cover.
2. I own and listen to a record player. Vinyl.
3. I’d prefer to be either a good deal more clever or a good deal dumber.
4. I’ve struck my mother with an airplane. It was a a RC model sailing airplane “Finikofi” by Robbe, 1.60m wingspan. My mother is fine, thanks. Lesson learned: Don’t do bungee launches facing the airplane, left ain’t left. I don’t have that plane anymore, but I still have a bunch of other RC and control-line gear, though I haven’t put it to use in decades.
5. I’m rather addicted to TV series. I miss the times in Kiel when I could watch them un-dubbed on Danish TV. Pokkers.
6. I’m randomly allergic to wheat. When I get hit, I get hit hard for about two hours.
7. Dogs yes, cats no.

And now to the hard part, finding 7 folks that haven’t yet been tagged.

1. Runa Bhattacharjee, which reminds me that I still owe a T-Shirt to
2. Mary Colvig, oops. Sorry.
3. Darius Zarrabi, in the event that his blog just accidently 404s. Otherwise I’d know exactly 0 non-FOSS people with a blog.
4. Pascal Bleser, who’s our mate for the FOSDEM developer room for years now, it’s gonna be good to see him again at FOSDEM 2009.
5. Mark Surman, because he’s going to make that a beautiful post.
6. Chris Hofmann, who’s blogging less than I do :-).
7. Gijs Kruitbosch, taking him over that bridge.