whereas the original (triple-licensed) translation suggestion is on l10n.mozilla.org/narro.

Another interesting aspect of crowd sourcing, box-of-chocolates style. You never know what you get.

l10n-merge is originally an idea by our Japanese localizer dynamis. The current implementation used in the builds is by me, integrated as an option to compare-locales. There are spin-offs of that algorithm in the silme library, too.

l10n-merge attempts to solve one reason for “yellow screens of death”, i.e., XML parsing errors triggered by incomplete localizations. This is really crucial as localizations don’t just pop up by swinging magic wands, they’re incremental work, and a huge chunk of that. So in order to test your work, you need to see the strings you have in, say, Firefox, without having the other 4000 strings done yet. Other l10n-infrastructures handle this by falling back to the original language at runtime (gettext), but doing that at runtime of course has perf impact, and size. l10n-merge does the same thing at compile (repackaging) time.

Design goals for l10n-merge were:

• not mess with any source repositories
• not do any file-io that’s not really needed

Thus, in order to not mess with the source repos, l10n-merge doesn’t modify the sources inline, but creates copies of the files it touches in a separate dir. Commonly, we’re using ‘merged‘ in the build dir. Now, creating a full copy of everything would be tons of file io, so l10n-merge only creates copies for those files which actually need to get entities added to existing localized content. This plays together with code in JarMaker.py which is able to pick up locale chrome content from several source dirs.

A Firefox localization contains some 450 files, and say for the current 9 B1-to-B2 missing strings in two files, it would copy over those two files from l10n, and add the missing entities to the end. Then JarMaker is called with the right options, and for those two files, will pick them up from merged, the rest of the localization is gotten from l10n. For missing files, it actually looks into the en-US sources, too, so we don’t have to do anything for those. To give an example, for chrome/browser/foo in the browser ‘module’, it searches:

1. .../merged/browser/chrome/foo
2. l10n/ab-CD/browser/chrome/foo
3. mozilla/browser/locales/en-US/chrome/foo

Now it’s time to list some pitfalls that come with l10n-merge:

• If you’re passing the wrong dir for mergedir, nothing breaks. All build logic breakage would come from missing files, and due to the fallback to en-US, there are no missing files.
• l10n-merge, as compare-locales, doesn’t cover XML parsing errors inside entity values yet. Bug 504339 is filed, there are some tricky questions on reporting, as well as having to write an XML parser from scratch.
• l10n-merge only appends entities, but that’s fine 95% of the time. Only counter-examples are DTDs including other DTDs.
• People using l10n-merge need to manually maintain the merge dir. Pruning it via compare-locales is risky business if you specify the wrong path by accident, so I consider this a feature. But if you’re seeing Spanish in a French build, clobber the mergedir and build again :-)

Bad for planet mozilla, I did take a picture and here I’m blogging about it.

Yes, I love the web so much, you can hardly see me. But you do see real full screen on a mac, German namoroka nightly. And you the same one web on my N810 (not so current 1.9.2 nightly). And the infamous “Don’t hurt the web” shirt with that, of course.

• String freezes are one only, and done with. No “I’ll get that in the next milestone”. First string freeze is the last.
• String freeze for mobile-browser and toolkit was last Friday. For mobile-browser seriously this Friday, Sept 11th. (*).
• String freeze for Firefox 3.6 is Sept 14th.

It already supports marking up errors, to a similar extent that they’re found in compare-locales these days. Better parsing for the actual values is a bigger task, in particular one that should improve Bespin’s XML mode at the same time. ‘Cause that didn’t impress me yet.

Code will come in bug 510579.

• Builds in progress are associated with the check-in that triggered the build.
• Builds in progress show individual build steps.
• Finished builds are displayed in compact form. In this case, all builds end with warnings, and thus come back in a shade of orange.
• Builds not yet started that are already requested are displayed on top.

I didn’t go into the detail of mentioning which builders are having pending builds. This is mostly me waiting for django 1.1 and aggregation support, but in the end it’s as simple as a GROUP BY. Nor did I try to make that display visually pretty, hell no.

In the context of our regular builds, it’s worthwhile mentioning what unit test and talos runs would look like, i.e., “builds” that are scheduled after the binary builds are done. Those wouldn’t show up until they’re actually scheduled, which is fair enough, as that is what’s actually happening. If a windows build fails, there won’t be unit tests nor talos builds run. You wouldn’t end up in a situation where you think you’re done and you aren’t, though. (Talos not working this way aside, that needs thought due to different masters etc.) The actual builders don’t finish until they triggered the spin-off runs, so you either see the binary build as still running, or you see the triggered runs as pending. Here, as soon as you don’t have anything running or pending and your boxens are green, you’re off the hook.

I also added microsummaries and RSS feeds for this view, so you can use the web to learn about your the fire you lit up.

As RC 1 was based on new strings, we required each localization to sign-off on the status of their localization to be ready for release. We’re still doing this by opening what we call a “opt-in thread”, a message sent to .l10n after the last l10n-impact landing to which localizers reply with a link to the revision of their localization that is good to go. Part of that communication is the message when code-freeze is planned to be, and the message that plans don’t always work out. So we’re keeping the the opt-in thread open actually up to the point where we really kick off the builds.

The output of that process are two files which control our release automation process, shipped-locales and l10n-changesets. For the curious, we’re tracking which locales ship on which platforms in the first, and it’s part of the code repo, and which locales ship which hg revision in the second, which is in the buildbot-configs repo.

The whole process lead to 150 different opt-in sourcestamps which came in by either public replies in the newsgroup, or as private mails in my inbox (or both). I pick those up, and click on some buttons on a version of the l10n dashboard running on my home machine, review the changes to previous sign-offs (yes, I do have a web interface that does comparison between two revisions in hg), and accept or reject the sign-off. If I reject, I follow up in .l10n with why I did that. That adds up to 159 posts in that thread, by 72 authors. Dependent on how imminent the release is, or seems to be, I “back up” my local data by attaching files to the tracking bug. This led to one version of shipped-locales, and a whopping 16 versions of l10n-changesets. Or, in short …

What’s really cool here is that we’re actually at a point where we pick up improvements to our localization up to the last minute, with tools that make us feel comfortable about that, and with a release environment that is able to digest all that noise and produce builds for 73 localizations in a matter of a few hours.

I’m picking you up at the point where

twistd get-pushes -n -t 1 -s l10n_site.settings

is running stable. You probably want to tail -f twistd.log to follow what it’s doing. This piece is going to watch the hg repositories in stage/repos and feed all the pushes to those into the db. Next is to make sure that you actually get builds.

The first thing you need to do is to configure the l10n-master to access the same database as the django-site. Just make sure that DATABASE_* settings in l10n-master/settings.py and django-site/l10n_site/settings.py match. The other setting to sync is REPOSITORY_BASE, which needs to match in both settings.pys. I suggest setting that to an empy dir next to the l10n-master. I wouldn’t use the stage/repos dir, mostly because I didn’t test that. Now you set up the master for buildbot, by running

buildbot create-master .

inside the l10n-master clone. Next thing is to create a slave dir, which is well put next to the l10n-master. Despite what buildbot usually likes, this slave should be on the same machine that the rest is running on.

mkdir slave
buildbot create-slave slave localhost:9021 cs pwd

So much for the general setup. With the twistd daemon running to get the pushes, you can now switch on the master and the slave, by doing a buildbot start . in both dirs. I suggest that you keep a tail -f twistd.log running on the master. If you decide to set things up to track the upstream repositories, I start the master, then the slave, and if both are up fine, I start the twistd daemon for get-pushes.

Now let’s trigger some builds:

Open stage/workdir/mozilla/browser/locales/en-US/file.properties in your favorite editor, and do a modification. I suggest to just do a whitespace edit, or to change the value of the existing entity, as that is going to keep your localizations green. Check the change in to the working clone, and then push. The get-pushes output should show that it got a new push, and then on the next cycle, feed the push into the database. You’ll notice by the output of a hg pull showing up in the log. On the next poll on the l10n-master, builds for all 4 locales should trigger. You should see an update of four builds on the waterfall, and 4 locales on the test tree on the local dashboard.

We’ve had l10n builds as long as I’m working on l10n, actually, I got involved around the time when we started to do them upstream. They always were considerably better than each localizer doing their build at home on whatever (virus-infected) hardware they found, with help from other community members for the platforms they didn’t have. But at the light of day, it was more

That? Yeah, I know. That’s crap.

And I know you can hear my voice in your head right now :-).

Those days are gone. We’re running Firefox and Fennec builds on the releng infrastructure now for a few days that are actually sound builds made to service our l10n community. Some highlights:

• Builds are finished some 10 minutes after a localizer landing, on all three platforms.
• There’s no deadlock between different locales, thanks to all l10n builds running on a pool of slaves.
• Builds are “l10n-merged”, against the actual build that’s repackaged. Independent of missing strings or files, you have a build that can be tested.
• No more race conditions between nightly and trunk source status.

The impact of this shouldn’t be under-estimated. We are, for the first time in years, producing builds that allow a localizer to actually immediately test. Localizers can work incrementally, translate one feature, check-in, test. No worries if something landed in en-US in the meantime, or whatnot. With the new builds, I have seen various localizers coming from hundreds of missing strings to a tested build on two or three platforms in a matter of a few hours. Back in the days, that was the waiting time for the first build. The new locales all pull all-nighters to get their final bits in. They want to, and now they actually can.

I want to thank coop and armenzg for their great help in making this happen, aki for porting it over to fennec. Of course thanks go to joduinn and sethb, too, for bearing with the ongoing meetings we have, trying to battle the crap down. To dynamis for the initial work on l10n-merge. Also thanks to bsmedberg and Chase for the initial works on both automation and build process, and ted for the various reviews on making our build system catch up.

Finally, we’re not going to stop here. Armen is working on creating the necessary files to get l10n builds on a nightly update channel. Yep, you heard right, that’s where we are right now. I know that KaiRo is working on getting the goodness over to the comm-central apps. And yours truly is hacking on the dashboard together with gandalf, more on that in a different post.

Things we might want to search for:

• Localized strings
  • in a particular locale
  • in all locales
  • going into a particular app
• entity names
  • in all of the above

As with the rest of the dashboard, I’d favour a pythonic solution. I’ve run across Whoosh, which seems to offer me what I’d need. In particular I can mark up searches in just keys or just values of our localized strings with the Schemas it offers.

All sounds pretty neat and contained, I’m just wondering if there’s something cool and shiny elsewhere that I’m missing, or if someone came back with “ugh, sucks” from trying Whoosh.

Ad-hoc design for the curious:

For each changeset, we’d parse the old and the new version of the file, getting a list of keys and values, and I’d create two searchable TEXT entries for all changed keys, and added entries. We’d tag that “document” with path, locale, apps, revision, branch. That way, you could search even for strings that aren’t currently in the tip, and get a versioned link to where it showed up first, and last, possibly. Given that we have a lot of data and history, I wouldn’t be surprised if that corpus would get large pretty quickly. I’d expect to not only index l10n but en-US, too. Thoughts?