-
Help me test two Kiswahili versions of Firefox
Surely, you saw me fire off a response two weeks ago about playing politics with our Kiswahili localization communities. Let’s move on from that flame war by summarizing our situation and presenting a path to a solution.
Presently, we have two communities, the tzLUG and the Kilinux teams, who have translated the Firefox application into Kiswahili (sw-TZ). Unfortunately, we have had tough luck in getting an unbiased, thorough evaluation of each body of work to help us decide which one to use. As it turned out, it was hard to find a number of individuals familiar enough with technical writing and Kiswahili who had time on their hands to volunteer for Mozilla. Furthermore, we didn’t have an easy package to evaluate, except for the “diff” of the code differences between the two. Yeah, that sounds ugly and it was. Still is.
To solve what has become a long-standing debate, we asked each team leader to create a Mozilla language pack of their work as an add-on that we would then host on and promote though our addons.mozilla.org website. Both teams agreed and uploaded their versions. Since then, I created two separate “collections” that bundle each language pack with Ben Smedberg’s Locale Switcher addon. Our hope is that end-users ready to test will install both versions and use the addons.mozilla.org site to provide feedback to each developer team.
If you are interested in testing each version, please install the following two collections:
- Kilinux: https://addons.mozilla.org/en-US/firefox/collection/kiswahili.kilinux
- tzLUG: https://addons.mozilla.org/en-US/firefox/collection/kiswahili.tzlug
Once you have installed these, you can switch between the two versions and your English interface by going to the menu item Tools –> Languages…
Now for testing…
Requirements: You must be able to read Swahili and English fluently and you must use Firefox.
If you choose to test these localization language packs, you’ll need to follow something similar to the “Firefox 3.5 Localizer Test Run” that has been created in Litmus, Mozilla’s testing application. If you use Litmus, please follow the steps I have posted in the first comment on this blog post.
You can also just use each language pack and keep notes of errors you spot. Whether you choose to use Litmus or not, please record any translation errors that you find in the user interface of each version. Please be very descriptive and thorough with any notes you keep, and write the notes in English. Take a look at the word choices, terminology, spelling, grammar, etc. and keep a record of errors you see. When you are finished, you can submit your evaluation to me. Just ping me on this blog.
As always, please ask some questions if you have them. Nothing is off limits.
-
What happened to two localizations on the day of Firefox 3.5’s release
Not everything is picture perfect in the world of Mozilla localization. Though it pains me to say that, we hit two snafus at the release of Firefox 3.5. Here’s what happened with our Macedonian (mk) and Serbian (sr) localizations, complete with a mea culpa and a plan on how to fix things going forward.
On the day we released, our community found two very similar errors in our mk and sr builds. In both cases, a misspelling of &brandShortName; inside an <!ENTITY> triggered the “the yellow screen of death” when users selected the Help -> Check for Updates… option to get the new version.
A thousand apologies to our localizers and mk and sr users for not catching these errors pre-release.
With damage control in full swing, we removed the two localizations from the Firefox 3.5 release channels so that users would not receive a broken version of Firefox. The two localizers and the l10n-drivers then worked through our options. We could either release a special post-Firefox 3.5 Macedonian and Serbian version, or wait until the release of Firefox 3.5.1.
The unexpected timing of the Firefox 3.5.1 release helped us with the above decision. Although the circumstances of the security update were not ideal, it did allow us to release mk and sr earlier than expected, getting users of those localizations back on the release track. Furthermore, any users who might have gotten the broken mk or sr version of Firefox 3.5 on release day will be updated behind-the-scenes without having to check for updates. [1]
What happened on the 3.5 release day underscored a few errors in our system that need to be fixed. Here are the proposed and soon-to-be or already implemented measures we are taking to reduce this margin of error:
- Localization sign-off via the Web: Rather than opting-in with change sets, localizers will soon select the Mercurial change set they want to use for a release from a list of IDs pulled from their locale’s repository. How does this help? In Macedonian’s case, a localizer *had* submitted to Axel a change set that corrected the error prior to RC3. However, Axel was unreachable at a conference and couldn’t relay that update to me. Sadly, I submitted the incorrect change set. Mea culpa. This application allows localizers, l10n-drivers, and Firefox project managers to view the choices that have been made, and to the extent possible, test to make sure that the version the localizer wants is good for release.
- Test automation: We are working on creating a script that can be run by our QA team before each milestone that will scan for misspellings in things like &brandShortName;. This bug is tracking that progress. Our QA team also is increasing the number of automated tests that will be run on each locale before each milestone release.
- L10n-testable builds: Presently, we are producing testable nightly builds with Axel’s l10n-merge code that will create localizations with en-US strings replacing any untranslated ones. Now, localization teams can give their testing communities something to test from the beginning of the release process. In the past, localizers had to wait until they had 100% translation before we provided them a nightly build.
- Localized builds with nightly updates: Added to the testable builds above, we’ll soon be able to offer nightly updates to localizers and their testing communities. Right now, only en-US testers of Mozilla’s pre-release versions get nightly updates pushed to them. Soon, *all* localizations will get these nightly updates pushed their way so our global testing community can see the most recent additions made by Mozilla’s developers.
These tools empower the localizers and the testing community, and we believe will help narrow our margin of error so that we don’t repeat what happened to our mk and sr builds.
Many thanks to our Macedonian and Serbian localization teams for their understanding and patience and sorry for the errors discovered at the time of the Firefox 3.5 release.
[1] Mozilla developer rstrong and his team fixed this bug and cleaned up a lot of code for the Firefox 3.5 release so that users of Firefox get updated behind-the-scenes without having to check for updates or get prompted unnecessarily if they want/need an update.
-
Responding to Walt’s rhetorical criticism
If you advance to the two minute thirty-five second point of this interview of Mitchell Baker and John Lilly, you’ll hear Walt Mossberg remark about the quality of Mozilla’s localizations by saying,
“I have a deep distrust of somebody who I don’t know to be actually responsible for the quality of the end product.”
We’ve heard that before, haven’t we? To entertain the point, I’ll answer a question, “Just how do we know that our translated product is high quality?” by linking to several posts (with very brief summaries) as a response to Mossberg’s rhetorical criticism.
- Testing the latest localized version of Firefox 3.5 — In this post, I ask our localizers to test a release, with specific steps that each locale can follow.
- Moving a locale out of beta — This is a basic software release principle. No, our localizers don’t get a free pass into “official status”. We give each locale a proper amount of time to bake so the beta users can provide feedback to our localizers. After feedback is “triaged”, bugs are fixed, and signs of user adoption become obvious, we move a locale out of beta.
- Localization-QA survey results — At the end of 2008, we conducted a survey to gauge our teams’ testing efforts. The posted results point us to where localizers might need our assistance. From this, we have begun an experiment to provide a third-party QA service to help test a sample of our localized versions.
- Adding contextual information to a localized build – Some of our localizers even create, share, and use customized tools to help perfect translations
Perhaps it’s hard to express without sounding naive or idealistic, but maybe there is an important theme that didn’t make Mossberg’s conversation that should be articulated:
We take localization very seriously. This is not just a hobby for our community, and many have the battle scars to prove it. Just ask someone who has stayed up all night to perfect a translation before a code freeze and you’ll understand what I am getting at. Each of our localizers is keenly aware that greater than fifty percent of our end-users are NOT using an en-US version. When a localizer is responsible for a translation, the quality of their work impacts a massive amount of end users. We could ask our German localizer Kadir, whose localized version of Firefox is being used by an estimated twenty-five million people. Or, Romi, our Indonesian localizer, who’s translated version has climbed to sixty-three percent market share. That level of impact keeps our localizers sharp and tremendously dedicated.
Other highlights from the transcript:
“Walt: 71 of the foreign-language versions of Firefox are written by volunteers. Why should I use a product like that? Lilly says Mozilla has a system for verifying the quality of these other versions and vets them prior to release. Beyond that, users will alert the company to any problems.”
“Walt: Why wouldn’t it just be better for the consumer to go with the company that’s hired experts to do its translations? Baker: How much software do you really think is great? Walt: Not very much. Lilly: But it’s all written by experts. Walt nods, point taken.”
NB: John Lilly mentions that we’ll have seventy-one localizations for Firefox 3.5. We’re growing everyday! We are actually going to ship seventy-three localizations for Firefox 3.5’s release candidate, with an outside chance of seventy-five for final release.
-
Testing the latest localized version for Firefox 3.5
I posted this to the Mozilla L10n newsgroup, but for maximum coverage, I’ve reposted it on my blog. Special thanks to Marcia, Axel, and Chofmann the various resources I reference below.
——————————————–
To all of our great community localizers and testers…
Over the past few weeks, many of our Mozilla community members have done testing and landed fixes for Firefox 3.5 as we close in on our release. We are now in the last hours before we ship our release candidate that we can comfortably call Firefox 3.5. If you have time this weekend, it is a great opportunity to do some last minute testing for your localization.
Where to download the latest localized nightly version of the Firefox 3.5
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-1.9.1-l10n/
Please hammer on these builds mercilessly to make sure that things work well. If you notice things that worked in Firefox 3.5 beta 4, but do not work in this release, we would like to know about it right away.
What to Test
You can run a set of localization test cases by going to Litmus, Mozilla’s testing suite. This URL will take you to the “l10n run”.
https://litmus.mozilla.org/run_tests.cgi?test_run_id=36
If you don’t have a Litmus account, you should be able to create one quickly. Please email us if you need any help.
How to report feedback
Please try filing a bug for your locale with Bugzilla. The basic set of instructions are below. If you are not comfortable filing a bug, you can report it to your locale leader who should be listed in the specific locale on this main Teams page:
https://wiki.mozilla.org/L10n:Teams
Things to remember when filing a bug in https://bugzilla.mozilla.org/:
- Always include the Build ID that you tested on. If you type about: in the URL bar, this will give you the Build ID.
- Always include clear “Steps to Reproduce” the bug.
- Always check to see if your bug has already been filed. This link will help: http://tinyurl.com/2465be
- Use the regression keyword if it indeed a regression from a previous release. And, please tell us in the main comment of the bug if it is a regression from a previous release.
- If you happen to crash, please include the Breakpad ID in the bug. You can get this by typing about:crashes in the URL bar.
If you don’t wish to file a bug, report issues through http://feedback.mozilla.org or through the mozilla.feedback.firefox.prerelease newsgroup (I just linked to the Google Group). However, we prefer bugs as feedback since those are easier to track.
Finally, keep in mind that no comments or questions are off limits. Please send along any remarks or questions that you think are appropriate at you test. It’s all appreciated.
Thanks to all of you for helping test Firefox and making it the browser of choice for millions and millions of people all over the world!
-
Localization-QA survey results
At the end of last quarter, we launched and analyzed a survey about l10n testing needs. The hope was to find areas where we could provide help to our localization community around how to test their builds. The analysis was done by Mozilla contributor, Murali Nandigama, and he posted his slides here. The next step is to implement something from those findings for the community. Please take a look at the analysis and make help me with some conclusions.
Here are some takeaways our team noticed when we looked at the slides.
- Localizers are enthusiastic to see more automated tests and automation frameworks for testing compared to adding more Litmus manual tests.
- Survey participants might be assuming that more automated tests would provide the same functionality as Litmus manual tests.
- However, especially in the L10N context, font-rendering, text clipping, dialogue/character spacing etc., can be eye-only tests and might not be substituted by automated tests without a lot of non-trivial automation effort.
Conclusion: Litmus manual tests are important and hard to automate to replace visual verification. Has everyone played with Litmus?
- 95% of survey participants are using non-Bugzilla channels to report issues, problems, and bugs. (Or, only 5% use Bugzilla to report issues.)
- Because of this, we cannot quantify and archive the effort of l10n volunteers in finding bugs.
- How are people logging issues with local problems?
Conclusion: We might need to blog to demystify the bug filing process for l10n volunteers. Or, provide easy tools like bug-by-email templates.
- In the survey, we specifically asked if l10n communities are testing for a things like non-US keyboard input, clipping of text in boxes, RTL etc.
- A majority of communities are either not aware of these testing steps have no need to test these issues (like RTL).
Conclusion: We might have to provide a generic blue-print for effective test planning for any locale, indicating what are the key l10n steps in test planning.
- In the open comment section, we saw responses like this:
- Survey takers indicated that teams larger than 10 are using the l10n builds.
- Teams indicated that only one or two members are actually reporting problems and filing bugs.
- Some indicated that they do not have a formal test plan in place, no institutionalized knowledge-base on testing locales, and lack a coordinator/leader for their l10n testing efforts.
Conclusion: We might need a third-party QA service to help lead volunteer L10N test activities.
I’m still sifting through the data to find who might benefit from some of these recommendations and findings. If you’d like to make my job easier, please comment on what you think your locale needs. Questions are welcome too, please.
Many thanks to Murali who did the analysis posted above and helped me write this post.
community website
