At the end of last week, I received an unexpected email from Stefan Arentz, the new developer working with Dan and Ragavan on the project:

“I have automated the localization process with some simple scripts to grab the .po files from localize.mozilla.org and run the conversion tools on them to make them iOS compatible.

“I have also integrated this in my continuous build scripts. This means the most recent build will always contain the most recent localizations automatically. This is done ON TOP of the hg checkout. So that means that the localizations still need to be imported and committed manually when QA has approved. (with the same scripts, so it is a no-brainer).”

That’s a clever solution, isn’t it?  Thanks for taking that initiative, Stefan.

(understatement alert) It turns out that localization on the Apple iPhone platform has some pieces that are very easy to understand and manage, and other pieces that are unique, complex, and do not align well with a global open source project like Mozilla.

A Brief Technical Description on How to Localize an iPhone Application

I worked closely with the Firefox Home lead developer, Dan Walkowski, to learn how iPhone applications are localized. I will paraphrase some specific technology tidbits provided to me by Dan to give a brief description on what was necessary.

To begin, all Mac/iPhone localization projects use strings files, which are very similar to property files. After translation and testing, all of the localized files are built into a binary and loaded at run time. The twist from Apple is that there are two ways to generate strings files, and they are used differently.

Way #1: For all places where the application uses a localizable string in code (ex: displayAlert: “uh oh!”), strings are wrapped in a macro. The developer then runs a tool to generate strings files for everything. That is the easy part.

Way #2: Most of the Firefox Home UI is defined using Interface Builder, which generates serialized object trees known as xib files. These xib files are loaded at runtime, ready to go, and attached to the rest of the running code. Most of the strings a user will see are embedded in xib files. In this case, xib files require a two-step process to create. A different tool is run over each xib file to generate a strings file mentioned above. After that new file is translated, an inverse tool runs to create a *new* xib file by substituting in the new, localized strings from the strings file. The new xib file is placed in the project in the correct place and loads at runtime when that locale is active.

The disadvantage of this is that, unlike the code-based strings, the strings files from xibs are only an intermediate step, not the actual entity needed for the build. The advantage is that since every locale gets their own serialized object tree, and we can actually alter the arrangement of UI elements (resize buttons, etc.) to better suit the locale.

Here is how Dan set it up in Hg.

Project Management with the Localizers

For this project, I chose to file all tasks needing l10n as individual bugs inside Bugzilla. Under this scenario, every locale would have several bugs representing tasks that would block a locale-specific tracking bug. Then, all the locale-trackers would block a project-wide tracking bug. Here is the dependency tree from Bugzilla showing all the bugs and the structure of the tasks.

The Localization Work

To provide the easiest way to localize the application strings, we loaded them into our Verbatim web translation tool. Once translations were gathered in Verbatim, I downloaded the .po files, and with the help of the Translate Toolkit, converted the .po files to strings files. Many thanks to Dwayne Bailey who updated the po2prop script in his toolkit that I used for this project. Dwayne also blogged about his experience with this project.

For other tasks like localizing the Mozilla website, the Application Store Description, and the emails that are sent to Firefox Home users after download, we filed bugs to gather the localization work. You can see from the dependency tree above all the tasks that needed l10n.

In hindsight, I would have like to used Verbatim and the Translate Toolkit to do more with the project. After chatting with Dwayne, we could have used Verbatim to manage most of the tasks that needed l10n. Furthermore, I intend to follow up with our French localizer’s recommendation to use SVN to store all of our text based translations, rather than text files in Bugzilla. Luckily, I learned from Alex Buchanan, who is the Mozilla webdev on this project, that he has placed all the translated emails into Github. Because of that, we should be able to extract them as .po files from Github and put them into Verbatim.

Testing

Because many of our localizers do not have iPhones, and because Apple gives us a limited pool of beta testing slots, we decided to place an emulated iPhone with the multi-locale build on a remote machine for our localizers to test. I wrote a test plan with Dan and we left that as an open text document on the desktop of the remote machine where the emulator was running. Localizers logged in with a VNC client and followed the test plan. Because more than one person was able to log in at the same time, everyone had to state in bold font at the top of the page that their locale was testing. When finished, the localizers had to return the OS of the emulator back to English so the next team could test. I think my tweet says a bit about how tricky this was. It may have been a bit of a rudimentary solution, but it was the best way we could think to get 15 different locales testing under the constraints placed by Apple.

Next Steps

Firefox Home’s past minor update (1.0 -> 1.0.1) took Apple seven days to approve. Unfortunately, I do not know how long this time will take because Apple tells us very little. We are presently waiting to hear from them.

Since the app is served as a multi-locale build, users will be able to shift to any of the supported 15 locales we are shipping. In some cases where Apple provides a localized App Store, special content localized by our community will help advertise our app.

Thank you to everyone who participated to make this a success.

If all the new locales close out their outstanding bugs and perform some testing in time for FF 4′s final release, they will go out as official and final. If they do not complete all the tasks required to fully localize their version, they will be released in beta and keep working to close any open issues. In the very least, they need to translate all strings by the release date.

Here is a pretty rough HTML table for you to see all the new locales. The table shows the locale name, the tracking bug that lists all tasks, and where any users can download language packs or nightly builds. Please click on the tracking bugs to see everything a new locale has to do. Or, download and test some of the builds. These builds use our “l10n-merge” technology that supplants untranslated pieces in the user interface with English strings. It’s never too early to test, so please help if you can.

Lastly, I’ll mention it quickly now and blog more later, but we also added two new locales for the release of Firefox 3.6.9. For that release, you will see Asturian and Scottish Gaelic in beta.

18:30: I arrive home after work and return online for a webinar hosted by the community folks in Manila, the capital city of The Philippines where it is presently 09:30 Saturday morning.  (I just linked you to another MCS installation, which is a tool created by Gandalf.) Since Gen’s and my visit, a burgeoning Filipino community has been promoting Mozilla with community get-togethers and other activities, including this evening’s webinar on how to localize Firefox

18:45: Audio/Video test with Regnard Raquedan, the local community leader. I share with him my Tokbox video conference URL and we have audio with intermittent video. Not a big deal, with chat room functionality on Tokbox, I can speak and pass URLs to all who attend the webinar. Those with video can see me on a Friday night in my SF apartment.

19:00: Webinar starts with 7 people initially logging in.  Introductions made.  Active Filipino community includes Regnard, Joel, Kevin, Gian, Bob, Martin, Charmaine, and other guest users joining as we go.

19:15: I pass the following URLs to demonstrate localization:

1. How to create a new localization –  The centerpiece of our documentation written by Stas (and me) that we believe is the one-stop for all localizers to begin (or reference later when a question arises).
2. Localizing with a web tool (in this case Narro) — A sub-article of the above piece.  I gave Regnard a few options to look at before this webinar and he chose Narro.
3. Narro — The webtool developed by Alexandru, our Romanian localizer, and hosted on our l10n-server.

19:30: I walk the webinar attendees through localization of the two main Mozilla l10n file types: DTD and property files.  The demo we constructed has new localizers translating two highly visible strings so they can immediately see the impact of their work.  As show in the document in point 2 above, we choose the “Manage Search Engines…” DTD file and the “Add %S” property file as examples of where to start.  These strings are located in the search box UI of Firefox.  You know where it is, check for yourself! 

19:45: The Filipino community offers translations for these two strings and decides which to use.  Regnard is the initial Narro admin, so he reviews all the suggestions from the community participating in the webinar.  After consensus, he approves the appropriate translations.

19:50: I discuss how we can create a language pack for testing through out the process so we don’t have to wait until all strings are translated to see the fruits of the labor.  Narro allows teams to easily generate .XPIs for testing straight from the UI.  Regnard can do this for the team and we decide to version our language packs (using the date as the versioning number) so people can keep the archive if they choose.  (i.e. Tagalog_langpack_marso_20_2010, Tagalog_langpack_marso_21_2010, etc.,  or something like that…)

19:55: Final Q&A.  Joel asks, “If we install a testable Tagalog .XPI, how can we switch back and forth to our original English-only UI?”  I pass along the Quick Locale Switcher add-on.  Everyone smiles.

20:00: We end the evening with pretty solid progress having been made.  I think the evening was a success.  I retire on my couch to watch Cal and Ohio State win their opening round games of the NCAA basketball tournament.  Go Bears!  Go Buckeyes!  I fall asleep before the games are finished, much to the chagrin of my brother who excitedly texts me updates.

==Signing off==

Our last trip to India was no exception.

On our first Sunday night in Mumbai (Feb 22), we co-presented with one of these organizations at a Mozilla community meetup.  The group calls themselves Pad.Ma or, in longer form, the Public Access Digital Media Archive.  The project “is an online archive of densely text-annotated video material, primarily footage and not finished films. The entire collection is searchable and viewable online, and is free to download for non-commercial use.”  And, right on their website, they state their intentions to align with web standards:

Q:  Which browsers do you support, on which platforms?
A:  We currently support Firefox and Safari, on Linux, MacOS and Windows.  We do not support Internet Explorer. However, if you wish to endeavour to make the site work on IE, please appeal to IE to support web standards in their next version.

(In fact, for some fun, fire up IE and visit their website to view a strong statement from them regarding your present use of IE.)

More on the meetup, but let’s rewind by just a few hours before we met Pad.Ma face-to-face…

After a four hour roadtrip on the Pune-Mumbai highway, we arrived at our hotel in the cool neighborhood of Mumbai called Bandra.  Freshened up in about fifteen minutes, we piled two-by-two into autoricks and motored our way on a humid evening through the snaking streets to the event location.  Arun had treasure map-like directions that led us down narrow alleyways.  “When you see a cross on the wall, proceed a few more feet and you’ll see an apartment entrance on your right…” Arun read as we navigated through the Pali Hill district’s corridors.  Up a few stories and our eyes opened to a rooftop event with a large projection screen, bean bag chairs, a minibar with soft drinks and beer, and two big vats of local food.  We made it.

Invited by a local community member named Sanjay, about 25 people came to hear us speak about the Open Web and how we were building community in India.  After our presentation, the team from Pad.ma followed by showing their amazing work to archive movies on the web.   If you’re a movie person, this site will fascinate you, so please look around it.  The Pad.ma presentation was followed by their demo of the “Firefogg” addon, which allows you to easily convert videos to .ogg theora video compression format.

It was a nice tandem.  Arun chatting about the Open Web and explaining cutting-edge demos.  And, just when we thought we might lose the audience on how the technology could be applied, Pad.Ma presented their work and the Firefogg addon.  It was a nice blend of demos and practice and I believe the group’s imaginations were sparked.  The Open Web had been delivered to a rooftop audience in the Pali Hill neighborhood of Bandra.  Sometime during the evening, fireworks started to explode.  This was not planned.  An Indian wedding was taking place in around the corner.

Our first big event was the three-day conference at gnuNify ’10 held in Pune.  On Friday, February 19, we held an entire track to chat about Mozilla.  Arun gave a talk on web standards that seemed to be a crowd favorite throughout the entire trip.  Ragavan chatted about Mozilla Labs, and, since all of our Indian localizers are there, I invited Axel to join so we could chat about the new project “l20n” which I’ve blogged about in the past.  The rooms were pretty packed, in some cases, standing room was the only option.

I was particularly pleased that about half of our localizers cut out of work early on Friday to see Axel and me.  Not only did we chat about l20n, but we reviewed a new locale (मैथिली Maithilī), worked on moving Oriya out of beta status, and caught up in person with some of our most dedicated community members.

On Sunday, we held our annual entrepreneurs breakfast with local web entrepreneurs in Pune.   Arun lead the morning with a detailed discussion about the Open Web, covering the following:

• The Open Web as a platform
• Future of video on the Web
• Device orientation events in Firefox
• Font issues on the Indian Web
• The geolocation API
• WebGL and 3-D graphics

At the end of session, we held a small workshop where we split the groups into teams of three and asked them to come up with entrepreneurial ideas that encapsulated all that we had discussed.  The groups were given 10 minutes to come up with their ideas.  Then, each team had one minute to present to the audience.  We served as a panel, questioning about the idea, potential business model, etc.  Of the ideas we saw, four stood out:

1. A “Typekit.com” for indian languages – typekit.in
2. e-learning classrooms for physically impaired – using video capabilities of HTML5
3. A video mashup app – something like online maps with text to speech audio and video
4. Using the geolocation API from Firefox – giving users local search results through a map website

Everyone’s ideas were great, but we selected these as standouts and gave the winning team some Mozilla stuff.  We chose the e-learning idea.  We offered to follow up on specific ideas and questions if anyone had them.  The team that came up with the geolocation API use-case has already started their business and intends to experiment with the technology and promote Firefox on their local map website.

In conclusion, the event in Pune was a great weekend for us to kick off our Indian adventure.  We landed into familiar arms since we presented at gnuNify ’09.  Because of that, we saw many faces we already new and were able to really push our conversations to very technical levels and Mozilla-related ideas.

Because we field many requests by volunteers interested in becoming an official localization, we have to find what is best for us and them.  That can mean promoting smaller localizations as language packs on our addons website.   Oftentimes, we find that a language pack might be the best possible solution for a new localization.

To make that determination, the l10n-drivers will work with the volunteer to assess not only how many users there might be, but also what productization elements might be included. By productization elements, I am referring to things like

1. embedded search providers such as Wikipedia, Amazon, etc.,
2. web protocol handlers like Yahoo as a  “mailto:” option, or 30 Boxes as a  “webcal:”, and
3. “live bookmarks“, which is an RSS feed for a local news provider that will be included in the bookmark toolbar.

Among many other things to do around release time, localizers are responsible for selecting each of the above for the localized version based on what they believe is best for local users.

Now, for new localizers who request that we add a version where Firefox already has prominent usage, it is important to determine if the services they are suggesting will be measurably different from what is already included in the predominant version in the region.  For instance, I recently asked a Breton localizer if his productization pieces would be dramatically different from what our French version already provides.  If the answer is no, a language pack might be the best best to serve this niche user base.  In this scenario, users download the dominant regional version, and then install the language pack to change the strings to the niche locale.  They use the productization pieces from the dominant version since those were determined to be the same for the niche user base.  (As an added bonus, with an addon like Locale Switcher or Quick Locale Switcher, a user can switch between language packs that are installed on his or her profile.)

In closing, it’s not bad to be a language pack!  I get the sense that somehow contributors might think it is.  We don’t diminish someone’s contribution to an arbitrarily lower level if they are not an “official” localization.  But, for the resource and demand reasons mentioned above, we often like to promote language packs as a solution.

Initially, I sketched out what I thought would be valuable information for the report, ran it by the l10n-drivers, and sent it to the metrics team to start implementation.  In my opinion, an effective report provides both download and active daily user information to our localizers about their locales AND the geos in which their locales are being used.  Let’s review the contents for those who might need a guide.  Feel free to reference the attached screen shots as you read.

Locale-specific information

We are presenting both the download and active daily user (ADU) information (usages statistics and pie charts) for versions of Firefox.  ADUs are based on the blocklist pings we track.  (More on blocklist can be found at Morgamic’s post.)

Geographic-specific information

Each report will show both the download and blocklist for the top five locales inside a country where the localizer’s translated Firefox is most prominently used.  In many cases, this is easy to map.  Locale code “fr” is probably most prominently used in France.  “de” in Germany.  “es-ES” in Spain.  In some cases, we’ll have to make guesses, like for our Kurdish localizers.   Finally, we will provide a list of the top ten countries (by average blocklist pings) where the localizer’s Firefox is being used.

For the first time, our community of l10n volunteers will have a more comprehensive set of data points to help measure the progress and spread of their work.  By providing both locale and geographic information, these reports illustrate the impact that each localization team is providing.

Below are two images of a sample two page report.

and

“In Mozcamp i said, mozilla is no helping just to translate firefox to mayan, mozilla is helping to rescue and make strong the mayan language. Well, i’ll send it and we’ll be in contact.” [sic]

Aw, shucks.  That just makes me happy.

Maybe I have delusions of grandeur as I sit here and sip my Kool-Aid, but I think there is something critical to language preservation happening in the Mozilla localization project for cultural anthropologist and linguists to study.  I’ve discussed this topic with other Mozillans who are interested like Tiffney Mortensen, Chofmann, Staś, John Lilly, Søren Skrøder (Mozilla Denmark), and Kadir Topal (Mozilla Germany).  Every time we ship a new version, even for some of the most niche locales, Mozilla helps just a little bit to preserve the culture of language and communication.  Imagine how unique an experience it becomes for a total newcomer to browse the web with an application whose user interface is both translated and customized for local use.  That can be very powerful and is why we want Mozilla locale count to continue to grow.

To see a little more about what our Mayan friends are doing, check out these links:

• Mayan Firefox language pack
• Screen shot of Mayan Firefox
• Mozilla México en Mérida, Yucatán
• Local press

Do you know of a new localization effort?  I will pay chocolate dipped cake donuts for every referral that becomes a localization.