Localization-QA survey results

At the end of last quarter, we launched and analyzed a survey about l10n testing needs. The hope was to find areas where we could provide help to our localization community around how to test their builds. The analysis was done by Mozilla contributor, Murali Nandigama, and he posted his slides here. The next step is to implement something from those findings for the community.  Please take a look at the analysis and make help me with some conclusions.

Here are some takeaways our team noticed when we looked at the slides.

• Localizers are enthusiastic to see more automated tests and automation frameworks for testing compared to adding more Litmus manual tests.

1. Survey participants might be assuming that more automated tests would provide the same functionality as Litmus manual tests.
2. However, especially in the L10N context, font-rendering, text clipping, dialogue/character spacing etc., can be eye-only tests and might not be substituted by automated tests without a lot of non-trivial automation effort.

Conclusion:  Litmus manual tests are important and hard to automate to replace visual verification.  Has everyone played with Litmus?

• 95% of survey participants are using non-Bugzilla channels to report issues, problems, and bugs.  (Or, only 5% use Bugzilla to report issues.)

1. Because of this, we cannot quantify and archive the effort of l10n volunteers in finding bugs.
2. How are people logging issues with local problems?

Conclusion:  We might need to blog to demystify the bug filing process for l10n volunteers.  Or, provide easy tools like bug-by-email templates.

• In the survey, we specifically asked if l10n communities are testing for a things like non-US keyboard input, clipping of text in boxes, RTL etc.

1. A majority of communities are either not aware of these testing steps have no need to test these issues (like RTL).

Conclusion:  We might have to provide a generic blue-print for effective test planning for any locale, indicating what are the key l10n steps in test planning.

• In the open comment section, we saw responses like this:

1. Survey takers indicated that teams larger than 10 are using the l10n builds.
2. Teams indicated that only one or two members are actually reporting problems and filing bugs.
3. Some indicated that they do not have a formal test plan in place, no institutionalized knowledge-base on testing locales, and lack a coordinator/leader for their l10n testing efforts.

Conclusion:  We might need a third-party QA service to help lead volunteer L10N test activities.

I’m still sifting through the data to find who might benefit from some of these recommendations and findings.  If you’d like to make my job easier, please comment on what you think your locale needs.  Questions are welcome too, please.

Many thanks to Murali who did the analysis posted above and helped me write this post.