-
Localization-QA survey results
At the end of last quarter, we launched and analyzed a survey about l10n testing needs. The hope was to find areas where we could provide help to our localization community around how to test their builds. The analysis was done by Mozilla contributor, Murali Nandigama, and he posted his slides here. The next step is to implement something from those findings for the community. Please take a look at the analysis and make help me with some conclusions.
Here are some takeaways our team noticed when we looked at the slides.
- Localizers are enthusiastic to see more automated tests and automation frameworks for testing compared to adding more Litmus manual tests.
- Survey participants might be assuming that more automated tests would provide the same functionality as Litmus manual tests.
- However, especially in the L10N context, font-rendering, text clipping, dialogue/character spacing etc., can be eye-only tests and might not be substituted by automated tests without a lot of non-trivial automation effort.
Conclusion: Litmus manual tests are important and hard to automate to replace visual verification. Has everyone played with Litmus?
- 95% of survey participants are using non-Bugzilla channels to report issues, problems, and bugs. (Or, only 5% use Bugzilla to report issues.)
- Because of this, we cannot quantify and archive the effort of l10n volunteers in finding bugs.
- How are people logging issues with local problems?
Conclusion: We might need to blog to demystify the bug filing process for l10n volunteers. Or, provide easy tools like bug-by-email templates.
- In the survey, we specifically asked if l10n communities are testing for a things like non-US keyboard input, clipping of text in boxes, RTL etc.
- A majority of communities are either not aware of these testing steps have no need to test these issues (like RTL).
Conclusion: We might have to provide a generic blue-print for effective test planning for any locale, indicating what are the key l10n steps in test planning.
- In the open comment section, we saw responses like this:
- Survey takers indicated that teams larger than 10 are using the l10n builds.
- Teams indicated that only one or two members are actually reporting problems and filing bugs.
- Some indicated that they do not have a formal test plan in place, no institutionalized knowledge-base on testing locales, and lack a coordinator/leader for their l10n testing efforts.
Conclusion: We might need a third-party QA service to help lead volunteer L10N test activities.
I’m still sifting through the data to find who might benefit from some of these recommendations and findings. If you’d like to make my job easier, please comment on what you think your locale needs. Questions are welcome too, please.
Many thanks to Murali who did the analysis posted above and helped me write this post.
-
Some help from a Kiwi
Staś and I had a nice Mozilla Community moment last week.
Turns out, some people have been reading our blog about the Community Survey program. So, that’s not really a big deal since we syndicated it on Planet and have been writing a lot about the first three surveys. (Here are links to surveys #1, #2, and #3.)
But, the cool moment came when a statistician named John Williams from New Zealand approached me and Staś and offered his expertise for the survey program. He’d been reading our write-ups and decided that he could probably add a bunch, noting that he really liked what he read so far and would like to help take the analysis even further.
Here is a quick bio on John:
“John Williams is a lecturer in the Department of Marketing at the University of Otago in Dunedin, New Zealand. He specialises in teaching Marketing Research and has degrees in Marketing, Economics and Statistics. He has been using Free Software since before the turn of the millenium, and is attracted to the consumer empowerment and related business issues that Free Software raises.”
John introduced himself and we immediately set a meeting to discuss his involvement. The meeting was easy to plan even though I am in California, Staś is in Poland, and John is in New Zealand. After our initial discussion, he did the following:
- re-analyzed survey one to confirm results and give some new insight;
- introduced new techniques to our plan, including “Principal Axis Factoring” to compliment our “common factor analysis”;
- created this wiki: http://wiki.mozilla.org/Mozilla_Community_Surveys;
- started a new IRC channel #surveys;
- and challenged us to think about managerial questions we should ask after seeing the results of our “satisfaction-based” questions. Why did someone answer with a lower rating of satisfaction on one question compared to another person? And, what do we do to fix it?
John also volunteered to finish the analysis of the second survey, which has been lingering for a while due to the two other surveys we have been conducting and other obligations like Staś localizing stuff for Firefox 3 and me working on the Screencast Contest. (Did you submit your screencast yet? You know you want to enter, just get it over with and do it!)
So, why did John volunteer to do all this for us? His quote was something like, “Hey, I do this kind of stuff for big companies all the time. It’s nice to get to do it for an organization like Mozilla, which is really making a difference.” That’s paraphrased, but you get the picture. What a guy!
Thanks, John. We’re looking forward to your experitise and contribution.
-
The team writing about Mozilla’s community survey results
Lately, I’ve been managing two blogs, this one and the Community Surveys blog. I wasn’t sure how many people subscribe to planet.mozilla.org or just to the feed for my blog, so I thought I would cross-post what I’ve been writing there.
Two links:
- A series of posts reporting the results from our first survey about the Mozilla Community
- The ongoing series about the results from the second survey regarding Mozilla Support
Many thanks to my colleauge in this effort, Stanisław Małolepszy. Staś (his nickname pronounced Stosh) has created every graph you see in these posts and done the bulk of the raw data analysis. Together, we’ve done the interpretation of the data and drawn the conclusions we’ve made. Though I have done a bulk of the writing, Staś has also taken first cut at many of these posts, and then I have edited. As a non-native English speaker, Staś has really been able to write some eloquent parts.
Every now and again we all get to work with a colleague who seems to catch every error, suggest great ideas, and complete your thoughts. That’s Staś. Hope you all get to work with him one day. Here is his blog if you want to see what he’s up to.
Umm…you might have to read Polish…but the pictures are cool!
-
A peak at the Community Survey presentation at FOSDEM
On Sunday at FOSDEM, Stas Małolepszy and I did a presentation about the Community Survey Program, displaying the results from our first two surveys and really trying to engage the audience to ask questions and provide feedback. I thought the presentation went pretty well. (Please let me know what you thought if you were there.) Stas is still working on some final analysis and then he and I will write a blog post with a lot of detail about what we learned.
Here’s a taste of the presentation. Just some tidbits….
Below you will find the mean responses and standard deviations for each question we asked in the first survey. Remember, in the first survey we asked two “big” questions that asked community members to disagree/agree (on a scale of 1-5) with the statements we made.
You’ll see on the graph the first six answers with an “@” symbol. Those are the elements of the first question. The question said “I feel that
- … I am satisfied with the amount of people involved with my local community project in the past year
- … there are more active newcomers to our community compared to last year.
- … it is easy to become involved at some level (localizing, developing, marketing, etc.) with the Mozilla project
- … I am connected to Mozilla and that Mozilla is interested in our community efforts to work on the project.
- … Mozilla is helping to grow local communities.
- … the overall health (satisfaction, enthusiasm, leadership, participation) of the community is good.
The next question read “What resources and actions would be necessary to make the community more satisfied and productive?”
- web hosting
- web storage
- more goodies (hardware, software, community-wide tools, marketing supplies, T-shirts, badges, etc.)
- helping to plan community meetings and events
- financially supporting community meetings and events
- helping the local community organize its members
- helping establish a legal entity (e.g. a non-profit organization) if necessary
- having a PR agency to help with press inquiries
- a visit from Mozilla to the community members
- providing a Mozilla website template for community websites in different locales
The summary statistics were encouraging. All means of the response were above the midpoint of 3. So, it may not be a stretch to think that the community believes the health is good. When asked about areas where we could provide support, we saw some interesting things. For instance, if you look at the last response, you’ll see that providing a “website template” to communities has the highest mean. Makes sense, doesn’t it? If we can provide a template for communities, it makes them more official, creates a sense of team, helps with Mozilla branding, and allows new communities to start up quickly. Our next step? Simply, we will investigate creating a template for communities to use. Simple findings like this make the program very valuable.
Data visualization is critical when you collect volumes of information. Stas used a statistics program called SPSS to generate the graphs. You’ll see later in our extended write up a lot of great graphs and illustrations of information.
We also were able to perform more than just summary statistics. One interesting thing we tried is called factor analysis. When a survey has many response variables, you can run a factor analysis to see if any set of variable can be condensed into a more easily explained, larger variable. Again, more on this in our detailed report.
The trick with factor analysis is to look closely at how each response variable relates to the others that load onto the new variable. Then, you have to name the new variable that explains why the responses all loaded where they did. We were able to “make up” two new variable where all the other responses could load. One variable, we called “Community Health and Dynamics” and the other “Feeling close to Mozilla”. Then we started to compare. Because of the way we designed the survey tool, we were then able to look at each locale to see how they compared after the factor analysis.
Below is picture of the factor analysis of communities who use a Spanish translation of the browser. You can see Mexico feel a little “less healthy” than the Argentineans and the Spanish.
But, wait! Don’t jump to any conclusions just yet. This analysis is directly related to all the response variables we saw in the graph above. And, you’ll remember that all those answers were above the midpoint response of 3. So, this is saying that, the Mexicans, on average answered above 3, but were not as hyper-positive and exuberant as, say, the Argentineans. Like all statistical analysis, it’s really imperative to understand the data before jumping to conclusions.
Lessons learned?
We need to do a much better job of collecting demographic information. In the first survey, we had no idea we would have over 1,100 people take it. We sent it to a list of 200 or so people and it propagated to a point where over 8,000 people saw the survey and 1,100 took it. But we had no demographics… It doesn’t render the information useless, but we need that…so we added it in the second survey. A quick demographic question helped us qualify the responses even better. Next time, we’ll add a bit more to the demographic question just so we know exactly who is taking the survey.
We also learned that it is important to include a “Don’t Know” option when asking about very specific things like video or online support from Mozilla.
And finally, we learned the old lesson that not everything translates perfectly into all languages. No real way around this except to try our best.
As I mentioned, Stas and I are working on a final report that will summarize a bunch of this. If you have questions, please come get me. I’d be happy to listen to suggestions or provide answers where possible.
-
Behind the Scenes of the Community Survey Program
In November of 2007, Stas, Pascal and I formally launched the Community Survey Program. I blogged about it back then, describing just what we were hoping to do with the program. Today, I’d like to open up a bit on how a team of us has worked together to make a meaningful survey for the Mozilla community.
In the last survey, we focused on product support by Mozilla. Stas suggested the topic and we reached out to David Tenser to see if he was interested in helping us. Not surprising at all, David responded with an enthusiastic, “Let’s do this!”
So….on collaborating!!!
Each of us brought a specialty to the group. I have had some experience with survey design, having taken a few classes on survey methodology and launching a few with other organizations. Stas had much of the technical expertise, having developed the PHP online tool. And, like most software engineers, he brought some critical thinking with an excellent ability to tackle what we hoped to accomplish. David brought his own Scandinavian intensity, a deep knowledge of the support issues, and a good understanding of how to construct surveys. The three of us made a great team, bouncing ideas off each other, challenging our assumptions, and working out particulars like question phrasing and focus areas.
With each of us living in different countries, our geographic locations could have made our working group a bit challenging: David is in Sweden, Stas in Poland, and I am in California. But, we all tried hard to be open and flexible for meetings and working together, whether it was getting up early or staying up late. It may sound like a no-brainer, but it was a simple step toward success. Also, we used VoIP to chat for free. Before the end of the survey design, I felt like Stas and David were right across the office instead of the ocean.
Our next big task as a group was to agree on the goals of the survey. We agreed on the following:
1. To understand and measure how the Mozilla community is doing at support in the different locales.
2. To get a better understanding of people’s perception of support provided by Mozilla.
3. To find out how can we make support by Mozilla more useful.As we began writing the questions, we would occasionally come to a stopping point. If so, we always asked, “Does this issue relate to one of our goals for this survey?” If the impasse had nothing to do with the above three goals, we dropped it immediately…no further discussion. Focus was key, especially when collaborating across continents over the Internet.
As far as the content creation, we relied heavily on David to draft the questions. He is probably most familiar with the issues in the support community. We had the objectives and decided to create a question for each goal, hoping that we could get good insight into each of the three goals we articulated from the beginning. It turned out to be a pretty clean and productive process. Check out the survey and take it if you’d like.
Here’s an example to illustrate the process. Take the first goal:
1. To understand and measure how the Mozilla community is doing at support in the different locales.
We created the question:
“How would you rate the support that community provides for Mozilla products in your language?”
This question was particularly tricky to ask because we knew that each locale has its own support community. How would we ask appropriately across all the locales that provide support without leaving out something? David listed what he thought were the universal elements that existed in support across locales. Knowing that we were going to post questions in 15+ languages without knowing who exactly might take the survey, we were careful to use language that was easily understood, could be translated into 15+ languages, and touched on exactly what we had hoped to learn from the process.
For the actual survey design, we chose to create three “likert scale” questions to gather detailed information about Mozilla support, with one short demographic question at the end of the survey to help qualify who was telling us what.
(Skip this next part if you don’t want to read a bit of background about our design.)
I’ll take a moment to specifically describe the survey design methodology. The purpose of a likert scale question is multi-fold. For one, it allowed us to ask a very complex question, rating several variables on an ordinal scale. With this technique, we could keep our surveys short (a key to a high and accurate response rate in survey design) while still gathering a lot of meaningful data that can be analyzed to draw conclusions. The format appears as one question, but it is really several in one. Typically, the likert-style survey contains a set of possible response variables to a question and asks the survey taker to rate each response on some scale. (”1″ usually being something like “Completely Disagree”, and “5″ something like “Strongly Agree”) With the data gathered, we could then test each variable against the others. In our analysis, we’ll do simple cross-tabs and pivot tables that will show us information like “those who said X also said Y”. And, with many dependent response variables in one question, we can run statistical regressions on the data to find significance and eliminate those response variables that were not significant. I hope this explains well why we used the likert-style question format.
We also threw in one demographic question so we knew “who” was taking the survey. We weren’t sure exactly who would see it online since it was open to anyone. So, we asked each person to tell us just a bit about his or her involvement in the project. We gave people the ability to tell us if they were an “end-user” or more deeply involved like someone who writes code for the Mozilla project.
After three or four meetings among David, Stas and I, we had our survey ready for translation. Stas placed the survey on a staging server, invited localizers to translate, and then check-in their translations to the code repository he had created for the survey. After all translations were complete, we launched. The first survey (about the Mozilla community) was viewed by over 8,000 people and taken by over 1,000. The second survey (about Mozilla support) has had over 1,400 people take it. More than enough responses to have a good sample to test.
Right now, we are still analyzing the data from our first two surveys and hope to present it online soon. Additionally, David, Stas and I are hoping to present this material at FOSDEM, speaking more about this process and discovering new topics we can survey.
Finally, I have to say that it was a real pleasure working with David and Stas. With each of us having our own working style, we came together across time zones, languages, and continents not only to create a great survey, but also to have a lot of fun doing it. For instance, if we had a debate in a word choice that we might use for a particular question, I’d try to illustrate my point with an example, sometimes pitting Poland vs. Sweden in a hockey match, or something like that. (e.g. …on using the word superfluous: “The fourth goal by Poland was superfluous in the team’s victory over Sweden.”) By the end of the project, I am sure both Stas and David were ready to start slamming some of my favorite sports teams. Sorry guys!
Overall, a great project to work on. Hopefully we can do more, so if you have an idea for a survey, please do let us know.
And, let me know if this was at all a helpful post. Sometimes it’s hard to tell. Hope you’re still reading.
community website
