Community Survey Blog

This is a post about SUMO, the survey we conducted before our most recent survey.  Although we just wrote about the Mozilla Newsletter, we are still wrapping up our analysis of the SUMO survey.

In our series about the support experience survey, we already discussed who the survey takers were, as well as their average responses to all the questions (and the differences between user profiles). Today, it’s time to look into the differences between locales.

Question 1: Support experience rating

As we have often mentioned previously, doing an analysis of the differences between locales can be tricky. There are some cultural factors that can play significant role in how people from each locale respond to a question. We will not, however, explore these factors now. We will stick to the data we have from the survey, but keep in mind that some differences may be cultural ones.

This is how John Williams, who offered his help with the analyses we’re doing, explains this using the example of the first question of the support survey (”How would you rate the support that community provides for Mozilla products in your language?”):

The table below shows how many times each locale rated a product the lowest, and how many time each rated a product the highest. Note that because of the overlapping feature of multiple comparisons some the counts do not sum to the number of products.

It seems clear from this summary that German and French respondents generally rate the products lower than do respondents from other locales; conversely Polish and Italian respondents rate the products higher.

Let’s take a look at the average responses to this question:

You can certainly see that Firefox support has been rated the highest. There are two observation to be made here:

• Firefox is the main Mozilla product, so its support resources are the most developed
• Firefox is also the most known Mozilla product, and as such, many users are familiar with its support resources. There were very little missing values (’I don’t know’ responses) among the responses to the Firefox rating question

We also notice (just as John predicted) that the average responses to all three questions were generally lower for such locales as French and German, and higher for Hungarian, Italian and Polish.

The average responses to this question continue in the next chart.

Generally, the responses to these products were lower than to Firefox and Thunderbird. It is still interesting to see that the support for add-ons has been rated well (slightly below 4 for all locales), similar to Mozilla overall and Thunderbird from the first chart. And even if Calendar and SeaMonkey support are rated lowest in the chart, it’s important to remember that in general, all products have been rated on the positive side of neutral (above 3).

In this chart we’re showing the average responses for a product for each locale. You may remember that we also asked about Camino, which is not shown in the chart. This is because Camino had a lot of missing values (almost 75% of all responses) and the differences in responses for it between locales are not significant.  If you’re interested in the overall average across all locales, see our previous post about the SUMO survey.

Question 2: Characterizing support

Nine statements in our survey asked survey takers to characterize the support experience, which translated into nine variables. With nine different locales to compare, this gives us 81 figures to analyze, which was too many!  In order to make comparisons easier, we first looked at two variables covering quality of localization and then, we reduced the others to one variable that covered quality of content.

Localization of the support resources

We asked two questions related to the quality and sufficiency of localized support resources:

• Most of the relevant information can be found in my language
• Support in English is sufficient

The average responses to these two questions are show in the chart below:

Although statistically different, the average responses are very similar for all locales: in general, the survey takers agreed that most of the relevant information can be found in their language, as well as disagreed that support in English is sufficient (in the chart, value 2 means ‘Disagree’).

Support experience assessment

6 variables related to the quality of content were strongly correlated which allowed us to replace them with a single variable using the factor analysis. These six variables were:

• Most of the relevant information can be found in my locale
• Navigation on the support sites is easy
• The support sites are up-to-date
• Most common issues are covered well
• The support sites load quickly
• New issues are quickly addressed

We extracted one new factor (one new variable summarizing all six others) with similar loadings from all 6 variables. Consequently, we named it “support experience assessment” and used it to easily compare the support experience between locales.

Again, we see differences between locales that are most probably due to cultural factors. It is still interesting, however, to see that Italian and Russian locales score highest, whereas Polish, Portuguese and German score lower than other analyzed locales. We should also notice that in the beginning of the post, we included the Polish locale to the group of the ‘eager’ ones, so probably its low score in the last chart should be investigated in more detail.

What’s next?

In the next and last post about the SUMO survey, we will take a closer look at the last question, in which we asked what resources would make the user-to-user support more effective.

Trackback URI | Comments RSS