What resources make or would make the user-to-user support in your language more effective?

We already saw the average responses to this question in one of our previous post in the series. Here there are again:

As you can see, resources based on text were ranked higher than any others. On the other hand, Instant Messaging or Tutorial videos were ranked much lower, ending slightly above ‘Neutral’. It might be interesting to see what impact on these results had the profile of the survey takers. Thanks to the demographic variable that we discussed previously, we were able to learn how each profile responded to the question.

The above chart shows how many respondents of each profile took the survey. 19% of the survey takers were end-users and 32% were community members actively contributing to the project. Is it possible that the responses given by advanced users (who were more numerous) outweigh those of end-users in this question? What preferences do the end-users have and, consequently, what is the best way to improve their support experience?

In order to compare the average responses between the user profiles, we reduced the number of analyzed variables from 10 to 3, by means of factor analysis. The new extracted variables were:

• Textual support, representing:

Tutorials/How-to’s
Troubleshooting articles
Forums

• Visual support, representing:

Screenshots
Screencasts/videos

• Interactive support, representing:

Live Chat
IM
VoIP

Average values of these new variables are visualized in the following chart.

We clearly notice that the end-users are much less enthusiastic towards textual support, and in the same time they prefer interactive help slightly more than other users. The visual support has been similarly rated across all user profiles (the differences weren’t statistically significant—hence the grey color in the chart).

In summary, thanks to the factor analysis, it was possible to make the comparison of preferences between user profiles easier. It turned out that even though the interactive resources, such as Instant Messaging and LiveChat, were ranked lowest by all respondents, there were significant differences between user profiles in the responses. We learned that the end-users favor textual support much less than other groups and show a slight preference of the interactive support.

Good support is an essential part of the product experience in its holistic approach. It is even more important for end-users who may be even using the product for the first time. This is why it was important for us to see what this particular group though about Mozilla support experience. Our study may serve to confirm that indeed the introduction of LiveChat on support.mozilla.com was a valuable addition which will help users looking for support.

* * *

This is our last post in the series about the SUMO survey. The series included:

• the first post, in which we presented the objectives of the survey and the questions,
• the second post, in which we discussed who the survey takers where and what languages they spoke,
• the third post, in which we analyzed the overall average responses and dove into the differences between the user profiles,
• the fourth post, in which we focused on the differences between locales,
• and this post, the fifth one, in which we looked closer at the responses to the last questions of the survey.

We hope you liked the series and found the data as fascinating as we did. We encourage you to post your thoughts in the comments or join us on #surveys on irc.mozilla.org.

In our series about the support experience survey, we already discussed who the survey takers were, as well as their average responses to all the questions (and the differences between user profiles). Today, it’s time to look into the differences between locales.

Question 1: Support experience rating

As we have often mentioned previously, doing an analysis of the differences between locales can be tricky. There are some cultural factors that can play significant role in how people from each locale respond to a question. We will not, however, explore these factors now. We will stick to the data we have from the survey, but keep in mind that some differences may be cultural ones.

This is how John Williams, who offered his help with the analyses we’re doing, explains this using the example of the first question of the support survey (”How would you rate the support that community provides for Mozilla products in your language?”):

The table below shows how many times each locale rated a product the lowest, and how many time each rated a product the highest. Note that because of the overlapping feature of multiple comparisons some the counts do not sum to the number of products.

It seems clear from this summary that German and French respondents generally rate the products lower than do respondents from other locales; conversely Polish and Italian respondents rate the products higher.

Let’s take a look at the average responses to this question:

You can certainly see that Firefox support has been rated the highest. There are two observation to be made here:

• Firefox is the main Mozilla product, so its support resources are the most developed
• Firefox is also the most known Mozilla product, and as such, many users are familiar with its support resources. There were very little missing values (’I don’t know’ responses) among the responses to the Firefox rating question

We also notice (just as John predicted) that the average responses to all three questions were generally lower for such locales as French and German, and higher for Hungarian, Italian and Polish.

The average responses to this question continue in the next chart.

Generally, the responses to these products were lower than to Firefox and Thunderbird. It is still interesting to see that the support for add-ons has been rated well (slightly below 4 for all locales), similar to Mozilla overall and Thunderbird from the first chart. And even if Calendar and SeaMonkey support are rated lowest in the chart, it’s important to remember that in general, all products have been rated on the positive side of neutral (above 3).

In this chart we’re showing the average responses for a product for each locale. You may remember that we also asked about Camino, which is not shown in the chart. This is because Camino had a lot of missing values (almost 75% of all responses) and the differences in responses for it between locales are not significant.  If you’re interested in the overall average across all locales, see our previous post about the SUMO survey.

Question 2: Characterizing support

Nine statements in our survey asked survey takers to characterize the support experience, which translated into nine variables. With nine different locales to compare, this gives us 81 figures to analyze, which was too many!  In order to make comparisons easier, we first looked at two variables covering quality of localization and then, we reduced the others to one variable that covered quality of content.

Localization of the support resources

We asked two questions related to the quality and sufficiency of localized support resources:

• Most of the relevant information can be found in my language
• Support in English is sufficient

The average responses to these two questions are show in the chart below:

Although statistically different, the average responses are very similar for all locales: in general, the survey takers agreed that most of the relevant information can be found in their language, as well as disagreed that support in English is sufficient (in the chart, value 2 means ‘Disagree’).

Support experience assessment

6 variables related to the quality of content were strongly correlated which allowed us to replace them with a single variable using the factor analysis. These six variables were:

• Most of the relevant information can be found in my locale
• Navigation on the support sites is easy
• The support sites are up-to-date
• Most common issues are covered well
• The support sites load quickly
• New issues are quickly addressed

We extracted one new factor (one new variable summarizing all six others) with similar loadings from all 6 variables. Consequently, we named it “support experience assessment” and used it to easily compare the support experience between locales.

Again, we see differences between locales that are most probably due to cultural factors. It is still interesting, however, to see that Italian and Russian locales score highest, whereas Polish, Portuguese and German score lower than other analyzed locales. We should also notice that in the beginning of the post, we included the Polish locale to the group of the ‘eager’ ones, so probably its low score in the last chart should be investigated in more detail.

What’s next?

In the next and last post about the SUMO survey, we will take a closer look at the last question, in which we asked what resources would make the user-to-user support more effective.