Community Survey Blog

Mean responses and the missing values

In the last two posts about survey 2, we learned the purpose of the survey and who took it.

For this post, we will start looking into the responses to our questions. Remember that we asked the following:

1. Please rate the overall quality of the support for each product;
2. How much does the user agree or disagree with a list of statements describing the support experience;
3. Please rate the usefulness of particular content types (tutorials, screencasts, chats, etc.).

We can start by summarizing the means to each answer, typically the easiest statistic to interpret and visualize.  But, as a forewarning, the chart below is pretty long!

Darker bars are the mean responses and the thin vertical grey lines represent the levels of the scale for each question:

• ‘Lowest quality’ to ‘Highest quality’ for the first question
• ‘Strongly Disagree’, ‘Disagree’, ‘Neutral’, ‘Agree’ and ‘Strongly Agree’ for the remaining two questions

Perhaps it’s not surprising, but Firefox’s support was rated highest (our guess: because of its depth and completeness), whereas products like SeaMonkey, Calendar and Camino had a very high ratio of missing values (the ‘I don’t know’ responses).  As the support rating lowers for each product, the percentage of the missing values increases, which also constitutes an important piece of information.  60-70% of the survey takers didn’t know how to rate the quality of support for SeaMonkey or Calendar—probably because they hadn’t used support or the application itself.  For Firefox, we can say that its support was rated between 4 and 5, where 5 was “highest quality”.  NB, only about 6% of the survey takers didn’t know how to rate Firefox support.   Please keep in mind that every respondent was thinking about support in their language, so in our next post we will calculate the mean responses to this question for every locale.

For the next two questions, we should interpret the bars differently.  Each of the thin vertical grey lines corresponds to a level starting with ‘Strongly Disagree’ on the left, through ‘Disagree’, ‘Neutral’, ‘Agree’ and, for the rightmost line, ‘Strongly Agree’.  The length of the darker bar for the first statement means that, again on average, the respondents tended to agree that “the support sites loaded quickly.” As for the brighter bar, we interpret it in the same way as above: about 10% of the survey takers didn’t know how to react to this statement.

When you look at the average responses to the second question, you’ll see some interesting findings:

1. “Support in English is sufficient” scores lowest (between ‘disagree’ and ‘neutral’), and at the same time, it has very few missing values.  So, people certainly had an opinion on this one since there were not many “Don’t Know” responses.
2. Two to three other statements have neutral response means (the corresponding dark bars end close to the middle of the chart)
3. Six statements show that the survey takers well above the midpoint and reflecting agreement.

Is this helpful data if we see mostly “Agrees”, “Neutrals” and one “Disagree”? 

Keep in mind that every person who responded to these two questions reflected on their unique support experience and then submitted their response. Also, remember that support of the Mozilla products

• can be accessed in different languages (and the quality of the support depends on quality of the localization),
• is often decentralized (each local community has its own discussion board),
• and aims to satisfy each users distinct needs.

If the cumulative responses were “Agree”, the means may still differ across certain groups of people.  We can see that the “Most of the relevant support can be found in my language” statement scores high (respondents tended to agree with it), which is a good first step toward understanding how the local communities are doing in terms of the user-to-user support.  One interesting thing to note is the “The support sites make use of video” statement.  It scores low, but also has an outstanding ratio of missing values.  Could you argue that an “I don’t know” answer is close to “Disagree”?  If someone hadn’t ever seen a video on a support site, but was careful in their judgments and supposed that there might be other support sites where videocasts were available, they were probably more likely to answer “I don’t know” instead of “Disagree”.  In this survey, we assumed that an “I don’t know” answer equaled “I haven’t ever seen any of them”.  Therefore, we decided that the survey takers who described themselves as end-users were more likely to give an “I don’t know” answer to this statement.

Mean responses by user profile

Thanks to the user profile variable, we were ablt to split the cases and plot new charts for 3 different user profiles. These were community members (49% of respondents), active community members (localizing, coding, etc.; 32%) and end-users (19% of the survey takers).

We observe the biggest discrepancies between two extreme profiles: end-users and active contributors. The community member profile seems to be in the middle for most of the questions.   For example, end-users rate the quality of the support for each product poorer than the contributors (green bars are longer than the yellow ones). More visible differences can also be observed in the “Most of the relevant support can be found in my language” statement. It seems that active contributors are more enthusiastic here.   Why is this the case?  Perhaps, it’s due to prior knowledge about the products (and computers in general), greater familiarity with the technology, and a better command of English.  What do you think?

Two other statements bring forward even bigger differences in the way the end-users and the active contributors responded.

• “The support sites make use of video”

Why do end-users agree more than active contributors that there are support sites using video?  This is probably a result of what we have discussed above: respondents might have preferred the “I don’t know” answer to this question over “Strongly disagree” or “Disagree”, even if these ones would have been more accurate. We can see that there were nearly 50% of end-users answering “I don’t know” to this statement, and only half as many of the active contributors. Some end-users did agree on this topic, but some didn’t and we can assume that some of them selected the “I don’t know” answers instead of “Disagree”. Using this theory, we can understand why end-users scored this higher: there were not enough “disagree” responses to counterbalance the “agree” ones.

• “Users rarely ask the same question more than once”

It seems understandable that active contributors are more likely to disagree. After all, they’d probably seen lots of duplicated questions in the forums and Bugzilla.

In the third question we see that end-users tend to favor direct communication (IM, VoIP) over text-based communication, compared to other user profiles. We also notice that forums turned out to be one of the most favorite channels of communication for active contributors—and much less so for the other profiles.

Finally, here is the graph of the missing values across different user profiles.

* * *
In the next posts we will reduce the number of variables by means of the factor analysis. This will allow us to better synthesize the analysis without loosing much of the information enclosed in the original set of variables.

Thanks for reading this to the end!!

Today, we launched our third community survey, focusing on the about:mozilla newsletter.  It was announced in the newsletter this morning and will be open over the next couple weeks for you to take.  Please take a moment to complete the survey because we’d like to hear from you!

If you’d like to learn more about the newsletter, check out this wiki site.  Deb Richardson is the editor-in-chief of the newsletter and she puts so much work into it from week to week.  about:mozilla is written so readers can get a good summary of the key goings-on inside the organization and community.  But, we can make it better and this survey will help us to learn how.  Please take an extra 2 minutes to complete the survey to let us know what you think.  We promise it only takes that long…or less! 

Thanks everyone!

How many people took the second survey and who were they?

Over the course of three weeks, 1,558 people took the second community survey.  We published the survey in 16 languages, thanks to the work of the localizers from mozilla-europe.org.   This step was critical for the survey because we wanted to see how the user-to-user support was doing in each local community and to what extent English resources were useful to each community.

Below is the frequency of survey takers across the various languages in which we offered the survey.

For five of the languages (English, German, French, Portuguese and Spanish) we asked the survey takers to indicate their country of origin, since those languages are spoken in multiple countries. Here’s that breakdown:

Some initial comments:

• As you can see, 60% of the people who took the survey in English came from non-English speaking countries. We’ve done a pretty good job localizing this survey, but there are obviously people from other countries who speak other languages who are interested interested in the project.
• More than a half of surveys taken in Spanish were taken in the Latin and South America.
• France and the Germany dominated their locales.
• Look at the global participation!  We want to extend these surveys to Southeast Asia, Japan, and China…working with Gen and Mozilla China to do that this summer.

(Originally, the Community Survey project was targeted the European communities (as StaÅ›’s internship project at Mozilla Europe). We quickly shifted to a more global scope, and we were thrilled to see so many people from other parts of the world expressing their interest in the project and the survey. Thanks for your support!)

Here is another interesting question:  how many surveys were taken each day when the survey was open?  See the graph below:

As you can see, the peak happened two days after the launch of the survey. Quickly afterwards the number of surveys taken daily stabilized at around 50-60, but you can still see the Christmas and the New Year’s eve when we got the least responses. We hit a second burst two days before the end of the survey due to a friendly reminder posted on our blogs.

Demographics

This survey was the first one to feature a demographic question, a lesson learned from last time. Below is an illustration of the frequency of various responses to our question, “Who are you?”:

As you can see, we presented seven answers to the demographic question and allowed for multiple responses. To avoid difficulties in the analysis, we then aggregated this data into three separate variables:

• End-users — those who responded that they were users of Mozilla products, but didn’t select any other answer
• Community members — those who responded that they used Mozilla products and followed the news on the Mozilla project
• Active community members — those who respond that they were localizers, developers, users helping other users, and authors of the support articles for Mozilla products.

Keep in mind that creating these groups was an arbitrary decision we made to help with analysis. We assumed that people who selected “I follow the news on Mozilla” were part of what we call the “Mozilla community” or considered themselves members of this community.  What do you think? Should we simply ask another question “Do you consider yourself a member of the Mozilla community”? And, what about all those people who follow the news Mozilla just because they follow IT-related or Internet-related news in general? These are the kind of questions we face every time when working on our surveys. Feel free to join the discussion.

Below is a look at the number of survey takers in each of these new buckets that we created.

19% of the survey takers selected the ‘I am a user of a Mozilla product’ option as their only answer (the end-user profile), and 49% additionally responded that they followed the news on the project (the community member profile). Therefore, 68% of the survey takers (19% + 49%) don’t localize, don’t develop, don’t write documentation and don’t consider themselves actively participating in the forums. In fact, we reached a very wide group of people who feel connected somehow to the Mozilla project and were able to learn their opinions on support.

Finally, let’s take a look at the structure of the user profiles among the survey takers for each locale.

And the composition of locales by percentage…

The important next step, which we will start to discuss in the next post, is to look at how each question was answered by these profiles.  It should be pretty interesting to see how end-users, community members, and active community members are answering support-related questions across locales.

Thanks for your comments and reading this.  More to come on the second survey soon.

For the second community survey we focused our efforts on learning more about the community helping with support.mozilla.com (SUMO).

Where did we get this idea?

The topic for this survey came from Jesper Kristensen, a Danish localizer, who participated in a discussion on the Mozilla Europe’s mailing list about new survey ideas.  He wrote:

“SUMO (support.mozilla.com) is a great project, and it is probably going to support localization. But the project relies on volunteers from the local communities. I think it would be interesting to know if the local communities are interested in participating in that project. [...] David and Axel speculates if support of Firefox only vs. all Mozilla will influence how much volunteers will contribute to the project. Maybe a survey could be used to find the answer.”

We took this idea and expanded it to include Mozilla support in general.  It made most sense to then contact David Tenser, the project manager of SUMO, and invite him to help us with the objectives and questions.  Kindly, he agreed to help.

Preparing the questions

Just as we did in the first survey, we kept the questions short to help increase responses rates while still providing valuable information to us.  After a couple of long calls and e-mails, we agreed on a set of objectives listed below:

1. To understand and measure how Mozilla is doing at support in the different locales;
2. To get a better understanding on people’s perception of the quality of support;
3. To find out how can we make Mozilla support more useful.

We asked four questions in total, three of which directly related to our objectives.

In the first question,we asked the survey takers to gauge overall support on different Mozilla products.  In order to identify support-related issues in local communities, we provided a list of possible experiences for people to rate on a scale from one to five.

Question 1: How would you rate the support that community provides for Mozilla products in your language? (1- lowest quality, 5- highest quality)

• Mozilla overall
• Firefox
• Thunderbird
• Calendar
• SeaMonkey
• Camino
• Add-ons

Question 2: How accurate are the following statements for your community? (1 - least accurate, 5 - most accurate)

In my community…

• most of the relevant information can be found in my locale
• the support sites have a structure that is easy to read and navigate
• users ask questions in the right channels
• users rarely ask the same question more than once
• the support sites are up-to-date
• the support sites cover well the most common issues
• the support sites load quickly in my browser
• the support sites make use of screencasts and tutorial videos to help our users
• new issues/problems with Mozilla software are quickly added to support sites

Unlike last time, we made sure to include the “Don’t Know” option.  When we closed the survey, we turned up with a few missing values in our results, but it was still interesting to see how they were distributed, and how they could be interpreted.

For the third question, we asked about how support content might be delivered.

Question 3: What resources make or would make the user-to-user support in your language more effective?

• Tutorials/How-to’s
• Troubleshooting articles
• Support available in my language
• Support available from a single site
• Screenshots
• Screencasts/videos
• Specific versions of support depending on operating system (Windows/Mac/Linux)
• Forums
• Live Chat

And, you’ll remember the lesson learned from last survey:  Next time, ask a demographic question!  We did this time.  Without a specific target for this survey or control of who would take it, we could discover a bit more about who took the survey by asking the following:

Question 4: What’s your involvement in the Mozilla project:

[  ]  I’m a user of a Mozilla product
[  ]  I follow the news from the project
[  ]  I help other users in the discussion boards
[  ]  I write documentation
[  ]  I localize documentation
[  ]  I localize Mozilla products
[  ]  I develop Mozilla code

We then pushed these questions live in Stas’s survey tool and started to collect answers.  What do you think?  Do you think the questions are composed in a way that will elicit information that will meet our objectives?  Your feedback is appreciated.

By the way, a couple of weeks ago, Seth described on his blog the process of creating the second community survey. If you’re interested in a peek behind the scenes of the Community Surveys project, you might find his post interesting.

In closing, we realize this is coming a bit late, but with all the things we learned about survey analysis and presenting the results, time was not lost. Now that we know a few more of the challenges we may encounter, we can publish much more quickly.  A big thank you to everyone for supporting our efforts and keeping us motivated as you have done so far.  We hope it’s been enjoyable and informative.

It’s time to wrap up the first survey.

In our series of preceding posts, we excluded from the main analysis the countries that did not have enough responses for accurate statistical analysis.  But out of curiosity, we decided to present a short appendix to the main report just to see how other countries responded to our questions.  Each of the countries had between 5 and 20 responses, so it is more like reading the answers from a focus group rather than an analysis of volumes of data.  We used a more qualitative approach than we did in the report, but it’s still very interesting.

The list of regions we will focus on in the appendix include the following countries:

• Argentina
• Austria
• Catalonia
• Switzerland (German and French parts)
• Denmark
• Finland
• Great Britain
• Mexico
• Peru
• Romania
• United States

It is attached as appendix.pdf.

Last week, we posted a series of articles dissecting the results from the first Community Survey.   In this post, we’ll discuss our conclusions and then follow with one last post that will serve as an appendix to the series that we’ve written.

If you remember from the beginning, we talked about what we had hoped to learn from this exercise.  We’ve listed below a few expectations, and underneath each goal, what we discovered by conducting the survey.

• We will learn about the situation in the community


I think it’s fair to say that we accomplished this goal.  With our factor analyses and other descriptive statistics, we were able to draw some conclusions.  We found some already known and some new areas where Mozilla might consider supporting the community.  We also noticed that a lot of cross-cultural differences came into play.  Countries and culture are naturally different and will answer differently.

• We will understand how people perceive the relationship with Mozilla


Overall, it seems that the relationship with Mozilla is good.  We looked across all the responses and saw average answers consistently above the neutral midpoint.  We also learned more about what each locale expects in support from Mozilla.  The next step will be for Mozilla to initiate conversation with communities to discuss the findings.

• We will learn what they would like us to do  to improve


Perhaps most important, we learned about areas where we might be able to improve.  One specific finding was creating website templates for new communities.  A website template adds legitimacy to new communities and provides a valuable tool for people to use to get started quickly.  From Mozilla’s perspective, it helps establish a global brand by having consistent templates with messaging for the communities to use.  Mozilla can investigate if this is possible.

In addition to revisiting the goals of this survey, we also learned a couple lessons for next time around.   Here they are.

• Include “Don’t know” option


Did anyone notice that in the questions, we did not include a “Don’t Know” option?  We should have so people were not forced to answer something they may not have known about.  We will do this in all surveys going forward.

• Always include a “Demographic” question


After the survey closed, we found that we had collected a ton of data, but lacked any demographic information because we had no idea that so many people would take the survey.  We thought only 50 or so would take it and they were “active members” of our community.  Without restricting who can take the surveys, we will always add a demographic question (or few) to have a good understanding of exactly who is taking the survey.

In conclusion, our first survey with the Community Survey program had a lot of great findings, but also has room for improvement.  We learned so much about our community, and a lot about what we have to do for the future.

The Community Survey Program is meant for the community, so if you have a topic for a survey, please nominate it.  (If we select you, we’ll send you a t-shirt!)  Future survey topics will (or have already) include(d) support, Firefox 3, conference participation, and more.

In our last post, we talked about the factor analysis we used to help interpret the data we collected from survey 1.  Using that same technique in this post, you’ll learn about how the survey takers responded to Mozilla’s question about what resources might be helpful to the community.

Second question: Resources
The graph below shows how each country has responded to various questions about resources available for or important to the community.  Once again, the mean responses (for the most part) are above three across most of the countries.  It also seems that the best resource Mozilla Corporation could provide would be a visit form someone on Mozilla staff and a website template for the community.  Great findings for Mozilla!

Once again, we used a factor analysis to further understand how these responses related to each other.  With the factor analysis, we reduced the number of variables to analyze to three.  The chart below presents the correlation between the questions (old variables) and the factors (new variables).

You can see how responses clustered around these new factors.  The three hues of green correspond to three new variables we extracted, each relating to a different type of support the community might need. We can see that two first variables (web hosting and web storage) load significantly into the third factor (the lightest shade of green), which we decided to name “Websites”.  The second factor, which we call “Events”, relates to the questions about events, organization, financing, and providing the communities with more “goodies” (i.e. Mozilla T-shirts, badges, stickers etc.).  The last factor (dark green bars) is correlated with the last five questions about managerial and legal help from Mozilla, handling press requests, visiting local communities by Mozilla representatives, and providing an official website template for community sites.  We named this factor “Operational help”.

Like last time, our next step was to look how each locale tested against the new variable we created from the factor analysis.

For this factor analysis, we put more emphasis on comparing the three factors’ scores for each country.  Analyzing the factors country by country gives us a lot of knowledge about what what kind of support we should consider first for each community. 

 For instance, a longer bar for events for Hungary does not necessarily mean that Hungarians need events support more than the Polish community.  What it means is that support regarding events organization is more important for the Hungarian community than the websites support (the Hungarian pale green bar is negative).  For the Polish community, website support seems to be of a greater value. It also appears important to the French, Russian and Spanish.  But, we have to take the website support a bit deeper: what could be the challenges facing these communities that elicited these responses? Slow loading times? Outdated content or visual theme?  Need for more hosting space?  We have some ideas and now can ask to see what would be best.

We can also see that some of the communities might want Mozilla to help organize events.  We know from past experience that events really help the spirit of the community.  If Mozilla could help organize events, perhaps it would have a significant impact on the health of local communities in places like Hungary, Italy, Poland or Russia.  Perhaps an upcoming developer day would be best hosted in one of these countries?  Who knows…this is speculation time.  But, these are valuable conclusions to draw from the survey.

Finally, we can try to interpret the scores on the “operational” variable. It seems that the communities from Hungary, Italy and Spain feel that Mozilla could try to make the community seem more official through different ideas like Mozilla employee visits, helping with legal issues, and providing official Mozilla branding tools like a website template.

We can try to look at these results from a different angle as well.  We know that this survey propagated far beyond the intended audience of the “active members” list inside the European community.  Therefore, we can assume that the more people who took the survey, the farther we extended from the active members list, perhaps even to the end-users.  Consequently, the answers we received might show how all survey takers perceived their local communities, instead of what the original target, the active community members, might perceive as need.  For instance, in the case of the operational support variable, Poland and Russia have the lowest scores on this factor.  Could it be that many end users took the survey and they don’t believe that their communities need Mozilla “operational” help?  If you look at their main community sites: mozillapl.org and mozilla.ru respectively, you might admit that these communities have a very professional look.

In conclusion, each factor exposes areas that are decidedly different and important to each community.  It is interesting for us to see how the survey takers thought their communities would benefit.  Because the factor analysis shows that each community has distinct needs, interests, and is at a distinct stage of development, the appropriate next step will be for Mozilla to speak with each community about how it can help or where help is needed (based on survey results). But even more important, we would like the communities to confirm and discuss our findings so that we know how accurate they are.

We encourage you to post your comments to this post.  What are you thoughts, conclusions or questions about our analysis?  Please let us know where you are from and what you think about your country/community’s responses.

———————————————-

In our next post, we will summarize our findings with some conclusions.  We’ll also take a look at where we can improve and how we’ve already changed for upcoming surveys.

In our previous posts in this series, we covered demographic data and analyzed the community’s mean responses to the survey’s questions.  But, we suspected that answers would vary across locales.  As we continued to look at the data, we started to examine how different communities responded to the same question.  In this post, we will examine the differences in responses across locales to the first question about the overall health of the community.

First question: The overall health of the community
Here are locale specific answers to that familiar first question: how do community members feel about the community?  You can see that the responses seem pretty positive from country to country.

We then decided to go a bit further with that data we collected.

  One interesting thing we tried is called factor analysis.  When a survey has many response variables, you can run a factor analysis to see if any set of variables can be condensed into a new variable — one that is more easily explained by the common characteristic of all the original responses.  We did the factor analysis and were able to create two new variables called “Community Health and Dynamics” and “Feeling close to Mozilla”.

The above graph is a visualization of how we determined the two factors.  Notice how three of the variables gravitate to the vertical axis that we describe as “Feeling Close to Mozilla”?  Recognizing that, we decided to create a new variable to test, naming that variable “feeling close to Mozilla” since we felt that this was a common description of each of the variables.  The same goes for the horizontal axis.  Three variables gravitated toward what we have called “Community Health and Dynamics”.

With these two new variables from the factor analysis, we can test against our locales.

  In the next graph, you can see responses to the two new variables across locales.  But before anyone draws, remember that this analysis on the two new variables is directly related to the mean response variables we saw in the original question.  And, remember that all those answers were above the midpoint response of three.  Mean responses were above three and we felt pretty good about that.

From the graph, we can start to see that some locales have a slightly lower feeling of closeness to Mozilla (new variable one) and the health and dynamics (new variable two).  It is also important to note that some countries have much higher response rates than others.  So, drawing very concrete conclusions is hard to do.  Like all statistical analysis, it’s really imperative to understand the data before jumping to conclusions

Let’s focus first on Italy and Spain.  What might we conclude by these responses?  It seems that both Italy and Spain have responded with more exuberance than other countries about community health and dynamics and feeling close to Mozilla.  Remember though that this does not mean that Italy and Spain are “better” than others or more supported, it simply means that their communities responded with higher answers on the scale when compared to other locales.  

It also appears that the Russian community, on average, feels a bit less “connected” to Mozilla when comparing their answers to other countries.  Perhaps people from the Mozilla community can do better outreach in Russia to raise this response.  Good news: this may happen! We recently hired some new Mozilla staff to do direct outreach and community building in Eastern Europe.  In the future, it will be interesting to see if responses change after Mozilla takes action.

Interestingly, countries like Hungary and Poland had responses that were more toward the middle of the pack (less so for Poland).  They seem to have a pretty balanced opinion compared to other countries, feeling connected to Mozilla and that the community dynamics are good.

Time for some speculation.  Do cultural differences play into the results we see?  Is it unreasonable to think that different cultures would answer more or less enthusiastically?  We could ask the Spanish what they think.  But, we are not sociologists and are not geared to answer those questions — but, it is not hard to believe that they might.

—————————-

In the next post, we will look at the responses to our second question where we asked about the resources that Mozilla could provide to the community. We will apply again the factor analysis in order to reduce the number of variables and try to learn about the *real needs* of the communities.

This is the second post in our series about the first Mozilla Community Survey and it reveals the mean responses to all the questions that we asked.  You’ll remember from yesterday that we began with general information about the the survey takers.  Today, we’ll review how the community answered the questions we asked.

First look: The Means

Below is a graph visualizing the average responses of the community to questions 1 and 2.  We’ve ranked them in descending order to show the answers from highest to lowest score.  You’ll remember that we asked:

• “I feel that…”
• “What resources and actions would be necessary to make the community more satisfied and productive?”

You should be able to piece together what responses go with what question, but just in case, please refer to the questions in yesterday’s post to see which goes where.

A few interesting observations: 

Looking at the graph, you can see that each mean is above the midpoint of 3 (remember that 3 = neutral).  Therefore, we concluded that, on average, the community is optimistic about Mozilla.  Good news, right?  We thought so, but here’s some more…

• It appears that most Mozillans believe that more active members are joining their communities.  They agree that the communities are growing and Mozilla is helping to do that.
• They agree that it is easy to become involved in the community if someone chooses to do so.
• Most people think that the resources that are (or could be) provided to the community are helpful.
• Recognition is important and the Mozilla community members want to be perceived as more official local representatives of Mozilla.
• Mozilla can work on providing some quick ways to help the community with more resources.  One that scored particularly high was “providing a Mozilla website template for community websites in different locales”.  The other was “A visit from Mozilla to community members”. Interesting findings like this will help Mozilla provide better resources to its community.

The four questions that we see at the bottom of the chart are related to events organization and web hosting support.  While the mean responses were lower than the other questions, please keep in mind that the means are still above the midpoint!  In a forthcoming post, we will see that the responses to these four questions varied throughout the local communities.  The variety in answers ended up balancing out, giving us the aggregated mean that we see that is close to 3.

————————–

In our next post, we will investigate the differences between local communities using a statistics method called factor analysis.  You’ll have to put on the thinking caps because we’ll take a little bit of time to describe a factor analysis and why we used it…but not too much.  Mostly, we’ll just try to show you some pretty charts.

We hope to see you on our blog tomorrow!

At the end of 2007, Stas MaÅ‚olepszy and I closed Mozilla’s first community survey and began the experimentation of analyzing the survey responses and drawing some conclusions on what we had collected.  We’ve finally reached a pretty comfortable point and are ready to publish some results.  Keep in mind that we already presented much of this raw data at FOSDEM in February for the community to see.  While there, we received a lot of great questions and ideas about what to present.  We took that time to conduct to refine what we learned (and conduct one survey and start creating a third!).  We’ll present our findings over a series of three to four blog posts over the next few days.  Please stay tuned and ask as many questions as you like.

Measuring the Temperature of the Mozilla Community

Introduction
Mozilla created the Community Survey Program to learn more about the ideas and opinions of the people who make up our community.  We are using this program to ask various questions to learn how Mozilla can improve.  Survey topics will range across a wide variety of ideas nominated by individuals, ranging from measuring the level of happiness and growth of the community to specific ideas like how we can make end user support better.  A number of people have also volunteered to localize the surveys so we are able to hear from as wide a range of participants as possible.  In this report, we will focus on the overall “health” of the Mozilla community.

Objectives of the first survey

•   Understand if the community is satisfied
•   Find ways to improve to better serve the community’s needs
•   Uncover new ideas on how to make the community happier and more engaged

Identifying the target audience
The target audience for this survey included any community member who has participated on discussion boards and who is interested in either the technical or non-technical aspects of the Mozilla project.  We sent the survey to the Mozilla “Active Members” list, a collection of roughly 50 people actively participating in the European Mozilla community.  We also did not limit who could see the survey, opening up the possibility that the survey could be taken by any community member who might have found it on the Web.

Expected conclusions
In order to compare the survey results against our anticipated outcomes, we created a list of expected conclusions. We thought that it might be interesting to compare these expectations to the actual results later in this report to how our process measured up after we were finished.  Here is what we created:

• We will learn about the existing situation in the community
• We will understand how people perceive current relations with Mozilla
• We will learn how the community would like us to improve in terms of these relations

Creating the questions
In designing the survey, we chose to create three “Likert item” questions that allowed us to ask very complex questions, rating several variables on an ordinal scale.  This question format also allowed us to keep our surveys short (a key to a high and accurate response rate in survey design) while still gathering a lot of meaningful data; and they allow for significant complexity in the questions asked and responses gathered.  For a behind-the-scenes peek at the survey making process, see Seth’s post here.  Below are the questions presented on the first survey (asking survey takers to rate each response on a scale of 1 - 5, from strongly disagree to strongly agree).

First question: I feel that …

• … I am satisfied with the amount of people involved with my local community project in the past year
• … there are more active newcomers to our community compared to last year
• … it is easy to become involved at some level (localizing, developing, marketing, etc.) with the Mozilla project
• … I am connected to Mozilla and that Mozilla is interested in our community efforts to work on the project
• … Mozilla is helping to grow local communities
• … the overall health (satisfaction, enthusiasm, leadership, participation) of the community is good

Second question: What resources and actions would be necessary to make the community more satisfied and productive?

• web hosting
• web storage
• more goodies (hardware, software, community-wide tools, marketing supplies, T-shirts, badges, etc.)
• helping to plan community meetings and events
• financially supporting community meetings and events
• helping the local community organize its members
• helping establish a legal entity (e.g. a non-profit organization) if necessary
• having a PR agency to help with press inquiries
• a visit from Mozilla to the community members
• providing a Mozilla website template for community websites in different locales

How many people took the survey and who were they?

As mentioned above, first survey was sent to the Mozilla “Active Members” list.  From there, it propagated through various networks and ended up being responded to by…

…1,156 participants over a two week period. 

Over that period, there was a 15% conversion rate for those who viewed the survey and then took it.  We localized the survey in 17 languages.  Keep in mind that we sent it to the active members list, which has roughly 50 members. 

For further analysis, we selected eight locales, each having at least 70 responses.  We chose these eight locales because we had to find sample sizes large enough where we could draw some statistically significant conclusions.

Some graphs to illustrate the community’s participation

Below is a chart of the number of participants by locale.

Below, you will find a line graph showing the number of surveys taken over time after we publicly launched;

And below here, you can see pie charts showing the percentage of survey takers in each country for languages that are spoken across many different countries.

For each locale presented in the charts above, it is interesting to see which country the survey takers came from.  You can notice that 64% of the surveys taken in English come from people who do not live in anglophone countries. This actually aligns well with the data we have about the Mozilla worldwide community. (See Schrep’s blog post for similar discussion.)

As far as the analysis is concerned, this has two consequences:

1. For the main part of the report, we will not be able to provide statistically significant conclusions for the communities of the U.S. or Great Britain. The sample sizes for these countries were unfortunately too low for the purposes of the analysis.
2. With the survey being about local communities, we will not analyze the surveys coming from all “other” countries and taken in English. We have no way of knowing which local community the survey takers were referring to when submitting their answers. Lesson learned!  In the future, we need to give more options when asking about the country of origin. The “other” answer was not enough.

Not surprising, French and German survey takers mostly came from those countries.  Other francophone or German-speaking countries did accumulate responses (like Switzerland and Austria), but due to the small sample size, we will only be able to analyze these two countries.

The last of the pie charts represents all the countries who took the survey in Spanish language.  Interestingly, only 39% of the surveys taken in Spanish were taken by Europeans, showing us how the survey propagated from our European “Active Members” community (the original recipients of the survey) to other countries in South America.  With some of those countries having really small sample sizes who took the survey, we can only describe their answers in qualitative terms.  You can read more about this in our appendix.

———————–

That concludes the first post on the survey report of “Measuring the Temperature of the Community”.  In the next post, we’ll look at the mean response to each of the questions above.  Please stay tuned for more.  In the meantime, please start to give us some feedback.  We are eager to hear what you think.