Community Survey Blog

At the end of 2007, Stas Małolepszy and I closed Mozilla’s first community survey and began the experimentation of analyzing the survey responses and drawing some conclusions on what we had collected.  We’ve finally reached a pretty comfortable point and are ready to publish some results.  Keep in mind that we already presented much of this raw data at FOSDEM in February for the community to see.  While there, we received a lot of great questions and ideas about what to present.  We took that time to conduct to refine what we learned (and conduct one survey and start creating a third!).  We’ll present our findings over a series of three to four blog posts over the next few days.  Please stay tuned and ask as many questions as you like.

Measuring the Temperature of the Mozilla Community

Introduction
Mozilla created the Community Survey Program to learn more about the ideas and opinions of the people who make up our community.  We are using this program to ask various questions to learn how Mozilla can improve.  Survey topics will range across a wide variety of ideas nominated by individuals, ranging from measuring the level of happiness and growth of the community to specific ideas like how we can make end user support better.  A number of people have also volunteered to localize the surveys so we are able to hear from as wide a range of participants as possible.  In this report, we will focus on the overall “health” of the Mozilla community.

Objectives of the first survey

•   Understand if the community is satisfied
•   Find ways to improve to better serve the community’s needs
•   Uncover new ideas on how to make the community happier and more engaged

Identifying the target audience
The target audience for this survey included any community member who has participated on discussion boards and who is interested in either the technical or non-technical aspects of the Mozilla project.  We sent the survey to the Mozilla “Active Members” list, a collection of roughly 50 people actively participating in the European Mozilla community.  We also did not limit who could see the survey, opening up the possibility that the survey could be taken by any community member who might have found it on the Web.

Expected conclusions
In order to compare the survey results against our anticipated outcomes, we created a list of expected conclusions. We thought that it might be interesting to compare these expectations to the actual results later in this report to how our process measured up after we were finished.  Here is what we created:

• We will learn about the existing situation in the community
• We will understand how people perceive current relations with Mozilla
• We will learn how the community would like us to improve in terms of these relations

Creating the questions
In designing the survey, we chose to create three “Likert item” questions that allowed us to ask very complex questions, rating several variables on an ordinal scale.  This question format also allowed us to keep our surveys short (a key to a high and accurate response rate in survey design) while still gathering a lot of meaningful data; and they allow for significant complexity in the questions asked and responses gathered.  For a behind-the-scenes peek at the survey making process, see Seth’s post here.  Below are the questions presented on the first survey (asking survey takers to rate each response on a scale of 1 - 5, from strongly disagree to strongly agree).

First question: I feel that …

• … I am satisfied with the amount of people involved with my local community project in the past year
• … there are more active newcomers to our community compared to last year
• … it is easy to become involved at some level (localizing, developing, marketing, etc.) with the Mozilla project
• … I am connected to Mozilla and that Mozilla is interested in our community efforts to work on the project
• … Mozilla is helping to grow local communities
• … the overall health (satisfaction, enthusiasm, leadership, participation) of the community is good

Second question: What resources and actions would be necessary to make the community more satisfied and productive?

• web hosting
• web storage
• more goodies (hardware, software, community-wide tools, marketing supplies, T-shirts, badges, etc.)
• helping to plan community meetings and events
• financially supporting community meetings and events
• helping the local community organize its members
• helping establish a legal entity (e.g. a non-profit organization) if necessary
• having a PR agency to help with press inquiries
• a visit from Mozilla to the community members
• providing a Mozilla website template for community websites in different locales

How many people took the survey and who were they?

As mentioned above, first survey was sent to the Mozilla “Active Members” list.  From there, it propagated through various networks and ended up being responded to by…

…1,156 participants over a two week period. 

Over that period, there was a 15% conversion rate for those who viewed the survey and then took it.  We localized the survey in 17 languages.  Keep in mind that we sent it to the active members list, which has roughly 50 members. 

For further analysis, we selected eight locales, each having at least 70 responses.  We chose these eight locales because we had to find sample sizes large enough where we could draw some statistically significant conclusions.

Some graphs to illustrate the community’s participation

Below is a chart of the number of participants by locale.

Below, you will find a line graph showing the number of surveys taken over time after we publicly launched;

And below here, you can see pie charts showing the percentage of survey takers in each country for languages that are spoken across many different countries.

For each locale presented in the charts above, it is interesting to see which country the survey takers came from.  You can notice that 64% of the surveys taken in English come from people who do not live in anglophone countries. This actually aligns well with the data we have about the Mozilla worldwide community. (See Schrep’s blog post for similar discussion.)

As far as the analysis is concerned, this has two consequences:

1. For the main part of the report, we will not be able to provide statistically significant conclusions for the communities of the U.S. or Great Britain. The sample sizes for these countries were unfortunately too low for the purposes of the analysis.
2. With the survey being about local communities, we will not analyze the surveys coming from all “other” countries and taken in English. We have no way of knowing which local community the survey takers were referring to when submitting their answers. Lesson learned!  In the future, we need to give more options when asking about the country of origin. The “other” answer was not enough.

Not surprising, French and German survey takers mostly came from those countries.  Other francophone or German-speaking countries did accumulate responses (like Switzerland and Austria), but due to the small sample size, we will only be able to analyze these two countries.

The last of the pie charts represents all the countries who took the survey in Spanish language.  Interestingly, only 39% of the surveys taken in Spanish were taken by Europeans, showing us how the survey propagated from our European “Active Members” community (the original recipients of the survey) to other countries in South America.  With some of those countries having really small sample sizes who took the survey, we can only describe their answers in qualitative terms.  You can read more about this in our appendix.

———————–

That concludes the first post on the survey report of “Measuring the Temperature of the Community”.  In the next post, we’ll look at the mean response to each of the questions above.  Please stay tuned for more.  In the meantime, please start to give us some feedback.  We are eager to hear what you think.

Trackback URI | Comments RSS