Behind the Scenes of the Community Survey Program

In November of 2007, Stas, Pascal and I formally launched the Community Survey Program.  I blogged about it back then, describing just what we were hoping to do with the program.  Today, I’d like to open up a bit on how a team of us has worked together to make a meaningful survey for the Mozilla community.

In the last survey, we focused on product support by Mozilla.   Stas suggested the topic and we reached out to David Tenser to see if he was interested in helping us.  Not surprising at all, David responded with an enthusiastic, “Let’s do this!”

So….on collaborating!!!

Each of us brought a specialty to the group.  I have had some experience with survey design, having taken a few classes on survey methodology and launching a few with other organizations.  Stas had much of the technical expertise, having developed the PHP online tool.   And, like most software engineers, he brought some critical thinking with an excellent ability to tackle what we hoped to accomplish.  David brought his own Scandinavian intensity, a deep knowledge of the support issues, and a good understanding of how to construct surveys.  The three of us made a great team, bouncing ideas off each other, challenging our assumptions, and working out particulars like question phrasing and focus areas.

With each of us living in different countries, our geographic locations could have made our working group a bit challenging:  David is in Sweden, Stas in Poland, and I am in California.  But, we all tried hard to be open and flexible for meetings and working together, whether it was getting up early or staying up late.  It may sound like a no-brainer, but it was a simple step toward success.  Also, we used VoIP to chat for free.  Before the end of the survey design, I felt like Stas and David were right across the office instead of the ocean.

Our next big task as a group was to agree on the goals of the survey.   We agreed on the following:

1. To understand and measure how the Mozilla community is doing at support in the different locales.
2. To get a better understanding of people’s perception of support provided by Mozilla.
3. To find out how can we make support by Mozilla more useful.

As we began writing the questions, we would occasionally come to a stopping point.  If so, we always asked, “Does this issue relate to one of our goals for this survey?”  If the impasse had nothing to do with the above three goals, we dropped it immediately…no further discussion.  Focus was key, especially when collaborating across continents over the Internet.

As far as the content creation, we relied heavily on David to draft the questions.  He is probably most familiar with the issues in the support community.  We had the objectives and decided to create a question for each goal, hoping that we could get good insight into each of the three goals we articulated from the beginning.  It turned out to be a pretty clean and productive process.  Check out the survey and take it if you’d like.

Here’s an example to illustrate the process.  Take the first goal:

1. To understand and measure how the Mozilla community is doing at support in the different locales.

We created the question:

“How would you rate the support that community provides for Mozilla products in your language?”

This question was particularly tricky to ask because we knew that each locale has its own support community.  How would we ask appropriately across all the locales that provide support without leaving out something?  David listed what he thought were the universal elements that existed in support across locales.  Knowing that we were going to post questions in 15+ languages without knowing who exactly might take the survey, we were careful to use language that was easily understood, could be translated into 15+ languages, and touched on exactly what we had hoped to learn from the process.

For the actual survey design, we chose to create three “likert scale” questions to gather detailed information about Mozilla support, with one short demographic question at the end of the survey to help qualify who was telling us what.

(Skip this next part if you don’t want to read a bit of background about our design.)

I’ll take a moment to specifically describe the survey design methodology. The purpose of a likert scale question is multi-fold.  For one, it allowed us to ask a very complex question, rating several variables on an ordinal scale.  With this technique, we could keep our surveys short (a key to a high and accurate response rate in survey design) while still gathering a lot of meaningful data that can be analyzed to draw conclusions.  The format appears as one question, but it is really several in one.  Typically, the likert-style survey contains a set of possible response variables to a question and asks the survey taker to rate each response on some scale.  (“1″ usually being something like “Completely Disagree”, and “5″ something like “Strongly Agree”)  With the data gathered, we could then test each variable against the others.  In our analysis, we’ll do simple cross-tabs and pivot tables that will show us information like “those who said X also said Y”.  And, with many dependent response variables in one question, we can run statistical regressions on the data to find significance and eliminate those response variables that were not significant.  I hope this explains well why we used the likert-style question format.

We also threw in one demographic question so we knew “who” was taking the survey.  We weren’t sure exactly who would see it online since it was open to anyone.  So, we asked each person to tell us just a bit about his or her involvement in the project.  We gave people the ability to tell us if they were an “end-user” or more deeply involved like someone who writes code for the Mozilla project.

After three or four meetings among David, Stas and I, we had our survey ready for translation.  Stas placed the survey on a staging server, invited localizers to translate, and then check-in their translations to the code repository he had created for the survey.  After all translations were complete, we launched.  The first survey (about the Mozilla community) was viewed by over 8,000 people and taken by over 1,000.  The second survey (about Mozilla support) has had over 1,400 people take it.  More than enough responses to have a good sample to test.

Right now, we are still analyzing the data from our first two surveys and hope to present it online soon.  Additionally, David, Stas and I are hoping to present this material at FOSDEM, speaking more about this process and discovering new topics we can survey.

Finally, I have to say that it was a real pleasure working with David and Stas.  With each of us having our own working style, we came together across time zones, languages, and continents not only to create a great survey, but also to have a lot of fun doing it.  For instance, if we had a debate in a word choice that we might use for a particular question, I’d try to illustrate my point with an example, sometimes pitting Poland vs. Sweden in a hockey match, or something like that.  (e.g. …on using the word superfluous:  “The fourth goal by Poland was superfluous in the team’s victory over Sweden.”)  By the end of the project, I am sure both Stas and David were ready to start slamming some of my favorite sports teams. Sorry guys!

Overall, a great project to work on.  Hopefully we can do more, so if you have an idea for a survey, please do let us know.

And, let me know if this was at all a helpful post.  Sometimes it’s hard to tell.  Hope you’re still reading.