A peak at the Community Survey presentation at FOSDEM

On Sunday at FOSDEM, Stas Małolepszy and I did a presentation about the Community Survey Program, displaying the results from our first two surveys and really trying to engage the audience to ask questions and provide feedback.  I thought the presentation went pretty well.  (Please let me know what you thought if you were there.)  Stas is still working on some final analysis and then he and I will write a blog post with a lot of detail about what we learned.

Here’s a taste of the presentation.  Just some tidbits….

Below you will find the mean responses and standard deviations for each question we asked in the first survey.  Remember, in the first survey we asked two “big” questions that asked community members to disagree/agree (on a scale of 1-5) with the statements we made.

You’ll see on the graph the first six answers with an “@” symbol.  Those are the elements of the first question.  The question said “I feel that

1. … I am satisfied with the amount of people involved with my local community project in the past year
2. … there are more active newcomers to our community compared to last year.
3. … it is easy to become involved at some level (localizing, developing, marketing, etc.) with the Mozilla project
4. … I am connected to Mozilla and that Mozilla is interested in our community efforts to work on the project.
5. … Mozilla is helping to grow local communities.
6. … the overall health (satisfaction, enthusiasm, leadership, participation) of the community is good.

The next question read “What resources and actions would be necessary to make the community more satisfied and productive?”

1. web hosting 
2. web storage
3. more goodies (hardware, software, community-wide tools, marketing supplies, T-shirts, badges, etc.)
4. helping to plan community meetings and events
5. financially supporting community meetings and events
6. helping the local community organize its members
7. helping establish a legal entity (e.g. a non-profit organization) if necessary
8. having a PR agency to help with press inquiries
9. a visit from Mozilla to the community members
10. providing a Mozilla website template for community websites in different locales

The summary statistics were encouraging.  All means of the response were above the midpoint of 3.  So, it may not be a stretch to think that the community believes the health is good.  When asked about areas where we could provide support, we saw some interesting things.  For instance, if you look at the last response, you’ll see that providing a “website template” to communities has the highest mean.  Makes sense, doesn’t it?  If we can provide a template for communities, it makes them more official, creates a sense of team, helps with Mozilla branding, and allows new communities to start up quickly.   Our next step?  Simply, we will investigate creating a template for communities to use.  Simple findings like this make the program very valuable.

Data visualization is critical when you collect volumes of information.  Stas used a statistics program called SPSS to generate the graphs.  You’ll see later in our extended write up a lot of great graphs and illustrations of information.

We also were able to perform more than just summary statistics.  One interesting thing we tried is called factor analysis.  When a survey has many response variables, you can run a factor analysis to see if any set of variable can be condensed into a more easily explained, larger variable.  Again, more on this in our detailed report.

The trick with factor analysis is to look closely at how each response variable relates to the others that load onto the new variable.  Then, you have to name the new variable that explains why the responses all loaded where they did.  We were able to “make up” two new variable where all the other responses could load.  One variable, we called “Community Health and Dynamics” and the other “Feeling close to Mozilla”.   Then we started to compare.  Because of the way we designed the survey tool, we were then able to look at each locale to see how they compared after the factor analysis.

Below is picture of the factor analysis of communities who use a Spanish translation of the browser.  You can see Mexico feel a little “less healthy” than the Argentineans and the Spanish.

But, wait!  Don’t jump to any conclusions just yet.  This analysis is directly related to all the response variables we saw in the graph above.  And, you’ll remember that all those answers were above the midpoint response of 3.  So, this is saying that, the Mexicans, on average answered above 3, but were not as hyper-positive and exuberant as, say, the Argentineans.  Like all statistical analysis, it’s really imperative to understand the data before jumping to conclusions.

Lessons learned? 

We need to do a much better job of collecting demographic information.  In the first survey, we had no idea we would have over 1,100 people take it.  We sent it to a list of 200 or so people and it propagated to a point where over 8,000 people saw the survey and 1,100 took it.  But we had no demographics…  It doesn’t render the information useless, but we need that…so we added it in the second survey.  A quick demographic question helped us qualify the responses even better.  Next time, we’ll add a bit more to the demographic question just so we know exactly who is taking the survey.

We also learned that it is important to include a “Don’t Know” option when asking about very specific things like video or online support from Mozilla.

And finally, we learned the old lesson that not everything translates perfectly into all languages.  No real way around this except to try our best.

As I mentioned, Stas and I are working on a final report that will summarize a bunch of this.  If you have questions, please come get me.  I’d be happy to listen to suggestions or provide answers where possible.