The newsletter survey, part one: who took it?
June 27th, 2008 by seth bindernagel
Today, we are starting our next series of posts in which we will present the results and our findings from the 3rd community survey. In May, we worked together with Deb on creating a survey about the about:mozilla newsletter. After 6 months of publishing the newsletter, Deb decided it was a good moment to survey the readers about what they thought about the idea. Over the first six months, Deb sent out 21 issues of the newsletter, so indeed it was sufficient for the readers to get familiar with the style and form of the newsletter over this period.
In this post, we will focus who took the survey and what languages they spoke. We will also take a rather unusual look into the way in which the respondents accessed the survey, i.e. we will look at the referrer information recorded in our logs and try to extend our analysis with this data.
Background info
The newsletter has been published each week for the past 6 months. It’s written in English and distributed through various channels, such as plain text e-mail, HTML e-mail, RSS feeds and Mozilla DevNews site among others.
When we started drafting the newsletter survey, we asked Deb to create objectives for the survey, as well as the example questions. The three objectives for this survey were:
1. Figure out who is reading the newsletter and how (what channels they use)
2. Understand the value of the newsletter
3. Learn how to make it better
For the first objective, we used following four questions.
1. How do you read the about:mozilla newsletter?
2 What other sources of news about the project do they use?
3. How did you first hear about the about:mozilla newsletter?
4. Demographic question (”I am…”)
After some thought, we determined that the second and third objectives were closely related. We wanted to find a way to merge these into one smart question. That new smart question would encompass the value that the newsletter provided (objective 2) and the suggestions for improvement (objective 3).
We used following statements for this question:
The newsletter:
…repeats too much content that I’ve already seen in other places.
… topics are a good summary of activity on planet.mozilla.org
…content is generally useful and interesting to me every week.
…should contain more news about Mozilla projects other than Firefox.
…should contain more news about third-party projects using Mozilla technology.
…should focus more on technical and software development-related topics.
…should contain more general project news, such as governance and organizational issues.
…is published frequently enough.
…should be localized.
…is the appropriate length
…should contain more community content and human-interest stories.
To learn more about the topic the newsletter’s readers find interesting, we featured the following open question:
“I would like to see the newsletter contain more content about… [ specify ]. “
We created this question because we figured that there was no way to predict all possible choices and present them in a list of a finite length.
The survey
The survey was open for 14 days during which a total of 317 responses were submitted. We distributed it by the newsletter itself (it spanned three issues), as well as through posts on Deb’s blog, and our blog. If you look at the responses submitted over time, you will certainly notice the bumps due to the mention in the consecutive newsletter issues.
Among 317 respondents, a vast majority were English speakers. On the chart below we can see which language versions of browsers the respondent were using.
246 out of 317 used an English version of their browser. Because the newsletter is published in English only, such a result was to be expected. This fact, however, has also a second meaning. It shows us that only a small fraction of respondents were using different locales, which most likely suggests that the newsletter as a whole is mostly popular among English speakers.
Here are some demographics:
It turns out that 87% of the respondents described themselves as the end-users of Mozilla products. More over, 73% (201 respondents) of them didn’t select any other answer related to contributing in the project. If we group answers in two categories: “observes” and “contributes”, we’ll see that only 31% of the respondents contribute to the Mozilla project.
See how the percentages don’t add to 100%? It’s because respondents were able to select multiple answers in this question, so that some of them landed in both categories. And how many didn’t? 31 percent in the “contributes” category means that, in general, 69% of the respondents described themselves as observers not actively participating in the project. This means that the newsletter has a wide audience of end-users and observers. It will be interesting to see in our next posts what this group thinks about the newsletter, as well as if they use it as their only source of information about happenings in Mozilla.
In over 75% of cases, we were also able to detect where the respondents came from to take the survey (where they read about it). It turned out that almost one third of the respondent came to take the survey from the MDC DevNews site (blog-like site where the newsletter is published). 13% read about the survey in their RSS readers. The referral data is presented in the chart below:
This was just an introduction to the series about the about:mozilla newsletter survey. In our next posts, we will focus on the responses to the main questions of the survey.