Market research has shown that the Mozilla mission is a powerful attractor for Firefox users.  Furthermore, additional research has shown that recommendation is a strong method to promote the adoption of Firefox.

These observations lead to the following question: how does one’s willingness to recommend Firefox relate to their knowledge that Firefox is made by Mozilla, a mission-driven non-profit?  As an initial hypothesis, we posited that one’s willingness to recommend Firefox would be positively related to their knowledge of Firefox as a product of a mission-driven non-profit.

Using the beta survey interface, we asked Firefox 3.6 users the following questions:

1. Did you know that by using Firefox, you are supporting a mission-driven non-profit organization?
2. How likely are you to recommend Firefox to a friend or colleague? (0-10)

The response scale for question 2 was then used to calculate a Net Promoter Score (NPS), which is a marketing metric to gauge users’ willingness to recommend a product or service.  A person who responds at 6 or lower is considered a “detractor,” whereas one who says “9″ or “10″ is considered a promoter.  All else are considered “neutral.”

We calculate an NPS by subtracting the proportion of detractors from the proportion of promoters.  Thus, scores are between 0 and 1 and higher is better. By using this metric, we are able to investigate the relationship between the knowledge that Firefox is made by a mission-driven non-profit and one’s willingness to recommend Firefox to others.

As a survey experiment, we also reversed the presentation of the questions, meaning that for some of the time, we asked respondents to give their willingness to recommend before they indicated whether they knew Firefox was made by a mission-driven non-profit.  We did this in order to determine if simply informing users of this fact was enough to induce a “knowledge” effect.

Figure 1a shows that over every level of response, there are more users who say they did not know that Firefox was produced by a mission-driven non-profit than those who say they did.  In particular, the amount of “neutrals” (which can be interpreted as the 5s, since it is the midpoint) is greater in the “without knowledge” group than in the “with knowledge” group.  These data lend some credibility to the idea that knowing Firefox is made by a mission-driven non-profit relates to willingness to recommend and that more people are unaware of this fact than those who are.

Figure 1b shows the initial results of the question ordering.  By asking users to indicate knowledge first, it appears to reduce the amount of 8s, 9s, and 10s from users.   A potential explanation of this effect could be that users can tell that we are trying to induce positive feelings towards Firefox by asking them this information first.  We can interpret these data to indicate that simply telling users that Firefox is made by a mission-driven non-profit is not enough to boost their willingness to promote.

Figure 2 demonstrates the relationship of willingness to recommend by knowledge group.  This effect is quite pronounced.  The NPS of these groups (“Yes, I did know” versus “No, I didn’t know”) are different from each other, where those with knowledge are much more likely to say that they are willing to recommend Firefox to others.

These results support our initial hypothesis: one’s willingness to recommend Firefox is positively related to one’s knowledge that Firefox is made by a mission-driven non-profit.   Note that this is not a causal relationship; from this data, we cannot say that knowledge directly boosts one’s willingness to recommend Firefox.  No statistical tests of inference have been performed.  However, this survey study strongly indicates that this relationship bears further investigation.

As with many software companies, we are keenly interested in gauging active installations and understanding how people use our products. However, as a non-profit organization with a strong interest in promoting privacy, we also recognize there’s a fine line between this activity and tracking people in unwelcome ways. Mozilla has developed a new mechanism that further enhances privacy, while still meeting its objective to create usage metrics.

1. Measuring Usage Activity

1.1 Our Current Approach: Blocklist Cookies

Mozilla has a mechanism for maintaining installed add-ons called a “blocklist” [https://wiki.mozilla.org/Blocklisting]. This involves a  scheduled request to retrieve an updated blocklist file from Mozilla servers.  The request is currently performed not more than once per 24 hours by several Gecko powered applications maintained by Mozilla (e.g., Firefox Desktop and Mobile). Because the request only happens once per day, we can study the pattern and volume of requests to understand how many active installations of a product there are on a particular day. While requests do not collect or use any personal user information, they are currently using a cookie to study how many unique active installations we see during a given time period.

1.2 From FF4 Onwards: Days Since Last Ping

It’s been our experience that using cookies for tracking unique usage is somewhat unreliable  and cookies raise privacy issues for our users. Cookies can be cleared by the user and sometimes even corrupted via proxies. Because of these reasons, the Mozilla Metrics team filed bug 616835 [1]  to implement a new method for tracking unique installations without any need for a cookie or any other form of identifier. This not only gives us the ability to get better usage metrics, but also to strengthen user privacy by removing the old cookie entirely.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=616835

Design and Implementation of Days Since Last Ping

Each time a request is made for the blocklist data, the request includes a new parameter that indicates how many days it has been since the last request. There is very little possibility to derive a fingerprint from this new parameter since it is a low number of bits, it changes on every request, and users will not maintain outlier values unless they consistently have a pattern of extremely occasional usage (i.e. months between usage of the application). If Firefox is left open unattended for 2 weeks, the days last ping value will be 1 for every day even though the user might never have been at their computer.

Computing Active Installations

For each day in the desired time period, we add all the requests with a value indicating that either this is the first ever request to blocklist (a new installation), or the last time the application made the request was before the time period we are analyzing. This means that on the first day, we count all requests with a valid parameter (i.e. between 1 and max_valid). On the second day of the time period, we count all requests with a parameter between 2 and max_valid. After iterating through each day of the time period using this algorithm, we sum all the counts together and we have the number of unique active installations in that time period.

Example

Consider the date range 04 March – 07 March. We proceed as follows:

1. For March 4th, add the ‘new’ count and the number of days last ping (dlp) ==n, n>=1.
2. For March 5th, add the ‘new’ count and number of dlp==n, n>=2. We ignore dlp==1 because those installations would have made the blocklist check with some of value of dlp on March 4th.
3. Similarly for March 6th and March 7th, add ‘new’ and counts of dlp==n, n>=3 and n>=4 respectively.
4. Add all the counts in (1)-(3) to get number of unique active installations in the above period.

With this metric we can also compute the number of new installations being added on a daily basis. Also, we would like to confirm that we have a high proportion of profiles with days last ping equal to 1. This is not the same as a ‘daily user’ but it is encouraging to see a lot of installations using Firefox for two consecutive days.

Limitations

What we can’t compute the number of installations using Firefox for exactly ‘k’ days in a week or retention patterns.

2. Visualizing the  Behavior of the Days Last Ping Metric

Firefox 4 was released on 22nd March though beta and release candidates were available before that. This is a great opportunity to visualize the dynamics of a metric for a new product.

2.1 New Installations

We are eager to see what proportion of our daily ‘blocklist pings’ come from new installations. Figure 1 displays the proportion of blocklist pings that come from new installations. The heartening observation is the positive slope of the red smoother curve. The peaks are not day of the week effects but correspond to release dates. It is difficult to comment on day of week effects here because of the significant events that occurred in the time period. Nevertheless, new profile percentage appears to around 3.5-4.5% on a daily basis with a slight increasing trend towards the end of the period.

2.2 ‘Daily’ Usage

Figure 2 is a display of the proportion of blocklist updates that come from installations with days last ping equal to 1 which means the installation is active today and the previous day. The proportion varies from 72% to 86%, with a mean hovering around 82%. The red smoother indicates not much change. The 1st,3rd and 5th week have a similar pattern: a low at the beginning of the week (a value of dlp equal to 1 on Monday means that the profile used FF on Sunday), peaking towards the center of the week and dipping as we approach the weekend. The 4th week was the week of the release. Understandably this looks very different. In both Figures 2 and 1, week 2 looks different , probably because of the RC release. I would like to say there is no weekly effect, and indeed the shape is same (except for the two exceptional weeks) but the highs and lows are different.

2.3 ‘Recent’ Usage

Together with counts of days last ping less than 7, we capture more than 90% of the users. Figure 3 shows the proportion of blocklist pings that come from installations that last contacted between 2 to 7 days back. The alignment of troughs and valleys is opposed to the dlp==1 display (Figure 2). There does not seem to be an increasing trend, with the mean around 12-14%, peaking on Thursdays. Why is that? Because the bulk come from dlp=3. On average (across day of week), 95.3 % of the blocklist pings have dlp<=3 and 98.9%<=7. Figure 4 displays the mean cumulative percent of different values of days last ping (between 2 to 7) by the day of week. The key observation is that the curve doesn’t change much – meaning there is little interaction involved here.

2.4 ‘Infrequent’ Usage

Finally, in Figure 5, we get to see the dynamics of the days last ping distribution for a new product. In Figure 5, we plot the density of days last ping greater than 14 for all the days. Each row is 7 days so we can fix a day of week by moving along columns.

Firstly, we see the distribution shifting out to the tails. On one hand this is expected as there is more time available for installations to be used after a long period of inactivity. Also the maximum proportion (see Figure 6) decreases steadily over time from 0.1% to about 0.03 % on 3rd April, dramatically so a few days after FF4 launch. However in both Figures 5 and 6, we see the peak starting to rise again. This means we have more 14 days plus inactive users using FF4. Whether this panel stabilizes is something we can see over the next few months.

2.5 Daily Actives vs. Weekly Actives

Surprisingly, if we look at unique actives over a week, vs the mean daily actives in a week, the numbers are relatively stable, for the 6 weeks the ratios are: 0.692 0.593 0.666 0.994 0.685 and 0.672
In future, we can look at rolling 7 day periods and other window lengths( e.g.  14 days, monthly etc ) and week on week on growth for unique actives.
Reassuring results, and we are all very eager to monitor the progress as FF4′s adoption increases.

Credits:

Thanks to Daniel Einspanjer for the introduction to the cookie usage and the background on the metrics ping.

After a marathon Firefox 4 download ( 15.85 MM, see  http://blog.mozilla.com/blog/2011/03/25/the-first-48-hours-of-mozilla-firefox-4/ ) we prepared a location intensity graph. Figure 1 displays the cities by frequency of download (independent of how many downloads though the two are strongly related). Blue is infrequent to very few (bottom 10%), bright yellow is many (top 10%).

Figure 1. Location intensity of cities. Blue is rare, bright yellow is high.

Though the above picture is heartwarming at the same time it is disappointing. There are swaths of the world pitch black. The location information is constrained by the accuracy of the geo-location service but that is also related to internet penetration. With Internet access rapidly becoming (in the not too distant future)  a public utility, it is sad that the ‘have not’s have one more item on the list.

Based on reader input I’ve attached a large (2248 × 1268, 1.1MB) PNG file.Click here to download.

In our last post, we found that the number of installed extensions was a good discriminant of heavier users. In this short follow-up, we’ll delve into the survey data associated with the Beta Interface study.  Here is a snapshot of some of the research we’ve been conducting.

Users overshoot their estimated browsing time

The graph above demonstrates that users tend to simply overestimate how long they use Firefox. Those that typically use the browser less have a more accurate assessment of how long they are browsing. But for users who state a longer browsing time per day, the actual browser usage is lower than their own estimate.

First, a note about the methodology behind this graphic. We estimate the average daily browsing time by aggregating the session lengths of Test Pilot users over the course of the study. Previously we have defined a browser session as a continuous period of user activity in the browser, where successive events are separated by no more than 30 minutes. We subset on the users that state they only use Firefox, to avoid the problem of a different primary browser. 

We thought of a few possible explanations as to why, for heavier users, the estimated time is lower than the stated time. Those users might, for instance, be online and using their computers quite a bit during the day, but have integrated their online workflow with their offline ones. Software engineers are a good example of this – we might expect a programmer to be working on a computer all day, leaving the browser open, and using it every once in a while.  So there may be the perception of constant browser usage.  This certainly rings true from the experience of the Metrics team – we’re on our computers almost all day, with Firefox open, despite working.  This is, however, only speculative at this point, since we don’t have data about when users are on their computers.

There are still some obvious methodological issues with this approach: a user might, for instance, use Firefox on a work computer (with test pilot installed), and a different one for home use, which could account for the difference. As such, we hope to include a survey question asking “How much time a day do you spend on this computer?” in the next version of the study.  At that point, we can update this research.