Blog of Metrics

Summary: (Total Crashes)/(Active Daily Users) is a low resolution metric for crash trending. We can improve Firefox’s stability for more users if we understand the distribution of crashes.

Over the last few months, improving Firefox’s stability has become a top priority. And our early results are encouraging! Both our survey data and crash reporter data show a downward trend in crashes. We should be careful, however, not to read too much into this data.

Due to privacy concerns, we do not store user ids when collecting crash data. Accordingly, our crash data suffers from a number of limitations. For example, we can only determine the number of crashes per daily session, not per session length. As a result, the number of crashes per user will appear to rise if users browse more often per day. Changes to the crash reporter UI will similarly bias our data. A 5% increase in the reporter response rate will lead to a 5% increase in reported crashes.

Perhaps most importantly, the lack of a user id limits our ability to draw inferences about the distribution of our data. Why does this matter? If Firefox crashes are skewed, we may reduce overall crashes by 10%, but have 80% of users experience more crashes.

A quick look at the “Week in the Life of a Browser” study suggests that crashes are indeed highly skewed. By examining start-up events without corresponding shut-down events, Jono calculated the number of unexplained session interruptions per user. While there are other causes of session interruptions, such as a computer losing its power, we can reasonably assume that session interruptions serve as a (perhaps highly overstated) proxy for crashes.

Session interruptions per user does not take on the bell-curve shape of normally distributed data. Rather, it follows a power law distribution. 49% of users did not experience a single session interruption, while 70% experienced one or fewer. The mean number of session interruptions, however, was 1.4. If our crash data follows a similar distribution, the average crash per user metric tells us little about the experience of a typical Firefox user.

Anecdotal evidence supports this hypothesis. While we all know people who swear by Firefox’s stability, we also know people who complain of frequent failures. I, for one, haven’t experienced any crashes since upgrading to the 3.6 beta a few weeks ago.

With this in mind, I suggest we use Test Pilot to run a longitudinal study of true Firefox crashes. Because Test Pilot is opt-in and allows users to review their data before submitting it, we’re able to consider data at a more granular level. As with previous TP experiments, we will go to great lengths to respect the privacy of participants.

In addition to crash events and session length, I would like to collect data we can correlate with crashes. Firefox version, operating system, and Add-ons installed immediately come to mind.

Have suggestions for additional data that we should, or shouldn’t, collect? Please leave them in the comments.

Tuesday’s earthquake in Haiti devastated hundreds of thousands of lives and the aftershocks are felt physically and emotionally throughout the world.

The Mozilla community includes thousands of people in Haiti, and in our metrics, we can see the hole this disaster has left in our community.  The chart below shows Firefox usage in Haiti this week (it’s based on a once daily “ping” that we see from active users) and it’s broken down by hour (local time in Haiti).  This picture is not pretty and it suggests that much of the communications infrastructure within Haiti has been adversely affected.

Everyone in Haiti could use our thoughts and help, and we’d like to put out a personal plea to the Mozilla family to consider these members of our community and donate if you can.

(Thanks to Daniel Einspanjer for contributing to this post.)

Summary: In a survey of 966 visitors to the Update page, 47.8% open new empty tabs from the new tab button, 30.4% from the file menu, and 21.8% from a keyboard shortcut.

In addition to running A/B and multivariate tests, we have begun supplementing our quantitative analysis with more qualitative feedback. Historically, many of our most impactful insights have come from survey data.

With this in mind, we recently launched a usability survey on the Firefox Update page page. Our goal? Understand how users open new empty tabs.

The Update page is an ideal place to run this survey because nearly every Firefox user views the page once. Additionally, because this page receives such high traffic, we are able to expose the survey to ~1% of users and get statistically significant results within a week.

Of the 6000 visitors who saw the survey, 966 responded–resulting in response rate of 16.1%. The data breaks down as follows:

Our early results are encouraging, but we can make a number of improvements. First, we may reduce our response bias by randomizing the order in which the responses are presented. Second, we will have more representative data if we run the survey immediately after a dot release. This will be a factor if the population of users who update Firefox immediately and the population of users who update later are heterogeneous (a likely assumption). Third, if we collect a larger sample, we can segment our analysis by operating system. We can run this analysis now (using SiteSpect’s excellent web optimization tool), but our results won’t be statistically significant. And finally, we can elicit a higher response rate by testing a few different survey styles.

Update page surveys provide one additional benefit. By comparing our survey results with our Test Pilot data, we can roughly approximate the bias in the Test Pilot population. Both data sets are biased, but our survey data should be more representative because it has a lower barrier to entry and a much larger sample.

Next up? We want to determine what’s driving the unusual traffic on the First Run page. Ken already has a survey in the works.

Have a survey question that would help us build a better Firefox experience? Please leave it in comments!

Now that we’ve reviewed our growth in 2009, you may wonder which countries added the most Firefox users. Are a handful of countries with high Internet populations responsible for Firefox’s growth? Or does our growth primarily come from countries with few Internet users?

To answer this question, I looked at the cumulative distribution of Firefox use. The x-axis represents the cumulative number of countries, ordered by decreasing active daily users. The y-axis represents share of total Firefox users.

By plotting the distribution in 2007 against the distributions in 2008 and 2009, we can clearly see whether high or low Internet population countries are increasing their share of users and are thus growing more quickly.

The shift over the last two years has been dramatic. In 2007, five countries accounted for over 55% of Firefox usage. Today, that figure is below 48%. Countries with relatively few Internet user are responsible for an increasing share of Firefox’s growth.

This flattening distribution curve, no doubt, is largely due to increasing Internet penetration in developing countries. (And it should also be noted that many of these users wouldn’t have been reached without the fantastic contributions of our localization community!)

2009 wraps up another year of terrific Firefox growth! Roughly keeping pace with previous years, Firefox grew 40% worldwide. Two regions in particular continued adopting Firefox at a breakneck pace — South America (64%) and Asia (73%).

Most of the 40% growth occurred recently. In the 4 months leading up to the holiday season, Firefox added 22.8 million active daily users. During that same period last year, Firefox added 16.4 million users.

The chart below plots Firefox active daily users over the last 17 months. The top smoothed line shows our recent growth pattern, while the lower smoothed line shows growth in the previous year. The dotted blue line projects active daily users, based upon historical trending.

In case you’re curious, the seasonal holiday dip (above) falls in line with overall Internet usage patterns, and it was actually slightly milder than we predicted. Historically, we have seen a 15% decline in Firefox usage around Christmas.

All in all, 2009 was a great year for Firefox. The entire community deserves a ton of credit for the continued rapid pace of Firefox adoption.

We know that existing Firefox users update their browser remarkably quickly. However, it seems the same cannot be said of new users–those people who are installing Firefox, perhaps for the first time.

In the recent two week period since the release of Firefox 3.5.6, only 45% of new en-US users installed Firefox 3.5.6 (or a beta version of 3.6). What about the other 55%? 24% of new users installed an old 3.5 version, while 24% and 6% of installed Firefox 3.0 and 2.0, respectively.

A more detailed breakdown of installations provides few hints as to why so few new users installed our latest release. Lets start by looking at Firefox 3.5. Below, I have plotted the percentage of total First Run page visits for each page version.

As expected, the most recent versions, 3.5.6 and 3.5.5, are the most popular. Beyond this, the data doesn’t tell a clear story. Firefox 3.5.3 and 3.5.2 are both more popular than 3.5.4. The data for Firefox 2 and Firefox 3 is even more confounding! This results holds on non-English locales as well; 38% of German users, 53% of French users, and 62% of Russian users installed old versions of Firefox!

Now that we’ve identified the problem, it’s time to come up with a few hypotheses and see if any insights can help us improve the user’s experience. Perhaps users are intentionally downgrading to old versions of Firefox. A more likely scenario is that new users are mistakenly pointed to old download files. Somewhere in the ballpark of 60% of Firefox downloads occur at a Mozilla website (mozilla.com, mozilla-europe.org, etc.). Outside distribution channels (e.g., download.com, community sites, international sites, etc.) may take longer to update the files they are distributing.

Have a idea of your own? Please leave it in the comments!

In the short them, I suggest we update all versions of the First Run page. As a longer term solution, we may want to implement a lightweight installer that always downloads the most recent Firefox version.