Blog of Metrics

Summary: In a survey of 966 visitors to the Update page, 47.8% open new empty tabs from the new tab button, 30.4% from the file menu, and 21.8% from a keyboard shortcut.

In addition to running A/B and multivariate tests, we have begun supplementing our quantitative analysis with more qualitative feedback. Historically, many of our most impactful insights have come from survey data.

With this in mind, we recently launched a usability survey on the Firefox Update page page. Our goal? Understand how users open new empty tabs.

The Update page is an ideal place to run this survey because nearly every Firefox user views the page once. Additionally, because this page receives such high traffic, we are able to expose the survey to ~1% of users and get statistically significant results within a week.

Of the 6000 visitors who saw the survey, 966 responded–resulting in response rate of 16.1%. The data breaks down as follows:

Our early results are encouraging, but we can make a number of improvements. First, we may reduce our response bias by randomizing the order in which the responses are presented. Second, we will have more representative data if we run the survey immediately after a dot release. This will be a factor if the population of users who update Firefox immediately and the population of users who update later are heterogeneous (a likely assumption). Third, if we collect a larger sample, we can segment our analysis by operating system. We can run this analysis now (using SiteSpect’s excellent web optimization tool), but our results won’t be statistically significant. And finally, we can elicit a higher response rate by testing a few different survey styles.

Update page surveys provide one additional benefit. By comparing our survey results with our Test Pilot data, we can roughly approximate the bias in the Test Pilot population. Both data sets are biased, but our survey data should be more representative because it has a lower barrier to entry and a much larger sample.

Next up? We want to determine what’s driving the unusual traffic on the First Run page. Ken already has a survey in the works.

Have a survey question that would help us build a better Firefox experience? Please leave it in comments!

Now that we’ve reviewed our growth in 2009, you may wonder which countries added the most Firefox users. Are a handful of countries with high Internet populations responsible for Firefox’s growth? Or does our growth primarily come from countries with few Internet users?

To answer this question, I looked at the cumulative distribution of Firefox use. The x-axis represents the cumulative number of countries, ordered by decreasing active daily users. The y-axis represents share of total Firefox users.

By plotting the distribution in 2007 against the distributions in 2008 and 2009, we can clearly see whether high or low Internet population countries are increasing their share of users and are thus growing more quickly.

The shift over the last two years has been dramatic. In 2007, five countries accounted for over 55% of Firefox usage. Today, that figure is below 48%. Countries with relatively few Internet user are responsible for an increasing share of Firefox’s growth.

This flattening distribution curve, no doubt, is largely due to increasing Internet penetration in developing countries. (And it should also be noted that many of these users wouldn’t have been reached without the fantastic contributions of our localization community!)

2009 wraps up another year of terrific Firefox growth! Roughly keeping pace with previous years, Firefox grew 40% worldwide. Two regions in particular continued adopting Firefox at a breakneck pace — South America (64%) and Asia (73%).

Most of the 40% growth occurred recently. In the 4 months leading up to the holiday season, Firefox added 22.8 million active daily users. During that same period last year, Firefox added 16.4 million users.

The chart below plots Firefox active daily users over the last 17 months. The top smoothed line shows our recent growth pattern, while the lower smoothed line shows growth in the previous year. The dotted blue line projects active daily users, based upon historical trending.

In case you’re curious, the seasonal holiday dip (above) falls in line with overall Internet usage patterns, and it was actually slightly milder than we predicted. Historically, we have seen a 15% decline in Firefox usage around Christmas.

All in all, 2009 was a great year for Firefox. The entire community deserves a ton of credit for the continued rapid pace of Firefox adoption.

We know that existing Firefox users update their browser remarkably quickly. However, it seems the same cannot be said of new users–those people who are installing Firefox, perhaps for the first time.

In the recent two week period since the release of Firefox 3.5.6, only 45% of new en-US users installed Firefox 3.5.6 (or a beta version of 3.6). What about the other 55%? 24% of new users installed an old 3.5 version, while 24% and 6% of installed Firefox 3.0 and 2.0, respectively.

A more detailed breakdown of installations provides few hints as to why so few new users installed our latest release. Lets start by looking at Firefox 3.5. Below, I have plotted the percentage of total First Run page visits for each page version.

As expected, the most recent versions, 3.5.6 and 3.5.5, are the most popular. Beyond this, the data doesn’t tell a clear story. Firefox 3.5.3 and 3.5.2 are both more popular than 3.5.4. The data for Firefox 2 and Firefox 3 is even more confounding! This results holds on non-English locales as well; 38% of German users, 53% of French users, and 62% of Russian users installed old versions of Firefox!

Now that we’ve identified the problem, it’s time to come up with a few hypotheses and see if any insights can help us improve the user’s experience. Perhaps users are intentionally downgrading to old versions of Firefox. A more likely scenario is that new users are mistakenly pointed to old download files. Somewhere in the ballpark of 60% of Firefox downloads occur at a Mozilla website (mozilla.com, mozilla-europe.org, etc.). Outside distribution channels (e.g., download.com, community sites, international sites, etc.) may take longer to update the files they are distributing.

Have a idea of your own? Please leave it in the comments!

In the short them, I suggest we update all versions of the First Run page. As a longer term solution, we may want to implement a lightweight installer that always downloads the most recent Firefox version.

Mozilla.com occasionally breaks into the top 50 websites worldwide. This single domain, however, fails to capture Mozilla’s full web presence. Below, I have plotted two metrics, pages views (x-axis) and page views per visit (dot size), across our most popular domains.

Mozilla.com appears on top, accounting for nearly half of our total page views. This picture, however, is deceptive. Three product pages–the Update page, the First Run page, and the Getting Started page–account for 80% of Mozilla.com’s traffic.

To more clearly illustrate the relative popularity of our websites, I plotted these three sites separately from their domain. After doing so, a surprising picture emerges!

Addons.mozilla.org becomes our most popular website, garnering over three times the page views of Mozilla.com. Mozilla-europe.org, similarly, rises in relative popularity, receiving 1/3 the page views of Mozilla.com. A few additional highlights:

* Together, Mozilla’s websites receive 700 million page views per month
* MDC, Spreadfirefox.com, and the Creative Collective are our most engaging pages

Next, I’ll examine our most highly trafficked pages and suggest areas to focus our optimizations efforts. Know a Mozilla web page you’d like to change? Leave your suggestions in the comments.

This fall, we implemented a website optimization solution and our early results are outstanding!

Our first two A/B tests have seen double-digit engagement improvements — 33% on the First Run page and over 50% on the What’s New page (more analysis later this week). Thanks to Sean Martell and the Labs team for the creative changes you see above.

In the past, we have occasionally conducted website experiments and discussed our goals and our resulting insights. Look for the pacing to change. Moving forward, website optimization will be a continuous, long term activity within the Mozilla community. For example, we will be running at least two website experiments at all times – beginning with our top touch points. The Firefox Update page (“What’s New”), the Getting Started page, as well as the Support and Add-ons hompages are just a few of our top priorities.

If you have any experiment ideas, please leave them in the comments. Or, better yet, fill out this form, which we review weekly.

Our Metrics intern, Eric Hergenrader, did some terrific work analyzing fluctuations in Firefox usage.  Using a linear regression, he found that this summer’s decline in Firefox use was largely in line with expectations; a 2-3% decline in July and August is typical.  In addition to seasonality, Eric found that three factors driving usage drops are weekends (-18%), holidays (-30%), and Firefox releases (-13.5%).

Two months later, Eric’s model does a remarkable job predicting future usage.  Below, I have plotted actual against predicted usage.  The correlation between these two lines is remarkable.  Since September, the average prediction was off by just 1.63%!

To better understand daily growth variations, I created a heat map that visualizes month over month usage growth. Each cell contains one day’s growth rate over the same day of the week, 28 days earlier.

Some highlights:

• Strongest growth occurs during Spring and Fall
• Periods of growth and decline are clustered
• Growth is outperforming predictions by 2.58%

We must be careful not to overreach with our conclusions.  We are working with a limited range of data and have likely missed significant confounding variables.  That said–thanks to Eric’s work–we’re off to a great start.  Please leave your thoughts and any suggestions on how to improve our projections in the comments.

In part I of this two-part series (found here), I discussed the first level of my analysis of our support site. Next, we asked ourselves two questions –

• Which search terms lead to which articles?
• Are visitors ultimately satisfied with the articles they choose to view?

Let’s take a look at the data:

Highlighted in green and red are paths our support site seems to do particularly well and poorly with, respectively. A positive take-away from just glancing over the data is that in four cases (under searches private browsing, clear cache, cache, and export bookmarks) we  have top articles performing really well to visitors’ standards.

Focusing on some of our poorer marks, we see the cookies, enable cookies, bookmarks, and clear history searches are not directing users to the content they want. With help from David Tenser, we were able to come up with some reasoning behind these numbers.

Starting with the clear history search to the “Clear Recent History” article, there seems to be a solid explanation for the poor grade. A continuing problem we have had (documented concretely here) with the Awesome Bar is that users do not realize their bookmarks show up while typing in an address. When this is the case, users that believe they are clearing their history will still see any undesirable bookmarks in the Awesome Bar and hence think the “Clear Recent History” article has not helped their problem. Users need to change the content their Awesome Bar remembers in the Privacy section of Firefox preferences to accomplish this task.

Visitors hitting the “Lost Bookmarks” page from the  bookmarks keyword have given the article a low approval rating. An issue with the “Lost Bookmarks” page is that it does not link to an article explaining how to import bookmarks. Users who have upgraded their browser or have just come from another browser may not know how to import their bookmarks, and they navigate to “Lost Bookmarks” in order to find a solution.

We are looking deeper into users’ experiences with two other articles highlighted above in red — “Websites say cookies are blocked” and “Cannot log into websites” — to see if any content is missing and needs to be added. The problems are straightforward enough — visitors making their way to these pages are probably not clicking on them by accident or confused about their problem. The SUMO team is currently implementing changes across the site to increase user satisfaction in areas like those discussed above. This should provide for a much smoother help site experience across the board for our users.