Blog of Metrics

Thanks to recent editions of Funnelcake, along with two editions from 2007, we can now start to think about our ability to retain Firefox users over time.  In other words, if a user downloads and installs Firefox today, what are the odds of that user firing up their Firefox browser one month from today? six months from today? and has our retention ability and the experience of Firefox users changed over time?

Answering such questions can better inform our decision making – decision making across the Mozilla Project, from product decisions, to UI and UX decisions, to community specific decisions.

With this introduction aside, let’s dive into the analysis and insights.  First, let’s look at our most recent Funnelcake cohorts, i.e., those users who downloaded Firefox on Monday, December 15th and on Tuesday, January 13th.   The chart below shows that of those who successfully installed Firefox, the percentage who actively used it on a particular day since then:

The rate at which these two groups of users continued using Firefox on a day-to-day basis is fairly similar.  The group who installed Firefox on December 15th was affected by the holiday season, so we cheated a bit by ignoring about ten days around that time period to avoid having seasonality take us away from the broader story here.  You’ll notice that the peaks and valleys are slightly off, which is due to the day of week factors being off by one day.

As we’ve noted previously, Firefox is very, very sticky.  The fact that so many new users continue using it each and every day is an amazing testament to the strength and health of our communities.

Taking one other look at this data, we can make a comparison with similar groups of new users who first started using Firefox back in the fall of 2007.  As the internet population has grown and as Firefox has improved, has our ability to retain new users changed in any way?  Looking at previous analysis of those users – here and here – you’ll notice that at the 30-day mark after installation, we were seeing about 49% of original installers as active daily users.  Our new data above from the December and January cohorts suggests this rate is close to 44%.

How should we interpret this difference?

• First, we should be careful here.  The two sets of installation numbers (from 2007 and from present day) are coming from two different web analytics solutions.  The difference could be attributable solely to the measurement methodologies or reporting abilities of such solutions.
• Second, if we are to believe this difference could be real, it could suggest that as Firefox has reached more mainstream internet users (e.g, my parents), our ability to retain new users has changed.

Your thoughts?

Continuing our series of analysis based on the most recent edition of Funnelcake, we want to examine the experience of users when clicking the download button at mozilla.com – by browser.  In other words, what is the experience of a new Firefox user when he/she first arrives at our site while using Safari?  And is their experience any different than an IE user coming to mozilla.com and downloading Firefox?

First, let’s look at the breakdown of those clicking the Firefox download button (at the mozilla.com home page or main Firefox product page).

The picture above closely resembles the general internet population (i.e., worldwide market share).  I’m not sure if we would have expected to see something different.

Next, let’s examine the experience of each of the segments above.  Of those clicking the download button, how many receive the full data/file and how many don’t?

It doesn’t appear that there are any strong takeaways here.  The samples for Opera and “Other” are fairly small, so we should be cautious in drawing too many conclusions about the experience of those users.  Otherwise, the big three segments are relatively homogeneous.  One might expect the success of an existing Firefox user to be greater, but at the same time, there might be a higher percentage of accidental clicks by these users.

We recently dug into the user’s experience when downloading Firefox, answering questions such as, “of people clicking the Firefox download button at mozilla.com, how many receive the full file and how many complete the installation process?”

With the latest Funnelcake shipment (January 13th), we wanted to take these insights a bit further, and breakdown that question above in several dimensions – by OS, by browser, and by geographical region.  In this post, we’ll focus on the first dimension, i.e., how does the experience of users differ based on operating system?

To start, let’s look at the distribution of users who clicked the download button (on the mozilla.com homepage or main Firefox product page).

There’s nothing too surprising to note here.  This distribution should come close to mirroring the overall population of Firefox users and the population of internet users more generally.

Next, let’s see exactly what happened at the moment of clicking the download button.  You’ll recall from our previous post that we asked, “for every 100 people who initiate the download process, where exactly do they end up just a few moments later?”, and the answer was visualized through the diagram below:

(Note: while the percentages above highlight November’s Funnelcake edition, they remained almost identical in January).

Moving onto the punchline, below is the breakdown of these numbers by OS:

A couple key things to note:

• There seems to be a substantial difference between Mac and Windows users.  While both groups are able to successfully receive the file at roughly similar rates, the experience of users diverges once they move from downloading to completing the installation.
• We’re not entirely sure what’s going on with the Linux numbers.  Much more digging needs to be done.

Update: First, we should explain how we measure whether or not a user installs Firefox.  We count an installation as successful when a user visits the Firefox firstrun page.  Given this fact and considering Alix’s point below, it’s clear that Mac installations will be undercounted in this analysis, as Mac users have to manually launch Firefox for the first time in order to be counted as an install, while Windows users automatically hit the firstrun page.

We recently reinitiated Funnelcake, an effort begun in 2007 to better understand the full cycle of a user’s experience — from visiting www.mozilla.com all the way through to becoming a long-term Firefox user.  We are currently shipping Funnelcake roughly once per month, and we have some new insights to share based on some numbers from November’s edition.

You may recall that, last year, much of our analysis focused on the attrition or churn rate over time, i.e., what are the odds that a new user today continues using Firefox a month from today or six months from today?

For this discussion, we want to focus on another area – what is the experience of a new user at the precise moment he/she clicks on the download button?

At first glance, this may seem like an ultra simple thing to examine.  However, there are indeed different hoops a user has to navigate through at this moment in their experience.  Once a user clicks the download button, he/she is shown a dialog screen related to running or saving the .exe file.  At this moment, two things can happen:

1. A user succeeds in getting that download file
2. The user does not receive the full and complete file

Looking at our download mirror data, it appears that roughly 79% of people enjoy success (#1) and about 21% of users experience the latter (#2).

The next step in a user’s experience applies to group #1.  Moving forward, two things can happen to this user:

1. The user can launch/run the installation file, successfully go through all the steps during installation, and complete the installation process
2. The user does not complete installation

Looking at the data again, we see that of the users at this stage in the process, 79% of people enjoy success through the installation steps, whereas 21% do not (the 79-21 numbers repeating here is just a coincidence).

An alternate view of this user experience would be to ask, “for every 100 people who initiate the download process, where exactly do they end up just a few moments later?”  Looking at the diagram below, you can see that roughly 21 abandon the process when they don’t receive the full download file, 63 receive the download file and complete the installation process, and 17 receive the download file but don’t complete installation.

So, what are the business insights here?

There are likely some actions we can take to improve the user’s experience during the download and installation process.  I was surprised by the 21% number above… much more digging needs to be done to ensure this is fully accurate and to break this down further (e.g., by OS, by browser, etc.), so that we can start to consider product or UI related changes.  With respect to the 17% number above, we are already taking action.  With the help of Kampyle (a tool for these users to provide feedback), we’re eager to soon have some insights into the potential pain points or thoughts of users at this precise moment in their experience.