Blog of Metrics

When we first started talking about the Firefox Uninstall Survey last year, we introduced it with this:

The world of Mozilla includes many Firefox related feedback mechanisms.  Crash reporting, bugzilla and spreadfirefox.com represent a few such examples.  Another feedback mechanism is the Firefox uninstall survey.  When a Firefox user (Windows XP only) decides to uninstall their browser, they’re asked if they want to complete a survey during the uninstall process.

Often a user is merely temporarily uninstalling before immediately reinstalling.  However, in some cases a user enjoys a seemingly less than optimal experience with Firefox, and in these cases, it behooves us a community to figure out what’s going on and if there are true pain points that the user is experiencing (and perhaps that we’re not aware of).

From there, we eventually highlighted some key steps moving forward, including revised questions and design, along with more usable/insightful/actionable data and reporting.  Fast forwarding many months, we finally implemented the new survey this past Friday.

The old survey:

The new survey (with thank-you page):

Here are a few implementation details:

• We’ve embedded a SurveyGizmo survey within a www.mozilla.com page.  This allows us to do two things – (1) update the look and feel of the survey, and (2) have a reporting engine that will be useful and usable.
• Since the survey first came into existence back in 2005 or 2006, all of the data and responses have always been available publicly (at survey.mozilla.com).  The problem was the data wasn’t always fully up-to-date and it wasn’t available in a friendly format.  We’re currently looking at options for making the new reporting from SurveyGizmo both user friendly and publicly available.  More details to come very soon!
• Thanks to Fred Wenzel, Blake Cutler, among many others, for making this happen.

Lastly, let’s not forget why this initiative is so critical.  With tens of thousands of users telling us why they’re uninstalling Firefox, the Firefox team and the broader community should now be able to arrive at several key insights around usage pain points… insights that will ultimately drive user experience improvements… improvements that will eventually impact tens of millions of Firefox users.

I wanted to share my slides from last week’s analytics discussion at the Mozilla all hands meeting.  With those in attendance, we were able to successfully identify several ways in which the metrics team can be working in closer collaboration with other teams across Mozilla.  In addition, here are a few points of context that may not be evident via the slides below:

• Mozilla is already doing a great deal as a community and organization to be data driven in our actions and decision making
• That said, we’re just starting to scratch the surface
• Looking forward, my hope is that folks across Mozilla will rely more often on the analytics/metrics team before decisions are made and early on in initiatives (i.e., not just after the fact).  In other words, the potential here for greater impact (e.g., on the experience of Firefox users) and improved sustainability presents us (all of us) with many great opportunities.

Let me know what you think!

Last fall we initiated the first ever multivariate optimization testing at www.mozilla.com.  At the time, we drew the following conclusion from one of our tests:

How does this change our business/creative decision making?

As some of us already guessed, it appears that it’s the download button itself that matters most here.  Thinking about the size, shape, color, and placement (among other characteristics), our findings indicate that future testing could reveal surprising – and positive findings – based on changes to the download button.  In other words, it wouldn’t be a stretch to imagine that future tests, that adjust such characteristics of the download button, could easily translate to a 2% lift (or much greater), and in turn, a substantially improved experience for millions of new Firefox users each year.

We have plans in place to eventually roll-out those types of tests with regular frequency later this year (stay tuned), and in the short term, we’re planning a few strategic optimization efforts at a few critical Firefox related touch points – support.mozilla.com, Get Firefox in your language, and the main Firefox product page.

David Rolnitzky just highlighted our plans regarding that last touch point.  Among the changes being tested:

• If an existing Firefox user is trying to figure out how to upgrade to the latest version of Firefox, the process should be clearer.
• If a user is on IE, he/she will see content more relevant to their thoughts and potential interest in downloading Firefox (below).

This is only a test, so while these new page versions seem as though they’ll have a positive impact on users, we’re not necessarily rooting for them to exhibit greater success than our existing page (the “winner” is determined by the actions/preferences of site visitors).  Stay tuned for our findings in a future post.

When we last left off, we described an upcoming change to the Firefox installer – those users canceling out of the process (via en-US downloads initiated at www.mozilla.com) will be able to leave feedback if they’re interested:

Those checking the check box and clicking “Yes” will see the following feedback form open:

This form is hosted by Kampyle, a small start-up that has recently been making waves within the analytics world.  It’s difficult to tell via the screen shot above, but the interface and experience of completing their form is slick and should delight many users.

So, what do we expect as an outcome here?  What will be the impact?

We’re estimating that about 5,000 users will leave feedback during our one-day test (this is based on a cohort of 50,000 daily users who complete download, but don’t complete installation).  In crafting the questions in the form above (e.g., “Installation Problem”, “Suggestion?”), in some ways it’s difficult for us to precisely know what answers we want to see at the end of the process.  Mozilla has never previously done research around this moment in a user’s experience, so this first pass is expected to point us in the right direction, narrowing down what we should be thinking about and researching in the future.

The positive impact to Firefox users could be huge.  Direct feedback from thousands of users at a critical moment in time will likely provide a few key insights around our users, their pain points and their experiences.  Either from this initial test or from a longer term survey implementation, we should expect changes/improvements to Firefox (e.g., product, UI, UX, messaging) to result.

We’re obsessed with improving a person’s experience with Firefox at every turn, from finding Firefox, to downloading/installing Firefox, to actually using Firefox, to uninstalling Firefox. With an upcoming initiative, we’ve set out to improve the user’s experience at one specific moment in time – the installation process.

We know that about one in every seven Firefox downloaders complete the download process, but don’t actually install Firefox.  We’re also guessing that some percentage of people who do complete the installation process might traverse a less than optimal experience.

So what can we do about this?

Ask people if they’re interested in leaving feedback during that precise moment in time.

While an organization or community can spend all day doing data analysis to solve a problem, sometimes the best way to figure something out is to just ask people directly what they’re feeling.  To accomplish this, we’re making the following change to the Firefox installer.  If a user clicks “Cancel” during the installation process (Windows), he/she has traditionally seen the following:

By testing a new capability allowing users to leave feedback, the revised installer will change the “are you sure question” to look like this:

Our plan is to run this modified installer for a single day (sometime in March) and see what findings we arrive at before considering making such a change permanent.  There are a few things to note here:

• The default would be an unchecked check box.
• The image above is only a mock-up.  The actual modified installer will likely look somewhat different.

So, what happens if a user indicates they’re interested in providing feedback?  Stay tuned for a follow-up post…

We recently took a look at one of the most critical activities within the Mozilla community – Firefox Support (support.mozilla.com). In that initial discussion, we investigated how Firefox users get to SUMO in the first place.  Next, we want to consider the most common ways that users interact with the site once they arrive.

To accomplish this, we’ll look at the cohort of users arriving at SUMO via the inproduct page.

We previously saw that more than 71% of users find SUMO this way (all locales).   Again looking at all Firefox locales, let’s see at the highest level what happens when a user enters SUMO:

Of the 9.2 million users entering SUMO last month through inproduct help, 14.3% used the site in some way.  While the “bounce” rate of 85.7% is a little concerning, our SUMO community is currently making significant strides in an effort to increase that 14.3 percentage and improve the user’s experience.  For example, the SUMO team is planning to roll-out a couple A/B optimization tests of the SUMO home page to help determine some reasons behind the bounce rate and some solutions around improving it.

Next, let’s look at what a user actually does on the SUMO site.  Of the blue pie slice cohort above, what is the typical experience of this user?  At a high level across all locales, we see that 71% start by utilizing the knowledge base search box and the other 29% click on something else on the page.

Perhaps the most insightful story that we can glean… the 71% of users searching varies significantly depending on where in the world a Firefox user is located.  For users of the en-US locale of Firefox, that number is 78.4%, and for users of any non en-US locale, that number is 60.2%.  That’s a big difference in how a user’s interaction with SUMO varies by region!

Lastly, for the 29% of users who are clicking elsewhere upon their arrival at SUMO, what exactly do they find most interesting, i.e., what exactly are they clicking on?  For this analysis, we’re going to focus on just the en-US inproduct page.  The half dozen most popular articles/links are highlighted (i.e., ranked by most clicks) below.

Summing up this discussion, we’ve just walked through (at a very high level) what a typical user’s interaction looks like with Firefox Support.  We hope this dialogue will help draw added attention to the fantastic SUMO community, with the end goal of arriving at ideas that will drive a better experience for Firefox users seeking support.

Please comment!

When we recently shared an overview of the traffic to www.mozilla.com, we noted one item that stood out – the relatively high volume of traffic to the Getting Started page.

Digging into the data, we wanted to understand the typical user experience with this page.  In other words, why do users come to it?  How do they get to it? What are their expectations of the page?  Do they find the page helpful or confusing?  Considering these questions, we arrived at a startling finding…

• Approximately 90% of the traffic comes from the bookmark toolbar (image below) and about 10% arrives via the featured link on the firstrun page
• Ok, that fact alone isn’t surprising, but what is surprising is this:
• The users clicking through via our firstrun page showed a 63% bounce rate on the Getting Started page (bounce rate means the visitor immediately leaves the site upon viewing a particular page)
• The users clicking through from the bookmark toolbar showed a whopping 92% bounce rate!

Why is this such a big deal?

The Getting Started page saw more than 24 million visitors last month (all locales), and in each instance of a visit, the user is making a conscious decision to visit the page.  To put the scale of this number in perspective… of all “user driven” traffic at Mozilla.com, only the main Firefox product page comes remotely close to matching that traffic volume.

Of the total group of 24 million visitors, the cohort of visitors arriving via the bookmark toolbar was nearly 22 million strong.  In other words, 22 million users exhibited that 92% bounce rate.  To boot, we know from the other cohort of visitors (those arriving via the firstrun page), that the bounce rate doesn’t have to be that high.  Here is a breakdown in visitor behavior for the just the en-US version of the page (1 million visitors make up the first pie, 10 million in the latter, hence the scale difference):

If we made changes to Firefox and to the Getting Started page in such a way that lowered the 92% bounce rate to 63% (within the realm of possibility), that would equate to an improved experience for 7 million users each month.  Extrapolating to an annual impact – that’s 84 million users that could be positively affected.

Where do we go from here?

First, we need to determine the root of the issue.  Should “Getting Started” reside in the Firefox bookmark toolbar?  Or should we be focusing on the page itself?

Assuming we leave the browser as-is, it would behoove all of us (myself included) to consider radical solutions for better meeting the expectations of users visiting the Getting Started page (remember: these users are proactively choosing to visit this page).  Please leave a comment.  I’m eager to work towards a solution, and ultimately, towards a better experience for tens of millions of Firefox users.