Blog of Metrics

When we last left off, we described an upcoming change to the Firefox installer – those users canceling out of the process (via en-US downloads initiated at www.mozilla.com) will be able to leave feedback if they’re interested:

Those checking the check box and clicking “Yes” will see the following feedback form open:

This form is hosted by Kampyle, a small start-up that has recently been making waves within the analytics world.  It’s difficult to tell via the screen shot above, but the interface and experience of completing their form is slick and should delight many users.

So, what do we expect as an outcome here?  What will be the impact?

We’re estimating that about 5,000 users will leave feedback during our one-day test (this is based on a cohort of 50,000 daily users who complete download, but don’t complete installation).  In crafting the questions in the form above (e.g., “Installation Problem”, “Suggestion?”), in some ways it’s difficult for us to precisely know what answers we want to see at the end of the process.  Mozilla has never previously done research around this moment in a user’s experience, so this first pass is expected to point us in the right direction, narrowing down what we should be thinking about and researching in the future.

The positive impact to Firefox users could be huge.  Direct feedback from thousands of users at a critical moment in time will likely provide a few key insights around our users, their pain points and their experiences.  Either from this initial test or from a longer term survey implementation, we should expect changes/improvements to Firefox (e.g., product, UI, UX, messaging) to result.

We’re obsessed with improving a person’s experience with Firefox at every turn, from finding Firefox, to downloading/installing Firefox, to actually using Firefox, to uninstalling Firefox. With an upcoming initiative, we’ve set out to improve the user’s experience at one specific moment in time – the installation process.

We know that about one in every seven Firefox downloaders complete the download process, but don’t actually install Firefox.  We’re also guessing that some percentage of people who do complete the installation process might traverse a less than optimal experience.

So what can we do about this?

Ask people if they’re interested in leaving feedback during that precise moment in time.

While an organization or community can spend all day doing data analysis to solve a problem, sometimes the best way to figure something out is to just ask people directly what they’re feeling.  To accomplish this, we’re making the following change to the Firefox installer.  If a user clicks “Cancel” during the installation process (Windows), he/she has traditionally seen the following:

By testing a new capability allowing users to leave feedback, the revised installer will change the “are you sure question” to look like this:

Our plan is to run this modified installer for a single day (sometime in March) and see what findings we arrive at before considering making such a change permanent.  There are a few things to note here:

• The default would be an unchecked check box.
• The image above is only a mock-up.  The actual modified installer will likely look somewhat different.

So, what happens if a user indicates they’re interested in providing feedback?  Stay tuned for a follow-up post…

We recently took a look at one of the most critical activities within the Mozilla community – Firefox Support (support.mozilla.com). In that initial discussion, we investigated how Firefox users get to SUMO in the first place.  Next, we want to consider the most common ways that users interact with the site once they arrive.

To accomplish this, we’ll look at the cohort of users arriving at SUMO via the inproduct page.

We previously saw that more than 71% of users find SUMO this way (all locales).   Again looking at all Firefox locales, let’s see at the highest level what happens when a user enters SUMO:

Of the 9.2 million users entering SUMO last month through inproduct help, 14.3% used the site in some way.  While the “bounce” rate of 85.7% is a little concerning, our SUMO community is currently making significant strides in an effort to increase that 14.3 percentage and improve the user’s experience.  For example, the SUMO team is planning to roll-out a couple A/B optimization tests of the SUMO home page to help determine some reasons behind the bounce rate and some solutions around improving it.

Next, let’s look at what a user actually does on the SUMO site.  Of the blue pie slice cohort above, what is the typical experience of this user?  At a high level across all locales, we see that 71% start by utilizing the knowledge base search box and the other 29% click on something else on the page.

Perhaps the most insightful story that we can glean… the 71% of users searching varies significantly depending on where in the world a Firefox user is located.  For users of the en-US locale of Firefox, that number is 78.4%, and for users of any non en-US locale, that number is 60.2%.  That’s a big difference in how a user’s interaction with SUMO varies by region!

Lastly, for the 29% of users who are clicking elsewhere upon their arrival at SUMO, what exactly do they find most interesting, i.e., what exactly are they clicking on?  For this analysis, we’re going to focus on just the en-US inproduct page.  The half dozen most popular articles/links are highlighted (i.e., ranked by most clicks) below.

Summing up this discussion, we’ve just walked through (at a very high level) what a typical user’s interaction looks like with Firefox Support.  We hope this dialogue will help draw added attention to the fantastic SUMO community, with the end goal of arriving at ideas that will drive a better experience for Firefox users seeking support.

Please comment!

When we recently shared an overview of the traffic to www.mozilla.com, we noted one item that stood out – the relatively high volume of traffic to the Getting Started page.

Digging into the data, we wanted to understand the typical user experience with this page.  In other words, why do users come to it?  How do they get to it? What are their expectations of the page?  Do they find the page helpful or confusing?  Considering these questions, we arrived at a startling finding…

• Approximately 90% of the traffic comes from the bookmark toolbar (image below) and about 10% arrives via the featured link on the firstrun page
• Ok, that fact alone isn’t surprising, but what is surprising is this:
• The users clicking through via our firstrun page showed a 63% bounce rate on the Getting Started page (bounce rate means the visitor immediately leaves the site upon viewing a particular page)
• The users clicking through from the bookmark toolbar showed a whopping 92% bounce rate!

Why is this such a big deal?

The Getting Started page saw more than 24 million visitors last month (all locales), and in each instance of a visit, the user is making a conscious decision to visit the page.  To put the scale of this number in perspective… of all “user driven” traffic at Mozilla.com, only the main Firefox product page comes remotely close to matching that traffic volume.

Of the total group of 24 million visitors, the cohort of visitors arriving via the bookmark toolbar was nearly 22 million strong.  In other words, 22 million users exhibited that 92% bounce rate.  To boot, we know from the other cohort of visitors (those arriving via the firstrun page), that the bounce rate doesn’t have to be that high.  Here is a breakdown in visitor behavior for the just the en-US version of the page (1 million visitors make up the first pie, 10 million in the latter, hence the scale difference):

If we made changes to Firefox and to the Getting Started page in such a way that lowered the 92% bounce rate to 63% (within the realm of possibility), that would equate to an improved experience for 7 million users each month.  Extrapolating to an annual impact – that’s 84 million users that could be positively affected.

Where do we go from here?

First, we need to determine the root of the issue.  Should “Getting Started” reside in the Firefox bookmark toolbar?  Or should we be focusing on the page itself?

Assuming we leave the browser as-is, it would behoove all of us (myself included) to consider radical solutions for better meeting the expectations of users visiting the Getting Started page (remember: these users are proactively choosing to visit this page).  Please leave a comment.  I’m eager to work towards a solution, and ultimately, towards a better experience for tens of millions of Firefox users.

Over the past year, SUMO (also known as Firefox Support, Firefox Help, or support.mozilla.com) has grown into one of the most critical activities within the Mozilla community.  David Tenser and the SUMO group have done a stellar job of maintaining a SUMO metrics report, and to complement that, I’ve recently been talking with David about the broader story of a typical user’s experience with SUMO.

There are a bunch of different ways that users can find SUMO and there are many different ways for users to interact with the site and get the help they’re looking for, so I thought it would be interesting to start outlining these user experiences at a high-level.  In this discussion, I’d like to answer two questions:

1. How does a Firefox user get to SUMO in the first place?
2. What is a user’s most common interaction/experience on the site?

For question #1, there are two primary ways for a Firefox user to locate and arrive at SUMO.  First, there’s a support option within the browser itself (i.e., inproduct).  Click on “Help” at the top of your browser and you’ll see something called “Help Contents”.  Users clicking here go directly to support.mozilla.com.

The second primary way for users to enter SUMO is through the main navigation bar at www.mozilla.com and localized Mozilla sites.  The navigation bar for the mozilla.com web site includes five menu options – Products, Add-ons, Support, Community, and About.  Users clicking on “Support” (depending on the localized version) go directly to support.mozilla.com.

So, how are Firefox users actually getting to SUMO in the first place?

Last month, SUMO saw 12.9 million visitors.  Of these visitors, more than 71% entered the site via the “inproduct” path (Help -> Help Contents), about 13% entered the site via the main navigation menu on Mozilla’s web sites, and about 16% entered through some other path.

A few things seem surprising here:

• That 71% slice of the pie is huge!
• 12.9 million visitors to SUMO (last month) is also a massive number.  To put that in some perspective, last month about 4.3 million visitors entered the www.mozilla.com site through the home page.  It’s not quite a fair apples-to-apples comparison, but still…

Continuing this discussion, we’ll soon post some analysis around question #2 – What is a user’s most common interaction/experience on the site?