Mozilla as a Data Driven Community

Ken Kovash    plans, process    

I wanted to share my slides from last week’s analytics discussion at the Mozilla all hands meeting.  With those in attendance, we were able to successfully identify several ways in which the metrics team can be working in closer collaboration with other teams across Mozilla.  In addition, here are a few points of context that may not be evident via the slides below:

• Mozilla is already doing a great deal as a community and organization to be data driven in our actions and decision making
• That said, we’re just starting to scratch the surface
• Looking forward, my hope is that folks across Mozilla will rely more often on the analytics/metrics team before decisions are made and early on in initiatives (i.e., not just after the fact).  In other words, the potential here for greater impact (e.g., on the experience of Firefox users) and improved sustainability presents us (all of us) with many great opportunities.

Let me know what you think!

Why People Don’t Install Firefox – Part II

Ken Kovash    process, results    

As we described a few days ago, we recently shipped a modified version of Firefox with a different installer – one allowing users to leave feedback upon canceling.  In Part II of this series, we want to outline exactly what users saw and share some high-level findings.  An examination of users’ written feedback – and a discussion around associated insights, implications and next steps – will soon be coming in Part III.

What are all the steps of the Firefox installer (Windows)?  Where exactly can a user hit “cancel”? (Please click below for the full image)

When a user clicked “cancel”, what exactly did he/she see? (Please note that clicking on any sub-category, such as “Taking too long”, opens up a free form feedback box for the user to write their comments)

What were the high-level results from this feedback form?

Below is a screenshot from Kampyle’s dashboard (click for full image).  Kampyle, a start-up making huge strides within the analytics world, helped Mozilla with this initiative by hosting the feedback form you see above.  You’ll notice a few high-level pieces of info:

• What was the average sentiment/feeling of people upon canceling?
• How many total people left feedback?  (Answer: about 5,000 people hit the feedback form and about 330 left feedback)
• What were the most common “categories” of feedback left by users (i.e., types of problems cited)?
• What steps within the Firefox installer saw the most users abandoning?  And what was the feedback breakdown for each of these steps?

In our next post, we’ll dive into the free form feedback left by users (where we can more clearly identify a couple critical trouble spots) and discuss what it all means for possible product and/or marketing changes in the future.

Stay tuned!

Why People Don’t Install Firefox – Part I

Ken Kovash    process, results    

Installing a piece of software can be a challenging process for the typical person to traverse.  For example, with Firefox, we know that each day there are more than 50,000 people who fall into the following group – they complete the download process (receive the full data/file), but don’t then complete their installation of Firefox.

We feel we have a moral imperative to figure out what’s going on here.  Doing so provides us an opportunity to both add to the general knowledge base of the software industry and make significant strides in the Firefox user experience (and to boot, improving Firefox adoption rates isn’t a bad auxiliary outcome).

So, let’s start by outlining the exact steps that a software user needs to traverse after a successful download:

1. Does a user know they need to install?
2. Does a user remember where he/she saved the file?
3. Does the user open/run the file and avoid hitting “cancel” during extraction?
4. Does the user make it through all steps (either 3 or 6 steps in the case of Firefox) of the installer without hitting “cancel”?

Thanks to an important initiative launched last week, we’re now able to put some framework and answers around these questions.  On Thursday of last week, we shipped a modified version of Firefox with a different installer (en-US*Windows only for a single day), allowing users to leave feedback when canceling out of the installer.

Based on the results (coming in a follow-up post), we know that of our original cohort of 50,000 people, approximately 10,000 users fall under category #4 outlined above (i.e., they click on that cancel button).  That means steps #1-3 describes the experience/actions of about 40,000 people.

Thinking in terms of big picture implications, there are some possible installer changes that we can consider to remedy the issues encountered via the installer itself.  Perhaps more paramount – what can we do to help users from steps #1-3?  Content and design of the Firefox download page?  Any other ideas?

To be sure, any action that we take here could be wildly impactful.  Improving the experience for just 2,000 or 3,000 people from our original cohort would translate into upwards of 2,000,000 additional Firefox users annually.

More Improvements Coming to Firefox Web Sites

Ken Kovash    plans, process    

Last fall we initiated the first ever multivariate optimization testing at www.mozilla.com.  At the time, we drew the following conclusion from one of our tests:

How does this change our business/creative decision making?

As some of us already guessed, it appears that it’s the download button itself that matters most here.  Thinking about the size, shape, color, and placement (among other characteristics), our findings indicate that future testing could reveal surprising – and positive findings – based on changes to the download button.  In other words, it wouldn’t be a stretch to imagine that future tests, that adjust such characteristics of the download button, could easily translate to a 2% lift (or much greater), and in turn, a substantially improved experience for millions of new Firefox users each year.

We have plans in place to eventually roll-out those types of tests with regular frequency later this year (stay tuned), and in the short term, we’re planning a few strategic optimization efforts at a few critical Firefox related touch points – support.mozilla.com, Get Firefox in your language, and the main Firefox product page.

David Rolnitzky just highlighted our plans regarding that last touch point.  Among the changes being tested:

• If an existing Firefox user is trying to figure out how to upgrade to the latest version of Firefox, the process should be clearer.
• If a user is on IE, he/she will see content more relevant to their thoughts and potential interest in downloading Firefox (below).

This is only a test, so while these new page versions seem as though they’ll have a positive impact on users, we’re not necessarily rooting for them to exhibit greater success than our existing page (the “winner” is determined by the actions/preferences of site visitors).  Stay tuned for our findings in a future post.

Improving the Experience of Installing Firefox

Ken Kovash    plans, process    

When we last left off, we described an upcoming change to the Firefox installer – those users canceling out of the process (via en-US downloads initiated at www.mozilla.com) will be able to leave feedback if they’re interested:

Those checking the check box and clicking “Yes” will see the following feedback form open:

This form is hosted by Kampyle, a small start-up that has recently been making waves within the analytics world.  It’s difficult to tell via the screen shot above, but the interface and experience of completing their form is slick and should delight many users.

So, what do we expect as an outcome here?  What will be the impact?

We’re estimating that about 5,000 users will leave feedback during our one-day test (this is based on a cohort of 50,000 daily users who complete download, but don’t complete installation).  In crafting the questions in the form above (e.g., “Installation Problem”, “Suggestion?”), in some ways it’s difficult for us to precisely know what answers we want to see at the end of the process.  Mozilla has never previously done research around this moment in a user’s experience, so this first pass is expected to point us in the right direction, narrowing down what we should be thinking about and researching in the future.

The positive impact to Firefox users could be huge.  Direct feedback from thousands of users at a critical moment in time will likely provide a few key insights around our users, their pain points and their experiences.  Either from this initial test or from a longer term survey implementation, we should expect changes/improvements to Firefox (e.g., product, UI, UX, messaging) to result.

Does Installing Firefox Make a Person Feel Happy or Sad?

Ken Kovash    plans, process    

We’re obsessed with improving a person’s experience with Firefox at every turn, from finding Firefox, to downloading/installing Firefox, to actually using Firefox, to uninstalling Firefox. With an upcoming initiative, we’ve set out to improve the user’s experience at one specific moment in time – the installation process.

We know that about one in every seven Firefox downloaders complete the download process, but don’t actually install Firefox.  We’re also guessing that some percentage of people who do complete the installation process might traverse a less than optimal experience.

So what can we do about this?

Ask people if they’re interested in leaving feedback during that precise moment in time.

While an organization or community can spend all day doing data analysis to solve a problem, sometimes the best way to figure something out is to just ask people directly what they’re feeling.  To accomplish this, we’re making the following change to the Firefox installer.  If a user clicks “Cancel” during the installation process (Windows), he/she has traditionally seen the following:

By testing a new capability allowing users to leave feedback, the revised installer will change the “are you sure question” to look like this:

Our plan is to run this modified installer for a single day (sometime in March) and see what findings we arrive at before considering making such a change permanent.  There are a few things to note here:

• The default would be an unchecked check box.
• The image above is only a mock-up.  The actual modified installer will likely look somewhat different.

So, what happens if a user indicates they’re interested in providing feedback?  Stay tuned for a follow-up post…

How Do Firefox Users Use Firefox Support?

Ken Kovash    plans, process    

We recently took a look at one of the most critical activities within the Mozilla community – Firefox Support (support.mozilla.com). In that initial discussion, we investigated how Firefox users get to SUMO in the first place.  Next, we want to consider the most common ways that users interact with the site once they arrive.

To accomplish this, we’ll look at the cohort of users arriving at SUMO via the inproduct page.

We previously saw that more than 71% of users find SUMO this way (all locales).   Again looking at all Firefox locales, let’s see at the highest level what happens when a user enters SUMO:

Of the 9.2 million users entering SUMO last month through inproduct help, 14.3% used the site in some way.  While the “bounce” rate of 85.7% is a little concerning, our SUMO community is currently making significant strides in an effort to increase that 14.3 percentage and improve the user’s experience.  For example, the SUMO team is planning to roll-out a couple A/B optimization tests of the SUMO home page to help determine some reasons behind the bounce rate and some solutions around improving it.

Next, let’s look at what a user actually does on the SUMO site.  Of the blue pie slice cohort above, what is the typical experience of this user?  At a high level across all locales, we see that 71% start by utilizing the knowledge base search box and the other 29% click on something else on the page.

Perhaps the most insightful story that we can glean… the 71% of users searching varies significantly depending on where in the world a Firefox user is located.  For users of the en-US locale of Firefox, that number is 78.4%, and for users of any non en-US locale, that number is 60.2%.  That’s a big difference in how a user’s interaction with SUMO varies by region!

Lastly, for the 29% of users who are clicking elsewhere upon their arrival at SUMO, what exactly do they find most interesting, i.e., what exactly are they clicking on?  For this analysis, we’re going to focus on just the en-US inproduct page.  The half dozen most popular articles/links are highlighted (i.e., ranked by most clicks) below.

Summing up this discussion, we’ve just walked through (at a very high level) what a typical user’s interaction looks like with Firefox Support.  We hope this dialogue will help draw added attention to the fantastic SUMO community, with the end goal of arriving at ideas that will drive a better experience for Firefox users seeking support.

Please comment!

Interacting with One of the Web’s Most Popular Pages

Ken Kovash    process, results    

By far the most visited page across Mozilla’s web sites is the “whatsnew” (or update) page.  Each time there’s a release (e.g., 3.0.6), users automatically get updated to the latest version of Firefox, and as part of that process, the user’s browser shows the following page:

So, what makes us think this might be one of the most visited pages on the web?  Here are some numbers for the 3.0.6 associated page:

• We’ve seen a total of about 90,000,000 page views (all locales, over the past ten days or so)
• About 36,000,000 visits to the en-US locale version

The next question is:  How do users actually interact with this page?

To answer this, we looked at a heat map of the en-US page version.  You’ll see across the different places users can click, we’ve seen about 1.3 million total clicks.  That translates to a click-through rate of about 3.6%, which seems pretty good to me.  The other thing that stands out is the relatively high concentration of clicks in the middle section – “View the top new features” and “See what’s new in Firefox 3.0.6″.

Does anything else stand out to you?

Improving the Firefox Experience – 84,000,000 Users at a Time

Ken Kovash    plans, process    

When we recently shared an overview of the traffic to www.mozilla.com, we noted one item that stood out – the relatively high volume of traffic to the Getting Started page.

Digging into the data, we wanted to understand the typical user experience with this page.  In other words, why do users come to it?  How do they get to it? What are their expectations of the page?  Do they find the page helpful or confusing?  Considering these questions, we arrived at a startling finding…

• Approximately 90% of the traffic comes from the bookmark toolbar (image below) and about 10% arrives via the featured link on the firstrun page
• Ok, that fact alone isn’t surprising, but what is surprising is this:
• The users clicking through via our firstrun page showed a 63% bounce rate on the Getting Started page (bounce rate means the visitor immediately leaves the site upon viewing a particular page)
• The users clicking through from the bookmark toolbar showed a whopping 92% bounce rate!

Why is this such a big deal?

The Getting Started page saw more than 24 million visitors last month (all locales), and in each instance of a visit, the user is making a conscious decision to visit the page.  To put the scale of this number in perspective… of all “user driven” traffic at Mozilla.com, only the main Firefox product page comes remotely close to matching that traffic volume.

Of the total group of 24 million visitors, the cohort of visitors arriving via the bookmark toolbar was nearly 22 million strong.  In other words, 22 million users exhibited that 92% bounce rate.  To boot, we know from the other cohort of visitors (those arriving via the firstrun page), that the bounce rate doesn’t have to be that high.  Here is a breakdown in visitor behavior for the just the en-US version of the page (1 million visitors make up the first pie, 10 million in the latter, hence the scale difference):

If we made changes to Firefox and to the Getting Started page in such a way that lowered the 92% bounce rate to 63% (within the realm of possibility), that would equate to an improved experience for 7 million users each month.  Extrapolating to an annual impact – that’s 84 million users that could be positively affected.

Where do we go from here?

First, we need to determine the root of the issue.  Should “Getting Started” reside in the Firefox bookmark toolbar?  Or should we be focusing on the page itself?

Assuming we leave the browser as-is, it would behoove all of us (myself included) to consider radical solutions for better meeting the expectations of users visiting the Getting Started page (remember: these users are proactively choosing to visit this page).  Please leave a comment.  I’m eager to work towards a solution, and ultimately, towards a better experience for tens of millions of Firefox users.

Analytics and Firefox Support

Ken Kovash    plans, process    

Over the past year, SUMO (also known as Firefox Support, Firefox Help, or support.mozilla.com) has grown into one of the most critical activities within the Mozilla community.  David Tenser and the SUMO group have done a stellar job of maintaining a SUMO metrics report, and to complement that, I’ve recently been talking with David about the broader story of a typical user’s experience with SUMO.

There are a bunch of different ways that users can find SUMO and there are many different ways for users to interact with the site and get the help they’re looking for, so I thought it would be interesting to start outlining these user experiences at a high-level.  In this discussion, I’d like to answer two questions:

1. How does a Firefox user get to SUMO in the first place?
2. What is a user’s most common interaction/experience on the site?

For question #1, there are two primary ways for a Firefox user to locate and arrive at SUMO.  First, there’s a support option within the browser itself (i.e., inproduct).  Click on “Help” at the top of your browser and you’ll see something called “Help Contents”.  Users clicking here go directly to support.mozilla.com.

The second primary way for users to enter SUMO is through the main navigation bar at www.mozilla.com and localized Mozilla sites.  The navigation bar for the mozilla.com web site includes five menu options – Products, Add-ons, Support, Community, and About.  Users clicking on “Support” (depending on the localized version) go directly to support.mozilla.com.

So, how are Firefox users actually getting to SUMO in the first place?

Last month, SUMO saw 12.9 million visitors.  Of these visitors, more than 71% entered the site via the “inproduct” path (Help -> Help Contents), about 13% entered the site via the main navigation menu on Mozilla’s web sites, and about 16% entered through some other path.

A few things seem surprising here:

• That 71% slice of the pie is huge!
• 12.9 million visitors to SUMO (last month) is also a massive number.  To put that in some perspective, last month about 4.3 million visitors entered the www.mozilla.com site through the home page.  It’s not quite a fair apples-to-apples comparison, but still…

Continuing this discussion, we’ll soon post some analysis around question #2 – What is a user’s most common interaction/experience on the site?