Blog of Metrics

Let’s start with the punch line:

• People love Firefox!
• There is a small segment of existing users running into a particular problem – Firefox is not saving their preferences

Taking a step back, let’s walk through how we arrived at these findings.  We previously described our efforts integrating user outreach into mozilla.com.  For this post, we’re going to focus just on comments left by users during their visit to the Firefox “First Run” page, which brand new users hit upon downloading/installing Firefox for the first time (analysis of the Whatsnew and Getting Started pages will soon follow).

About 1,200 people left feedback over the past month, and below is a breakdown of the feedback.  If you’re curious how this worked from the user’s perspective, click here to the firstrun page and then click on the orange feedback button.

So, overall, the feedback looks really amazing.  For such a high percentage of people to go out of their way to say something positive is incredible.

On the downside, there were two issues identified by users that we weren’t previously aware of:

1. Many people complained about the open video freezing.  I’m not sure exactly what the issue is, but we can note that the next version of this page (for Fx3.6) will not include the same video.
2. Perhaps more critically, many users talked about their frustration around Firefox not saving their settings.  The most common comment was along the lines of, “why does this page open every time I open Firefox?  Please get rid of it.  I previously set my home page and Firefox won’t save the setting.” In addition, some people noted other preferences/settings not saving – add-ons, personas, favorites and bookmarks.

How do we know that this latter group of users is frustrated?  In addition to reading their actual comments, we can also look at the average satisfaction score (scale of 1 to 5) left by users in each feedback group.  The findings are below… it’s pretty clear that this group, with a satisfaction score of 1.9, has been feeling extremely frustrated.

What’s the good news?

Thanks to this insight, we’ve been able to prioritize a fix via bug 495735, and we’re hoping to ship it in the next release of Firefox!  Thanks to Dietrich Ayala, Mats Palmgren, the Firefox team, and several others for making this happen.

Mozilla recently announced the launch of a “plugin check” page (pictured below), allowing users to see a list of their installed plugins and whether or not any updates are needed.  This comes on the heels of a program where we successfully helped more than 10,000,000 Firefox users with a non-current version of Flash get to Adobe’s download/update page.

Now that this new program has been live for about a week, we thought we’d check the numbers (cumulative since last week):

Considering that this page hasn’t been directly pushed to users, these numbers are significant.  There are a couple spots within mozilla.com where users can click to find this page, but otherwise, our web analytics reporting tells us that about 90% of the traffic has been coming from news and technology related web sites.

Next, you may be wondering where people ventured upon clicking to “Update” or “Research”.  Here’s a breakdown:

We’ll continue to keep an eye on these numbers, and perhaps more excitedly, there are plans to soon integrate this feature into the browser itself.

We discussed a couple more prominent issues at the time, but when we last analyzed why people cancel out of the Firefox installation process (the actual installer), we found one other significant group of people who could use some assistance – a cohort of existing Firefox users who were confused about what they were doing (in this case, a fresh install) and what they were optimally supposed to be doing (e.g., going to Help -> Check for Updates).

This insight brought me back in time.

When I first started talking with Mozilla folks a few years ago (before joining the org), one of the first things I wanted to do was make sure I was using the latest version of Firefox.  At the time, I went to mozilla.com and tried looking for information about how to do so.  I didn’t find anything and then searched on Google (again, unsuccessfully).  Eventually, I gave up and just did a fresh download and install, all the while not feeling totally sure I was doing the right thing.

Now, armed with our new insight, a couple changes are either on the way or have just been implemented:

• The Firefox team is making one small UI change (mentioned in Rob Strong’s blog post)
• The marketing team changed the main Firefox product page at www.mozilla.com, as seen by existing Firefox users.  Below are the two page variations, and I’ve highlighted the new additions to the pages.  These features point to this newly created FAQ page answering many key questions related to… how do I update/upgrade?  do I need to do a fresh install?  do I need to uninstall first?  what will happen to my bookmarks?  etc.

Thanks to Laura Mesa, John Slater, among many others, for their efforts.  It’s always nice to see our metrics activities come full circle – data -> insights -> actions/changes -> improved user experience!

When we left off, we showed that the #1 reason why people refused the Firefox Major Update offer was their frustration/preference/confusion related to the awesome bar (or location bar).  There are a few things interesting with this piece of feedback and with all the feedback more generally:

1. Virtually everyone who provided feedback had previously used Fx3 and subsequently reverted back to Fx2.  This was 100% descriptive of this cohort and largely described all other users who were kind enough to share their thoughts.
2. Some of the issues cited have since been resolved (since last summer).  Clearly, many users tried Fx3 when it was initially released, then reverted back to Fx2 shortly thereafter, and now they still won’t consider upgrading because of their initial experience (even if that initial issue has since been resolved).  You never get a second chance to make a first impression!

Returning to our cohort of users not fully satisfied with or fully comfortable with the awesome bar, we’ve released a series of enhancements in Fx3.5 that should resolve their concerns.  Alex Faaborg provided an extensive walk-through, and here’s one key section that should help this group of users:

Ability to Control What Appears in the Location Bar Search Results (Proactive Privacy)

When we expanded the capabilities of the location bar to search against all history and bookmarks in Firefox 3, a lot of people contacted us to say that they had certain bookmarks they didn’t really want to have displayed. In some cases users had intentionally hidden these bookmarks in deep hierarchies of folders, somewhat similar to how one might hide a physical object. Having something from your previous browsing displayed to someone else who is using your computer (or even worse) to a large audience of people as you are giving a presentation, is really one of the most embarrassing things that Firefox can do to you. So now in Firefox 3.5, users have complete control over what types of information are displayed in the location bar (or suggestions can be turned off entirely):

So, what actions items can come of this user feedback?

If we think this concern surrounding the location bar is solely limited to existing Fx2 users, we could consider some special messaging for them.  However, I would venture to guess that some small fraction of the Firefox user base currently on either Fx3 or Fx3.5 (93% of all users) share similar thoughts about the location bar.  If so, then it would likely be worthwhile for us consider a couple options:

• Modify Firefox itself.

Below is a simplified mock-up of just one idea I came up with (please note that I don’t know anything about UI).  I’m not sure if the average user knows about Tools->Options, so the concept here is that a person could easily understand how the location bar works and adjust it (according to their preferences) within a fraction of a second.

• Up-level messaging about the location bar and its latest enhancements.

Utilizing key touch points, such as the firstrun page, whatsnew page, or firefox.com page, could help users feel more comfortable with the location bar and its awesome functionality.  For example, is it more important for us to be using these critical touch points to promote open video formats and Fx add-ons, or to highlight Firefox’s most utilized feature and how it can best help a user in his/her everyday life (and not frustrate them)?  At the very least, this seems like a question we should be asking.

What are your thoughts?

As mentioned a few days ago, we saw a massive impact to the Firefox new user experience with a recently implemented change to the Firefox installer.  A problem previously encountered daily by thousands of new Firefox users completely vanished with a single change.  We now want to share a few more details and suggest one or two additional changes moving forward.

First up, let’s look at the distribution of users’ comments — by step — in the Firefox installation process, i.e., on what step were they on when hitting the “cancel” button (a visualization of these steps is shown here).  The screenshots below come courtesy of Kampyle, the start-up who helped us with this initiative.

THEN (late March):

NOW (last week):

The difference between these two pictures is night and day.  A couple things stand out:

• As mentioned above, our fix successfully resolved the directory problem that users were primary encountering in Step #3 of our installer (“Choose Install Location”)
• There is now a very high concentration of people canceling out of the installer at the final step in the process (Step #6, aka “Summary”)

Looking at the actual comments from users canceling out of the final step, about 75% are related to the zombie process problem (a workaround solution is in the works), about 20% are people expressing their unhappiness over Firefox becoming their default browser, and about 5% fall into an “other” category.  You’ll recall that those two issues were the top issues highlighted by users overall.

What’s up with people commenting on Firefox becoming their default?

It’s a simple option to check or uncheck as desired, right?

Diving into the comments reveals something critical.  All of the people making this comment are doing so on the “Summary” step (or final step, below), not on the earlier “Setup” step (above).  Clearly, many new users are overlooking the checkbox, and just seconds later, are unsure as to how to install Firefox as their non-default browser.  And the outcome here is poor – the person is frustrated and refuses to install Firefox.

What fix should we consider here?  Should we make the checkbox more prominent on the “Setup Type” step?  Should we include instructions (e.g., “simply hit the back button to change”) on the “Summary” step?

Thanks to Rob Strong, the Funnelcake team, and the folks at Kampyle, among many others, for making this ongoing initiative a success.

A few months ago, we laid out the following:

“Installing a piece of software can be a challenging process for the typical person to traverse.  For example, with Firefox, we know that each day there are more than 50,000 people who fall into the following group – they complete the download process (receive the full data/file), but don’t then complete their installation of Firefox.”

To help solve this problem, we integrated the following feedback mechanism into the Firefox installer for a single day in late March.

As a result, we arrived at two critical issues that were preventing users from installing Firefox:

1. Users were running into directory issues.  Example comments were along the lines of — “telling me i don’t have access to write to the installation directory… .”  And non-Firefox users, in particular, were experiencing this problem.
2. Firefox not exiting/closing cleanly (e.g., a zombie process).

In response, the Firefox team integrated a fix for Problem #1 into the Fx3.5 installer (released last month).  In short, the installer now requests elevation for non-admin users on Vista and above when UAC is turned on (see bug 404541 for details) and fallback to non-privileged install locations when the user doesn’t have write access to the Program Files directory.

Did the fix work?  Did the modified Firefox installer actually lead to an improved user experience?

Yes.

This past Tuesday, we re-ran the feedback mechanism for a 24-hour period, so that any user canceling out of the Firefox installer was asked if they wanted to leave feedback.  And what did we find?  With the chart below, of the 400+ responses we received, I went through the individual comments left by users and grouped them into 5 different categories.  Problem #1 (noted above) virtually disappeared.  Problem #2 grew its overall share of the pie (compared with last time).

How did we arrive at our estimated impact of 2,000,000 users?  We estimate that roughly 5,000 people were running into Problem #1 – daily.  Multiplied by 365 days, that equates to roughly 2 million people who will be aided by an improved experience over the next year.

Up next on the horizon… the Firefox team is looking into implementing some workaround solutions for Firefox users falling into the blue pie slice above (the timeframe is a couple months out).  Also, with our next post, I’ll dive into the second most popular comment – selecting Firefox as your default browser during the installation process.  Stay tuned!

One of the ways Firefox users can easily and directly contribute to making the browser a better product is through our Help->Report Broken Web Site feature:

This allows users to let us know if they experience problems with Firefox on specific websites. Let’s take a high-level look at this “Broken Site Reporter” data over the past few years:

These numbers are significant.  Each day, over a thousand Firefox users are showing us pain points they have experienced while surfing the web. This sort of feedback can be wildly impactful for internet users everywhere.

A few things to note:

• The reporter started with Firefox 2.0.
• The reporter was down for about 2.5 months (6/25/08 – 9/9/08)

We can see a BIG spike for the 3.0 launch in June ’08 and a less pronounced jump for the 3.5 release in June ’09. The reports are trending upwards as our daily usage grew over this period, but it would be more telling to look at a direct comparison between daily users and daily broken site reports:

This view of the data conveys some good news. Just looking at the chart, it seems our daily usage and daily broken site reports are diverging – meaning we are getting fewer reports per user on average. Here are two more views revealing this information:

The chart above shows two Ordinary Least Squares (OLS) regression lines tacked onto our original graph. The linear regression lines are clearly diverging – more evidence that we are getting fewer reports per user.

Since July 1st, 2007 our daily reports per 100,000 users figure has decreased from about 2.5 to 2 (a fall of 20%).

I think it is safe to conclude that daily usage is pulling away from daily broken site reports for two main reasons. The first is that the web has improved – websites users navigate to are more likely to function correctly in any browser. The second is that Firefox is a better product today than it was in July 2007. Either way we look at it, the major win is that Firefox users are seeing fewer and fewer broken websites.

In parts II and III of our Reporter analysis (coming soon!), we’ll be looking at the types of problems reported by users, where these problems occur, how problems with websites on Firefox have evolved over time, and what recommendations could be made for improved user experiences moving forward.