The folks at Gemius have been kind enough to aggregate their individual country data (e.g., www.en.ranking.pl/) into a single view across their entire sample – a sample totaling more than 60 Billion page views each month.  For an overview of the various market share providers and their samples, please read here.

We’ll eventually look to expand the conversation around this data, but for now, we’ll highlight just one breathtaking view.  The chart below shows weekly browser market share data since the beginning of 2007 and it includes aggregated data from across nine countries – Czech Republic, Bulgaria, Estonia, Hungary, Lithuania, Latvia, Poland, Russia, and Ukraine.

• People love Firefox!
• There is a small segment of existing users running into a particular problem – Firefox is not saving their preferences

Taking a step back, let’s walk through how we arrived at these findings.  We previously described our efforts integrating user outreach into mozilla.com.  For this post, we’re going to focus just on comments left by users during their visit to the Firefox “First Run” page, which brand new users hit upon downloading/installing Firefox for the first time (analysis of the Whatsnew and Getting Started pages will soon follow).

About 1,200 people left feedback over the past month, and below is a breakdown of the feedback.  If you’re curious how this worked from the user’s perspective, click here to the firstrun page and then click on the orange feedback button.

So, overall, the feedback looks really amazing.  For such a high percentage of people to go out of their way to say something positive is incredible.

On the downside, there were two issues identified by users that we weren’t previously aware of:

1. Many people complained about the open video freezing.  I’m not sure exactly what the issue is, but we can note that the next version of this page (for Fx3.6) will not include the same video.
2. Perhaps more critically, many users talked about their frustration around Firefox not saving their settings.  The most common comment was along the lines of, “why does this page open every time I open Firefox?  Please get rid of it.  I previously set my home page and Firefox won’t save the setting.” In addition, some people noted other preferences/settings not saving – add-ons, personas, favorites and bookmarks.

How do we know that this latter group of users is frustrated?  In addition to reading their actual comments, we can also look at the average satisfaction score (scale of 1 to 5) left by users in each feedback group.  The findings are below… it’s pretty clear that this group, with a satisfaction score of 1.9, has been feeling extremely frustrated.

What’s the good news?

Thanks to this insight, we’ve been able to prioritize a fix via bug 495735, and we’re hoping to ship it in the next release of Firefox!  Thanks to Dietrich Ayala, Mats Palmgren, the Firefox team, and several others for making this happen.

Now that this new program has been live for about a week, we thought we’d check the numbers (cumulative since last week):

Considering that this page hasn’t been directly pushed to users, these numbers are significant.  There are a couple spots within mozilla.com where users can click to find this page, but otherwise, our web analytics reporting tells us that about 90% of the traffic has been coming from news and technology related web sites.

Next, you may be wondering where people ventured upon clicking to “Update” or “Research”.  Here’s a breakdown:

We’ll continue to keep an eye on these numbers, and perhaps more excitedly, there are plans to soon integrate this feature into the browser itself.

Why is this important?

• It provides users with an opportunity to share their praise/criticism or any other thoughts
• It strongly complements our web analytics efforts (i.e., we’re able to answer the “why” questions in terms of user experiences)
• It ultimately allows the Fx team, marketing team, and many other groups across Mozilla to gain valuable insights and improve the Fx user experience

To accomplish this, we’ve implemented a feedback button/form on three critical touch points at mozilla.com:

Getting Started page

First Run page

Whatsnew page

The feedback form on each page is slightly different (looking at the “categories” and “sub-categories”), depending on a user’s expectation of that page and the typical point in a user’s experience when he/she hits that page.  As an example, below is the form you will see upon hitting the feedback button on the whatsnew/update page.  And like our previous implementations, we’re not 100% sure what feedback to expect, so we’re going to closely monitor user comments and tweak the feedback forms accordingly.

Stay tuned!

You may have noticed that Mozilla Labs implemented its very first experiment for Test Pilot earlier this month.  The goal is to answer the following two questions (or at least put some quantitative framework around the questions):

1. What would be the best default tab behavior after users close a tab?
2. What should be default placement when users open a new tab?

As someone who downloaded Test Pilot and participated in this study, I was impressed by a couple things.  First, the introductory page did a nice job of summarizing what was going on (first image below).  And second, at any time I could view what data was being collected about me (latter image).  At the end of the experiment, it was then my decision whether or not to submit my data.

Moving forward, the metrics team looks forward to partnering the Test Pilot team in making an impact with the resulting data.  Stay tuned!

This insight brought me back in time.

When I first started talking with Mozilla folks a few years ago (before joining the org), one of the first things I wanted to do was make sure I was using the latest version of Firefox.  At the time, I went to mozilla.com and tried looking for information about how to do so.  I didn’t find anything and then searched on Google (again, unsuccessfully).  Eventually, I gave up and just did a fresh download and install, all the while not feeling totally sure I was doing the right thing.

Now, armed with our new insight, a couple changes are either on the way or have just been implemented:

• The Firefox team is making one small UI change (mentioned in Rob Strong’s blog post)
• The marketing team changed the main Firefox product page at www.mozilla.com, as seen by existing Firefox users.  Below are the two page variations, and I’ve highlighted the new additions to the pages.  These features point to this newly created FAQ page answering many key questions related to… how do I update/upgrade?  do I need to do a fresh install?  do I need to uninstall first?  what will happen to my bookmarks?  etc.

Thanks to Laura Mesa, John Slater, among many others, for their efforts.  It’s always nice to see our metrics activities come full circle – data -> insights -> actions/changes -> improved user experience!

1. Virtually everyone who provided feedback had previously used Fx3 and subsequently reverted back to Fx2.  This was 100% descriptive of this cohort and largely described all other users who were kind enough to share their thoughts.
2. Some of the issues cited have since been resolved (since last summer).  Clearly, many users tried Fx3 when it was initially released, then reverted back to Fx2 shortly thereafter, and now they still won’t consider upgrading because of their initial experience (even if that initial issue has since been resolved).  You never get a second chance to make a first impression!

Returning to our cohort of users not fully satisfied with or fully comfortable with the awesome bar, we’ve released a series of enhancements in Fx3.5 that should resolve their concerns.  Alex Faaborg provided an extensive walk-through, and here’s one key section that should help this group of users:

Ability to Control What Appears in the Location Bar Search Results (Proactive Privacy)

When we expanded the capabilities of the location bar to search against all history and bookmarks in Firefox 3, a lot of people contacted us to say that they had certain bookmarks they didn’t really want to have displayed. In some cases users had intentionally hidden these bookmarks in deep hierarchies of folders, somewhat similar to how one might hide a physical object. Having something from your previous browsing displayed to someone else who is using your computer (or even worse) to a large audience of people as you are giving a presentation, is really one of the most embarrassing things that Firefox can do to you. So now in Firefox 3.5, users have complete control over what types of information are displayed in the location bar (or suggestions can be turned off entirely):

So, what actions items can come of this user feedback?

If we think this concern surrounding the location bar is solely limited to existing Fx2 users, we could consider some special messaging for them.  However, I would venture to guess that some small fraction of the Firefox user base currently on either Fx3 or Fx3.5 (93% of all users) share similar thoughts about the location bar.  If so, then it would likely be worthwhile for us consider a couple options:

• Modify Firefox itself.

Below is a simplified mock-up of just one idea I came up with (please note that I don’t know anything about UI).  I’m not sure if the average user knows about Tools->Options, so the concept here is that a person could easily understand how the location bar works and adjust it (according to their preferences) within a fraction of a second.

• Up-level messaging about the location bar and its latest enhancements.

Utilizing key touch points, such as the firstrun page, whatsnew page, or firefox.com page, could help users feel more comfortable with the location bar and its awesome functionality.  For example, is it more important for us to be using these critical touch points to promote open video formats and Fx add-ons, or to highlight Firefox’s most utilized feature and how it can best help a user in his/her everyday life (and not frustrate them)?  At the very least, this seems like a question we should be asking.

What are your thoughts?

First up, let’s look at the distribution of users’ comments — by step — in the Firefox installation process, i.e., on what step were they on when hitting the “cancel” button (a visualization of these steps is shown here).  The screenshots below come courtesy of Kampyle, the start-up who helped us with this initiative.

THEN (late March):

NOW (last week):

The difference between these two pictures is night and day.  A couple things stand out:

• As mentioned above, our fix successfully resolved the directory problem that users were primary encountering in Step #3 of our installer (”Choose Install Location”)
• There is now a very high concentration of people canceling out of the installer at the final step in the process (Step #6, aka “Summary”)

Looking at the actual comments from users canceling out of the final step, about 75% are related to the zombie process problem (a workaround solution is in the works), about 20% are people expressing their unhappiness over Firefox becoming their default browser, and about 5% fall into an “other” category.  You’ll recall that those two issues were the top issues highlighted by users overall.

What’s up with people commenting on Firefox becoming their default?

It’s a simple option to check or uncheck as desired, right?

Diving into the comments reveals something critical.  All of the people making this comment are doing so on the “Summary” step (or final step, below), not on the earlier “Setup” step (above).  Clearly, many new users are overlooking the checkbox, and just seconds later, are unsure as to how to install Firefox as their non-default browser.  And the outcome here is poor – the person is frustrated and refuses to install Firefox.

What fix should we consider here?  Should we make the checkbox more prominent on the “Setup Type” step?  Should we include instructions (e.g., “simply hit the back button to change”) on the “Summary” step?

Thanks to Rob Strong, the Funnelcake team, and the folks at Kampyle, among many others, for making this ongoing initiative a success.

Installing a piece of software can be a challenging process for the typical person to traverse.  For example, with Firefox, we know that each day there are more than 50,000 people who fall into the following group – they complete the download process (receive the full data/file), but don’t then complete their installation of Firefox.

To help solve this problem, we integrated the following feedback mechanism into the Firefox installer for a single day in late March.

As a result, we arrived at two critical issues that were preventing users from installing Firefox:

1. Users were running into directory issues.  Example comments were along the lines of — “telling me i don’t have access to write to the installation directory… .”  And non-Firefox users, in particular, were experiencing this problem.
2. Firefox not exiting/closing cleanly (e.g., a zombie process).

In response, the Firefox team integrated a fix for Problem #1 into the Fx3.5 installer (released last month).  In short, the installer now requests elevation for non-admin users on Vista and above when UAC is turned on (see bug 404541 for details) and fallback to non-privileged install locations when the user doesn’t have write access to the Program Files directory.

Did the fix work?  Did the modified Firefox installer actually lead to an improved user experience?

Yes.

This past Tuesday, we re-ran the feedback mechanism for a 24-hour period, so that any user canceling out of the Firefox installer was asked if they wanted to leave feedback.  And what did we find?  With the chart below, of the 400+ responses we received, I went through the individual comments left by users and grouped them into 5 different categories.  Problem #1 (noted above) virtually disappeared.  Problem #2 grew its overall share of the pie (compared with last time).

How did we arrive at our estimated impact of 2,000,000 users?  We estimate that roughly 5,000 people were running into Problem #1 – daily.  Multiplied by 365 days, that equates to roughly 2 million people who will be aided by an improved experience over the next year.

Up next on the horizon… the Firefox team is looking into implementing some workaround solutions for Firefox users falling into the blue pie slice above (the timeframe is a couple months out).  Also, with our next post, I’ll dive into the second most popular comment – selecting Firefox as your default browser during the installation process.  Stay tuned!

This allows users to let us know if they experience problems with Firefox on specific websites. Let’s take a high-level look at this “Broken Site Reporter” data over the past few years:

These numbers are significant.  Each day, over a thousand Firefox users are showing us pain points they have experienced while surfing the web. This sort of feedback can be wildly impactful for internet users everywhere.

A few things to note:

• The reporter started with Firefox 2.0.
• The reporter was down for about 2.5 months (6/25/08 – 9/9/08)

We can see a BIG spike for the 3.0 launch in June ’08 and a less pronounced jump for the 3.5 release in June ’09. The reports are trending upwards as our daily usage grew over this period, but it would be more telling to look at a direct comparison between daily users and daily broken site reports:

This view of the data conveys some good news. Just looking at the chart, it seems our daily usage and daily broken site reports are diverging – meaning we are getting fewer reports per user on average. Here are two more views revealing this information:

The chart above shows two Ordinary Least Squares (OLS) regression lines tacked onto our original graph. The linear regression lines are clearly diverging – more evidence that we are getting fewer reports per user.

Since July 1st, 2007 our daily reports per 100,000 users figure has decreased from about 2.5 to 2 (a fall of 20%).

I think it is safe to conclude that daily usage is pulling away from daily broken site reports for two main reasons. The first is that the web has improved – websites users navigate to are more likely to function correctly in any browser. The second is that Firefox is a better product today than it was in July 2007. Either way we look at it, the major win is that Firefox users are seeing fewer and fewer broken websites.

In parts II and III of our Reporter analysis (coming soon!), we’ll be looking at the types of problems reported by users, where these problems occur, how problems with websites on Firefox have evolved over time, and what recommendations could be made for improved user experiences moving forward.