Blog of Metrics

We’ve heard quite a bit of anecdotal evidence suggesting that Firefox usage saw an upswing in the frequency of crashes during the second half of 2009.  This experience has generally been tied to Firefox 3.5, which was released in late June.  Ideally, we’d have a lot of really rich data to tell a story around the crashiness of Firefox and the associated improvements that are being made.  While some new data systems are about to hit the scene (see here and here) allowing us to figure out precisely what’s going on, we do have a couple data sets that we can rely on today.

Let’s now try to answer the question, “have Firefox users noticed a decrease in the frequency of Firefox crashes during recent months?”

First, let’s look at the total number of crashes.  Below you’ll see a trended ratio that is total daily crashes (i.e., crash report submissions) divided by total active daily users of Firefox.  For example, 1M instances of Firefox crashes in a day divided by 100M users of Firefox on that day would give a ratio of 1%.  It’s a slight trend, but you can see that since early October, the ratio has decreased.  (Please note that this trended ratio assumes a constant client side throttling percentage.)

Next, let’s look at some slightly more qualitative information – the Firefox Uninstall Survey (data is available here).  For the question, “Why did you uninstall Firefox?”, we can see if there’s any trend in the frequency with which people are selecting “Too many crashes” as an response option.  While it is concerning that this option continues to be the #1 reason cited by users, the trend has been in decline since August.

I want to stress that the data points above merely represent some very early analysis – much more is on the way!  That said, the charts above do suggest, at least directionally, that efforts by the Mozilla community (e.g., CrashKill, Socorro team) have been making an impact in recent months.

Each month nearly 40,000,000 people click on this button within Firefox:

And then land here:

From past analysis, we know there are many unknowns about this user interaction.  For example, what are a user’s expectations upon landing here? and why do so many people instantly leave the page without clicking anything?

To help address some of these unknowns, we recently extended our user outreach efforts to this page, providing users with the option of leaving feedback.  (We previously discussed our current user outreach program in relation to both the Firefox install and update experiences.)  More than 4,000 people have provided feedback over the past two months (to see how, click here and then click the orange feedback button), and below is high-level breakdown of those comments:

Next, and perhaps most importantly, we can also see the “satisfaction” score given by each of the cohorts above (scale is 1 to 5):

Given this information, a couple things seem abundantly clear:

• Of the 40,000,000 people arriving at this page, many are beginner or newbie users of Firefox, or of the internet/computers more generally
• This cohort of people is left feeling dissatisfied with the experience we provide them

So, how can we take these insights and turn them into action?

First, we’ll plan to make this page a part of our optimization roadmap where we’ll be able to experiment with radical new concepts, while also considering how this touch point fits in with the broader Firefox user experience.  In the nearer term, though, we should probably consider some quick and easy wins as a temporary band-aid.  Would adding a Firefox Support related element (example below) suffice?

Like last time, let’s start with the punch line:

• People love Firefox!
• People are also notably frustrated with two experiences – (1) Firefox crashing too often, and (2) Firefox not saving their settings (e.g., home page)

These insights were arrived at via our current user outreach efforts.  When people visit the Firefox “Whatsnew” page – which users automatically hit after each Firefox update – they have the option of providing feedback and comments.  More than 16,000 people have been kind enough to provide feedback over the past six weeks.  If you’re curious how this works from the user’s perspective, click here to the Fx3.5.4 whatsnew page and then click on the orange feedback button.

Here’s a breakdown of what has recently been on the mind of users:

We previously shared analysis of comments from people visiting the Firstrun (installation) page, and the chart above looks somewhat similar.  The fact that so many people made a point of saying something positive is amazing!

On the other hand, there is one negative item that we first took note of last time.  A significant number of people left a comment along the lines of, “why does this page open every time I open Firefox?  Please get rid of it.  I previously set my home page and Firefox won’t save the setting.” On the plus side, we have already taken this insight and turned it into action.  The Firefox team has prioritized a fix via bug 495735, and it should ship it in the next release of Firefox.

Lastly, there was one other group of people who were particularly frustrated.  For each of the groups in the pie chart above, we can also look at their average satisfaction score (scale of 1 to 5):

Fortunately, the crashiness of Firefox is a known issue, and making Firefox less crashy has been a top priority for the Mozilla community in recent months.  Given our current focus in this area, I would guess that if we rerun this same user outreach program a few months from now, we’d see that green pie slice either dramatically shrunk or completely eliminated.

At least in one large region of the world, the answer is “yes”.

The folks at Gemius have been kind enough to aggregate their individual country data (e.g., www.en.ranking.pl/) into a single view across their entire sample – a sample totaling more than 60 Billion page views each month.  For an overview of the various market share providers and their samples, please read here.

We’ll eventually look to expand the conversation around this data, but for now, we’ll highlight just one breathtaking view.  The chart below shows weekly browser market share data since the beginning of 2007 and it includes aggregated data from across nine countries – Czech Republic, Bulgaria, Estonia, Hungary, Lithuania, Latvia, Poland, Russia, and Ukraine.

Let’s start with the punch line:

• People love Firefox!
• There is a small segment of existing users running into a particular problem – Firefox is not saving their preferences

Taking a step back, let’s walk through how we arrived at these findings.  We previously described our efforts integrating user outreach into mozilla.com.  For this post, we’re going to focus just on comments left by users during their visit to the Firefox “First Run” page, which brand new users hit upon downloading/installing Firefox for the first time (analysis of the Whatsnew and Getting Started pages will soon follow).

About 1,200 people left feedback over the past month, and below is a breakdown of the feedback.  If you’re curious how this worked from the user’s perspective, click here to the firstrun page and then click on the orange feedback button.

So, overall, the feedback looks really amazing.  For such a high percentage of people to go out of their way to say something positive is incredible.

On the downside, there were two issues identified by users that we weren’t previously aware of:

1. Many people complained about the open video freezing.  I’m not sure exactly what the issue is, but we can note that the next version of this page (for Fx3.6) will not include the same video.
2. Perhaps more critically, many users talked about their frustration around Firefox not saving their settings.  The most common comment was along the lines of, “why does this page open every time I open Firefox?  Please get rid of it.  I previously set my home page and Firefox won’t save the setting.” In addition, some people noted other preferences/settings not saving – add-ons, personas, favorites and bookmarks.

How do we know that this latter group of users is frustrated?  In addition to reading their actual comments, we can also look at the average satisfaction score (scale of 1 to 5) left by users in each feedback group.  The findings are below… it’s pretty clear that this group, with a satisfaction score of 1.9, has been feeling extremely frustrated.

What’s the good news?

Thanks to this insight, we’ve been able to prioritize a fix via bug 495735, and we’re hoping to ship it in the next release of Firefox!  Thanks to Dietrich Ayala, Mats Palmgren, the Firefox team, and several others for making this happen.

Mozilla recently announced the launch of a “plugin check” page (pictured below), allowing users to see a list of their installed plugins and whether or not any updates are needed.  This comes on the heels of a program where we successfully helped more than 10,000,000 Firefox users with a non-current version of Flash get to Adobe’s download/update page.

Now that this new program has been live for about a week, we thought we’d check the numbers (cumulative since last week):

Considering that this page hasn’t been directly pushed to users, these numbers are significant.  There are a couple spots within mozilla.com where users can click to find this page, but otherwise, our web analytics reporting tells us that about 90% of the traffic has been coming from news and technology related web sites.

Next, you may be wondering where people ventured upon clicking to “Update” or “Research”.  Here’s a breakdown:

We’ll continue to keep an eye on these numbers, and perhaps more excitedly, there are plans to soon integrate this feature into the browser itself.