Please reach out to us if you (or someone you know) has a passion for data and building a better internet.

• Analytics Engineer – Data Visualization
• Metrics Software Engineer
• Data Analyst – User Experience
  

The Mozilla community itself is also growing – so if data isn’t your thing, be sure to check out the other career listings as well!

We look forward to soon sharing the methodology, data, results, and implications from this latest experiment.  Stay tuned!

(image attributable to http://www.flickr.com/photos/sarahbelle1/ under a creative commons license.)

In the past, soliciting feedback from users has helped us understand why users were opting to not upgrade, so this time around, people who clicked on the “No thanks” option above were directed to a survey. Over 40,000 people were kind enough to share their thoughts with us.

So what do the results look like this time?

Let’s start by taking a look at responses to the first question:

Over half of the respondents stated that they were simply content with Firefox 3. And compared to what we saw last survey, far fewer users selected “Other reason.” Still, a quarter of users took the time to give an alternative explanation, so exploring these answers can definitely help us gain additional insight. A list of the most common phrases entered into the “Other reason” text field is presented below:

Two things immediately jump out:

• First, confusion over cost has virtually disappeared — this had been a problem previously.
• Second, the vast majority of comments are now about a lack of time.

The lack of comments about costs is a pleasant surprise. Acting on feedback from the last survey, we revised the update prompt, highlighting that the upgrade is free in two places. It seems that this simple addition was enough to eliminate most of the confusion.

The update prompt itself may also partly account for the rise in “no time” comments. With prior upgrade initiatives, users saw an advertisement and button similar to what you see below:

Users could then click “Get the new version” to initiate a 30-second update procedure. This time, however, instead of seeing a typical software dialog box, people encountered the update prompt via the Firefox “Whats new” page and were presented with the customary green Firefox download button.  Hitting the button also initiated the update, but via the more involved process of downloading and installing a fresh version of the browser.

Pushing out the update in this way was a one-time situation, and we’ll revert to the usual process and software dialog box in the future. To alleviate this issue further, we should also perhaps add to the prompt some indication that updating Firefox is relatively quick and painless.

Next, lets turn to Question 2. This was a free form text box so we manually read through a random sample of 10% of the responses and parsed them into categories.

Firefox 3.6’s compatibility, both with add-ons and with specific websites/applications, remains a key issue (although user perception might be playing a small role as well). On the other hand, general stability and performance issues are cited considerably less often: crash and speed comments together account for 11% of responses, down from the 25% we saw last survey.

Our two main insights from above are also evident here. Cost comments have essentially disappeared from Question 2 as well, although they previously made up 7% of responses. And again, time concerns have become a real sticking point for users.

The rise of these “no time” responses, coupled with the still considerable “too many updates” category, has triggered suggestions that extend beyond prompt and wording revisions. For example, some propose that updates occur automatically in the background unbeknownst to the user (à la chrome), perhaps with an easy option to downgrade (unlike chrome).

Clearly, we are a long ways from making any major changes to the update process, but we’d love to hear your thoughts on any of this!

Despite all of our efforts, there are always more improvements for us to strive towards.  For example, the Metrics team has started taking notice of one area that has seemingly been a blind spot in the past:

• What happens to people in the moments between landing on firefox.com and seeing the download button?  (Blake will soon outline more specificity around this question.)
• And what happens to people in the moments between clicking the download button and actually completing the download process (i.e., getting the full file)?

We’ll plan to discuss these questions and user experiences in much greater detail in follow-up posts.  The key takeaway for now is that we believe there are additional “funnel” improvements to be made – improvements that can easily translate into millions of additional happy new users of Firefox.

(image attributable to http://www.flickr.com/photos/ivanteage/ under a creative commons license.)

How do we know this?

Last week, we re-ran our installer feedback mechanism for a short period of time.  If a user clicked “cancel” while walking through the Firefox installer, they were asked if they wanted to provide feedback.

After making an initial round of product improvements based on our first time feedback (March ’09), here are the transformed feedback results from our more recent efforts (both July 2009 and last week):

While we still have plans to tackle the remaining big slice of the pie (see concluding paragraph), we were able to successfully solve the red and green pie slices from last time.  In our latest feedback results (pie on the right), the big pie slice now represents nearly 100% of the total feedback (the previously seen categories virtually evaporated).  One way to interpret this is that we’ve now successfully identified and resolved 3 of the top 4 issues originally encountered by users.

Here were the specific actions we took addressing those red and green slices (details are in bug 508684):

Don’t Want Firefox as Default

People indicating this issue were missing the selection option earlier in the installation process, arrived at the end, and mistakenly believed that we were making Firefox their default without being given a choice.  So, we added the choice to the final step in the installer:

Confusion About Updating-Upgrading-Installing

We did a few different things to help address this area of confusion.  First, we added content to mozilla.com and prominently displayed it on the main Firefox product pages seen by existing users:

Second, within the Firefox installer, we changed the Install button to say “Upgrade” instead of “Install”:

Thanks to Rob Strong, the Firefox team, the Funnelcake team, John Slater, and Laura Mesa, among others, for implementing the changes highlighted above.

Lastly, there remains one outstanding problem for installers of Firefox – “it tells me to close Fx, but it’s not open” (the big pie slice in the charts above).  This issue affects people who already have Firefox and are attempting to reinstall it, and as we’ve noted previously, this cohort becomes fairly frustrated during the experience.    Some fixes are starting to be contemplated (e.g., bugs 496207, 544356)… and I’ll make sure to talk more here once some progress is made.

As a side benefit to this initiative, we also took the opportunity to see what feedback people had, specifically asking users to tell us what was on their mind if they were choosing not to upgrade.  We did this once before when upgrading users from Fx2 to Fx3, and the results were extremely impactful, so we wanted to continue this once again as part of our broader user outreach efforts.

For people interesting in leaving feedback, here is the survey they saw:

A little more than 5,000 people were kind enough to share their thoughts.  In turning to the results, let’s start with question #1:

The most surprising insight above is that 53% of respondents selected the “Other reason” check box.  That suggests that the proposed answers we listed were found somewhat unsatisfactory and that people had other ideas on their mind.  Clearly, understanding “Other reason” and seeing what users said within question #2 should provide us with much more insight than the chart above.

Below is a list of the most common phrases people typed into the “Other reason” box.  What’s most surprising here?  The vast majority of comments are about cost, i.e., “is this upgrade free?”.  Taking this insight and turning it into action, we’re planning to make clear that “Firefox is free” within future upgrade prompts/advertisements.

Next, let’s look at how people responded to question #2.  It’s a free form text box, so we manually read through comments, sorting them into different categories:

One easy way to interpret this pie chart is to compare it with what we saw last time (when users were upgrading from Fx2 to Fx3).  UI related comments have almost vanished.  On the other hand, add-on and extension compatibility comments (Norton was far and away the #1 cited) and crash comments have both risen dramatically as their total share of the pie.  For “not compatible with specific website”, Facebook and specific Google pages (e.g., calendar) seemed to be the most frequently mentioned.

Moving forward, it will be critical that we acknowledge and address the concerns faced by these users.  We’ve been working hard in recent months to reduce the crashiness of Firefox, and some positive results are already evident.  For addressing the add-on and toolbar compatibility issues, the Firefox Support team has been raising the visibility of the top extension issues cited by users (Norton, Roboform, etc.).

And how do we properly communicate all of this the next time we advertise a major upgrade?  We currently highlight these three bullet points:

• Twice as fast as Firefox 3.
• Private browsing, tear-off tabs and more.
• The most advanced Firefox yet.

The last two should probably be changed to “This upgrade is free” and “Improved stability, fewer crashes.”  Your thoughts?

Let’s now try to answer the question, “have Firefox users noticed a decrease in the frequency of Firefox crashes during recent months?”

First, let’s look at the total number of crashes.  Below you’ll see a trended ratio that is total daily crashes (i.e., crash report submissions) divided by total active daily users of Firefox.  For example, 1M instances of Firefox crashes in a day divided by 100M users of Firefox on that day would give a ratio of 1%.  It’s a slight trend, but you can see that since early October, the ratio has decreased.  (Please note that this trended ratio assumes a constant client side throttling percentage.)

Next, let’s look at some slightly more qualitative information – the Firefox Uninstall Survey (data is available here).  For the question, “Why did you uninstall Firefox?”, we can see if there’s any trend in the frequency with which people are selecting “Too many crashes” as an response option.  While it is concerning that this option continues to be the #1 reason cited by users, the trend has been in decline since August.

I want to stress that the data points above merely represent some very early analysis – much more is on the way!  That said, the charts above do suggest, at least directionally, that efforts by the Mozilla community (e.g., CrashKill, Socorro team) have been making an impact in recent months.

And then land here:

From past analysis, we know there are many unknowns about this user interaction.  For example, what are a user’s expectations upon landing here? and why do so many people instantly leave the page without clicking anything?

To help address some of these unknowns, we recently extended our user outreach efforts to this page, providing users with the option of leaving feedback.  (We previously discussed our current user outreach program in relation to both the Firefox install and update experiences.)  More than 4,000 people have provided feedback over the past two months (to see how, click here and then click the orange feedback button), and below is high-level breakdown of those comments:

Next, and perhaps most importantly, we can also see the “satisfaction” score given by each of the cohorts above (scale is 1 to 5):

Given this information, a couple things seem abundantly clear:

• Of the 40,000,000 people arriving at this page, many are beginner or newbie users of Firefox, or of the internet/computers more generally
• This cohort of people is left feeling dissatisfied with the experience we provide them

So, how can we take these insights and turn them into action?

First, we’ll plan to make this page a part of our optimization roadmap where we’ll be able to experiment with radical new concepts, while also considering how this touch point fits in with the broader Firefox user experience.  In the nearer term, though, we should probably consider some quick and easy wins as a temporary band-aid.  Would adding a Firefox Support related element (example below) suffice?

• People love Firefox!
• People are also notably frustrated with two experiences – (1) Firefox crashing too often, and (2) Firefox not saving their settings (e.g., home page)

These insights were arrived at via our current user outreach efforts.  When people visit the Firefox “Whatsnew” page – which users automatically hit after each Firefox update – they have the option of providing feedback and comments.  More than 16,000 people have been kind enough to provide feedback over the past six weeks.  If you’re curious how this works from the user’s perspective, click here to the Fx3.5.4 whatsnew page and then click on the orange feedback button.

Here’s a breakdown of what has recently been on the mind of users:

We previously shared analysis of comments from people visiting the Firstrun (installation) page, and the chart above looks somewhat similar.  The fact that so many people made a point of saying something positive is amazing!

On the other hand, there is one negative item that we first took note of last time.  A significant number of people left a comment along the lines of, “why does this page open every time I open Firefox?  Please get rid of it.  I previously set my home page and Firefox won’t save the setting.” On the plus side, we have already taken this insight and turned it into action.  The Firefox team has prioritized a fix via bug 495735, and it should ship it in the next release of Firefox.

Lastly, there was one other group of people who were particularly frustrated.  For each of the groups in the pie chart above, we can also look at their average satisfaction score (scale of 1 to 5):

Fortunately, the crashiness of Firefox is a known issue, and making Firefox less crashy has been a top priority for the Mozilla community in recent months.  Given our current focus in this area, I would guess that if we rerun this same user outreach program a few months from now, we’d see that green pie slice either dramatically shrunk or completely eliminated.

The folks at Gemius have been kind enough to aggregate their individual country data (e.g., www.en.ranking.pl/) into a single view across their entire sample – a sample totaling more than 60 Billion page views each month.  For an overview of the various market share providers and their samples, please read here.

We’ll eventually look to expand the conversation around this data, but for now, we’ll highlight just one breathtaking view.  The chart below shows weekly browser market share data since the beginning of 2007 and it includes aggregated data from across nine countries – Czech Republic, Bulgaria, Estonia, Hungary, Lithuania, Latvia, Poland, Russia, and Ukraine.