Helping People Upgrade to the Latest Version of Firefox

Ken Kovash    plans, results    

We discussed a couple more prominent issues at the time, but when we last analyzed why people cancel out of the Firefox installation process (the actual installer), we found one other significant group of people who could use some assistance – a cohort of existing Firefox users who were confused about what they were doing (in this case, a fresh install) and what they were optimally supposed to be doing (e.g., going to Help -> Check for Updates).

This insight brought me back in time.

When I first started talking with Mozilla folks a few years ago (before joining the org), one of the first things I wanted to do was make sure I was using the latest version of Firefox.  At the time, I went to mozilla.com and tried looking for information about how to do so.  I didn’t find anything and then searched on Google (again, unsuccessfully).  Eventually, I gave up and just did a fresh download and install, all the while not feeling totally sure I was doing the right thing.

Now, armed with our new insight, a couple changes are either on the way or have just been implemented:

• The Firefox team is making one small UI change (mentioned in Rob Strong’s blog post)
• The marketing team changed the main Firefox product page at www.mozilla.com, as seen by existing Firefox users.  Below are the two page variations, and I’ve highlighted the new additions to the pages.  These features point to this newly created FAQ page answering many key questions related to… how do I update/upgrade?  do I need to do a fresh install?  do I need to uninstall first?  what will happen to my bookmarks?  etc.

Thanks to Laura Mesa, John Slater, among many others, for their efforts.  It’s always nice to see our metrics activities come full circle – data -> insights -> actions/changes -> improved user experience!

Why People Don’t Upgrade Their Browser – Part II

Ken Kovash    plans, results    

When we left off, we showed that the #1 reason why people refused the Firefox Major Update offer was their frustration/preference/confusion related to the awesome bar (or location bar).  There are a few things interesting with this piece of feedback and with all the feedback more generally:

1. Virtually everyone who provided feedback had previously used Fx3 and subsequently reverted back to Fx2.  This was 100% descriptive of this cohort and largely described all other users who were kind enough to share their thoughts.
2. Some of the issues cited have since been resolved (since last summer).  Clearly, many users tried Fx3 when it was initially released, then reverted back to Fx2 shortly thereafter, and now they still won’t consider upgrading because of their initial experience (even if that initial issue has since been resolved).  You never get a second chance to make a first impression!

Returning to our cohort of users not fully satisfied with or fully comfortable with the awesome bar, we’ve released a series of enhancements in Fx3.5 that should resolve their concerns.  Alex Faaborg provided an extensive walk-through, and here’s one key section that should help this group of users:

Ability to Control What Appears in the Location Bar Search Results (Proactive Privacy)

When we expanded the capabilities of the location bar to search against all history and bookmarks in Firefox 3, a lot of people contacted us to say that they had certain bookmarks they didn’t really want to have displayed. In some cases users had intentionally hidden these bookmarks in deep hierarchies of folders, somewhat similar to how one might hide a physical object. Having something from your previous browsing displayed to someone else who is using your computer (or even worse) to a large audience of people as you are giving a presentation, is really one of the most embarrassing things that Firefox can do to you. So now in Firefox 3.5, users have complete control over what types of information are displayed in the location bar (or suggestions can be turned off entirely):

So, what actions items can come of this user feedback?

If we think this concern surrounding the location bar is solely limited to existing Fx2 users, we could consider some special messaging for them.  However, I would venture to guess that some small fraction of the Firefox user base currently on either Fx3 or Fx3.5 (93% of all users) share similar thoughts about the location bar.  If so, then it would likely be worthwhile for us consider a couple options:

• Modify Firefox itself.

Below is a simplified mock-up of just one idea I came up with (please note that I don’t know anything about UI).  I’m not sure if the average user knows about Tools->Options, so the concept here is that a person could easily understand how the location bar works and adjust it (according to their preferences) within a fraction of a second.

• Up-level messaging about the location bar and its latest enhancements.

Utilizing key touch points, such as the firstrun page, whatsnew page, or firefox.com page, could help users feel more comfortable with the location bar and its awesome functionality.  For example, is it more important for us to be using these critical touch points to promote open video formats and Fx add-ons, or to highlight Firefox’s most utilized feature and how it can best help a user in his/her everyday life (and not frustrate them)?  At the very least, this seems like a question we should be asking.

What are your thoughts?

More Changes Coming to the Firefox Installer

Ken Kovash    plans, results    

As mentioned a few days ago, we saw a massive impact to the Firefox new user experience with a recently implemented change to the Firefox installer.  A problem previously encountered daily by thousands of new Firefox users completely vanished with a single change.  We now want to share a few more details and suggest one or two additional changes moving forward.

First up, let’s look at the distribution of users’ comments — by step — in the Firefox installation process, i.e., on what step were they on when hitting the “cancel” button (a visualization of these steps is shown here).  The screenshots below come courtesy of Kampyle, the start-up who helped us with this initiative.

THEN (late March):

NOW (last week):

The difference between these two pictures is night and day.  A couple things stand out:

• As mentioned above, our fix successfully resolved the directory problem that users were primary encountering in Step #3 of our installer (“Choose Install Location”)
• There is now a very high concentration of people canceling out of the installer at the final step in the process (Step #6, aka “Summary”)

Looking at the actual comments from users canceling out of the final step, about 75% are related to the zombie process problem (a workaround solution is in the works), about 20% are people expressing their unhappiness over Firefox becoming their default browser, and about 5% fall into an “other” category.  You’ll recall that those two issues were the top issues highlighted by users overall.

What’s up with people commenting on Firefox becoming their default?

It’s a simple option to check or uncheck as desired, right?

Diving into the comments reveals something critical.  All of the people making this comment are doing so on the “Summary” step (or final step, below), not on the earlier “Setup” step (above).  Clearly, many new users are overlooking the checkbox, and just seconds later, are unsure as to how to install Firefox as their non-default browser.  And the outcome here is poor – the person is frustrated and refuses to install Firefox.

What fix should we consider here?  Should we make the checkbox more prominent on the “Setup Type” step?  Should we include instructions (e.g., “simply hit the back button to change”) on the “Summary” step?

Thanks to Rob Strong, the Funnelcake team, and the folks at Kampyle, among many others, for making this ongoing initiative a success.

An Improved Experience for 2,000,000 non-Firefox Users

Ken Kovash    plans, results    

A few months ago, we laid out the following:

“Installing a piece of software can be a challenging process for the typical person to traverse.  For example, with Firefox, we know that each day there are more than 50,000 people who fall into the following group – they complete the download process (receive the full data/file), but don’t then complete their installation of Firefox.”

To help solve this problem, we integrated the following feedback mechanism into the Firefox installer for a single day in late March.

As a result, we arrived at two critical issues that were preventing users from installing Firefox:

1. Users were running into directory issues.  Example comments were along the lines of — “telling me i don’t have access to write to the installation directory… .”  And non-Firefox users, in particular, were experiencing this problem.
2. Firefox not exiting/closing cleanly (e.g., a zombie process).

In response, the Firefox team integrated a fix for Problem #1 into the Fx3.5 installer (released last month).  In short, the installer now requests elevation for non-admin users on Vista and above when UAC is turned on (see bug 404541 for details) and fallback to non-privileged install locations when the user doesn’t have write access to the Program Files directory.

Did the fix work?  Did the modified Firefox installer actually lead to an improved user experience?

Yes.

This past Tuesday, we re-ran the feedback mechanism for a 24-hour period, so that any user canceling out of the Firefox installer was asked if they wanted to leave feedback.  And what did we find?  With the chart below, of the 400+ responses we received, I went through the individual comments left by users and grouped them into 5 different categories.  Problem #1 (noted above) virtually disappeared.  Problem #2 grew its overall share of the pie (compared with last time).

How did we arrive at our estimated impact of 2,000,000 users?  We estimate that roughly 5,000 people were running into Problem #1 – daily.  Multiplied by 365 days, that equates to roughly 2 million people who will be aided by an improved experience over the next year.

Up next on the horizon… the Firefox team is looking into implementing some workaround solutions for Firefox users falling into the blue pie slice above (the timeframe is a couple months out).  Also, with our next post, I’ll dive into the second most popular comment – selecting Firefox as your default browser during the installation process.  Stay tuned!

When a Firefox User Reports a Broken Web Site

ehergenrader    plans, results    

One of the ways Firefox users can easily and directly contribute to making the browser a better product is through our Help->Report Broken Web Site feature:

This allows users to let us know if they experience problems with Firefox on specific websites. Let’s take a high-level look at this “Broken Site Reporter” data over the past few years:

These numbers are significant.  Each day, over a thousand Firefox users are showing us pain points they have experienced while surfing the web. This sort of feedback can be wildly impactful for internet users everywhere.

A few things to note:

• The reporter started with Firefox 2.0.
• The reporter was down for about 2.5 months (6/25/08 – 9/9/08)

We can see a BIG spike for the 3.0 launch in June ’08 and a less pronounced jump for the 3.5 release in June ’09. The reports are trending upwards as our daily usage grew over this period, but it would be more telling to look at a direct comparison between daily users and daily broken site reports:

This view of the data conveys some good news. Just looking at the chart, it seems our daily usage and daily broken site reports are diverging – meaning we are getting fewer reports per user on average. Here are two more views revealing this information:

The chart above shows two Ordinary Least Squares (OLS) regression lines tacked onto our original graph. The linear regression lines are clearly diverging – more evidence that we are getting fewer reports per user.

Since July 1st, 2007 our daily reports per 100,000 users figure has decreased from about 2.5 to 2 (a fall of 20%).

I think it is safe to conclude that daily usage is pulling away from daily broken site reports for two main reasons. The first is that the web has improved – websites users navigate to are more likely to function correctly in any browser. The second is that Firefox is a better product today than it was in July 2007. Either way we look at it, the major win is that Firefox users are seeing fewer and fewer broken websites.

In parts II and III of our Reporter analysis (coming soon!), we’ll be looking at the types of problems reported by users, where these problems occur, how problems with websites on Firefox have evolved over time, and what recommendations could be made for improved user experiences moving forward.

Does Mozilla Champion the Voice of Firefox Users?

Ken Kovash    plans, process    

While presenting at and attending the eMetrics conference a couple months back, I noticed one concept heavily emphasized by several presenters — “Voice of the Customer.”  Several leaders within the analytics world were able to show how they’ve transformed their organizations and industries by allowing “Voice of the Customer” to drive most actions and decision making across their respective enterprises.

Mozilla has traditionally been very much in-tune with the temperature and feelings of its users, much more so than I’ve seen at other organizations.  However, we’ve sometimes noticed our own blind spots, and over the past year or so, we’ve been working to get on top of our “Voice of the User.”

What do I mean by “Voice of the User”?

Every day, across many avenues, tens of thousands of Firefox users describe a pain point they’ve encountered or comment about their experience with Firefox or make a suggestion.  The sum of all of these voices constitutes what I’m referring to as “Voice of the User”.  To provide more specificity, here is a list of some channels through which Firefox users are speaking to us every day:

• Hendrix
• Crash reporting
• Uninstall Survey
• Report Broken Web Site
• Major Update Survey
• Firefox Support (search, live chat, etc.)
• Bugzilla
• Customer Satisfaction Survey
• Installer Survey
• Web site testing
• webmaster@mozilla.com, webmaster@mozilla.org
• Facebook
• Blogs
• Twitter

This list is not fully exhaustive, but it provides a good overview for how we can think about “Voice of the User” at a high-level.

So, what strides have we made in the past year?  And where do we go from here?  First, let’s look at some of the ground we’ve covered so far (this is in no way meant to capture everything):

• Uninstall Survey – data was analyzed for the first time, a new version was launched, and the current data is now more accessible and usable
• Firefox Support — understands its role in quickly discovering emerging issues affecting users and escalating them, and more work is currently being done along these lines
• Installer Survey — was conceived and launched
• Major Update Survey — was conceived and launched (details coming soon!)
• Report Broken Web Site — we’ll be publishing the first breakdown and analysis of Reporter data this summer
• Bugzilla – there’s a current project underway aimed at answering some key analytical questions related to Bugzilla and providing folks with a bunch more information (e.g., bug burn down rate by product, issues by status and product, average days to resolution by priority and product, open vs close trend by product, etc.)
• Customer Satisfaction Survey – was conceived and launched
• Social media – we’ve started looking at tools allowing us to synthesize user feedback and comments

But we’re not done yet.  We have more work ahead of us and I’m sure we’ll come across more stones unturned.

Taking a step back from all the specific examples just highlighted, how do you think we should be approaching “Voice of the User” more generally?  How can we optimally stay on top of feedback from Firefox users on a daily basis, considering that that feedback comes from a wide range of channels and that it numbers in the tens of thousands (daily)?  Are there broad organizational/community structures or strategies that can help us, i.e., structure/routines/culture that can transform the “Voice of the User” into decisions and actions?

(images from www.journalist.co.uk)

Places Stats – Analysis in the Open

Ken Kovash    Uncategorized, plans    

I was excited to see Andy Edmonds’ post yesterday about Places Stats.  It’s not every day that we see someone in the Mozilla community talking about R, a popular statistical package.

Places is described as “a system for storing bookmarks, history, and other user information about the Web, intended to improve developer access to this data and make it easier for people to Bookmark pages.”  So far, several hundred people have volunteered to participate, and a few folks have started to put some framework ideas around analyses that could ultimately lead to useful insights.

I followed up with Andy to ask about his early analysis, and here’s a high-level summary he shared:

• 30% of users use tags heavily.  An additional 20% of users have tried tagging once or twice and didn’t continue for one reason or another.
• Over the years, people seem to be visiting more and more different pages (revisitation rate is decreasing).  This is likely a reflection of the growing content on the web and the growing relevance of the web to more aspects of people’s lives, but is also likely an artifact of the more dynamic nature of the web.

If you’re interested in participating/collaborating in this stats project, check out Andy’s post.

Just Launched – the New Firefox Uninstall Survey

Ken Kovash    plans, process    

When we first started talking about the Firefox Uninstall Survey last year, we introduced it with this:

The world of Mozilla includes many Firefox related feedback mechanisms.  Crash reporting, bugzilla and spreadfirefox.com represent a few such examples.  Another feedback mechanism is the Firefox uninstall survey.  When a Firefox user (Windows XP only) decides to uninstall their browser, they’re asked if they want to complete a survey during the uninstall process.

Often a user is merely temporarily uninstalling before immediately reinstalling.  However, in some cases a user enjoys a seemingly less than optimal experience with Firefox, and in these cases, it behooves us a community to figure out what’s going on and if there are true pain points that the user is experiencing (and perhaps that we’re not aware of).

From there, we eventually highlighted some key steps moving forward, including revised questions and design, along with more usable/insightful/actionable data and reporting.  Fast forwarding many months, we finally implemented the new survey this past Friday.

The old survey:

The new survey (with thank-you page):

Here are a few implementation details:

• We’ve embedded a SurveyGizmo survey within a www.mozilla.com page.  This allows us to do two things – (1) update the look and feel of the survey, and (2) have a reporting engine that will be useful and usable.
• Since the survey first came into existence back in 2005 or 2006, all of the data and responses have always been available publicly (at survey.mozilla.com).  The problem was the data wasn’t always fully up-to-date and it wasn’t available in a friendly format.  We’re currently looking at options for making the new reporting from SurveyGizmo both user friendly and publicly available.  More details to come very soon!
• Thanks to Fred Wenzel, Blake Cutler, among many others, for making this happen.

Lastly, let’s not forget why this initiative is so critical.  With tens of thousands of users telling us why they’re uninstalling Firefox, the Firefox team and the broader community should now be able to arrive at several key insights around usage pain points… insights that will ultimately drive user experience improvements… improvements that will eventually impact tens of millions of Firefox users.

Mozilla as a Data Driven Community

Ken Kovash    plans, process    

I wanted to share my slides from last week’s analytics discussion at the Mozilla all hands meeting.  With those in attendance, we were able to successfully identify several ways in which the metrics team can be working in closer collaboration with other teams across Mozilla.  In addition, here are a few points of context that may not be evident via the slides below:

• Mozilla is already doing a great deal as a community and organization to be data driven in our actions and decision making
• That said, we’re just starting to scratch the surface
• Looking forward, my hope is that folks across Mozilla will rely more often on the analytics/metrics team before decisions are made and early on in initiatives (i.e., not just after the fact).  In other words, the potential here for greater impact (e.g., on the experience of Firefox users) and improved sustainability presents us (all of us) with many great opportunities.

Let me know what you think!

Improvements Have Arrived at Firefox.com

Ken Kovash    plans, results    

Previously, we’ve briefly touched on our long-term roadmap for optimization of Mozilla’s web sites and our users’ experience on them.  There are many facets of “optimization”, from design related decision making processes, to improved user experiences, to improved download conversion rates.  As an early step in our longer term plan, David Rolnitzky recently led an effort tackling one significant low hanging fruit – providing relevant content to visitors of the main Firefox product page based on their browser.

Prior to running this test, we saw about 400,000 daily visits to this page, about 75% of whom were on another browser (e.g., IE) and about 25% of whom were existing Firefox users.  Delivering tailored content to IE visitors and a different set of content to existing Firefox users seems intuitive, right?  Based on David’s findings, visitors to the test pages agreed.  The download conversion rate of IE visitors increased slightly, and the experience of existing Firefox users seemed to improve as well.  The ultimate impact?  An improved web site experience for upwards of two million users (new or existing) annually.

Please check out David’s post and let us know what you think.