• Yesterday (just after 3.0.18 and 3.5.8 were released) we saw a spike in fresh downloads/installs of Firefox.  The typical daily number is in the ballpark of 2 Million and yesterday it shot up to over 4 Million.
• As Daniel highlighted, nearly all of the download activity was for Firefox 3.6.
• Digging a little deeper, we also discovered that the entire spike in 3.6 downloads was coming from people on Firefox 3.5.8.  This means people successfully got the update yesterday (3.5.8), and then went out of their way to manually do one further update (i.e., get 3.6).

Why or how is this happening?

It turns out that the answer was right under our nose.   When people get an update, they see an update page.  And if they’re not on the current major version (e.g., 3.6), the page suggests that they go and download the lastest and greatest.   So, what happened within this user interaction yesterday?

So, this explains a common experience for millions of users each time a Firefox update is shipped.  And it’s good to see the messaging on that 3.5.8 update page (and all older update pages) is paying off.  Perhaps we should consider changing the concept of those pages to be even more aggressive in getting people to update to the latest and greatest.

As an early step in that process, we’ve come upon one specific area where being more open should prove helpful.  Working with Seth Bindernagel and the l10n team, we’ve created a set of custom reports that will be provided to Firefox localizers on a regular basis.  The reports were recently launched and Seth has a full write-up highlighting the key components (e.g., locale and geo interactions).  As an example, below is a portion of what pt-PT localizers receive.

Thanks to Pedro Alves and Seth, among many others, for their hard work.

Why is this important?

• It provides users with an opportunity to share their praise/criticism or any other thoughts
• It strongly complements our web analytics efforts (i.e., we’re able to answer the “why” questions in terms of user experiences)
• It ultimately allows the Fx team, marketing team, and many other groups across Mozilla to gain valuable insights and improve the Fx user experience

To accomplish this, we’ve implemented a feedback button/form on three critical touch points at mozilla.com:

Getting Started page

First Run page

Whatsnew page

The feedback form on each page is slightly different (looking at the “categories” and “sub-categories”), depending on a user’s expectation of that page and the typical point in a user’s experience when he/she hits that page.  As an example, below is the form you will see upon hitting the feedback button on the whatsnew/update page.  And like our previous implementations, we’re not 100% sure what feedback to expect, so we’re going to closely monitor user comments and tweak the feedback forms accordingly.

Stay tuned!

You may have noticed that Mozilla Labs implemented its very first experiment for Test Pilot earlier this month.  The goal is to answer the following two questions (or at least put some quantitative framework around the questions):

1. What would be the best default tab behavior after users close a tab?
2. What should be default placement when users open a new tab?

As someone who downloaded Test Pilot and participated in this study, I was impressed by a couple things.  First, the introductory page did a nice job of summarizing what was going on (first image below).  And second, at any time I could view what data was being collected about me (latter image).  At the end of the experiment, it was then my decision whether or not to submit my data.

Moving forward, the metrics team looks forward to partnering the Test Pilot team in making an impact with the resulting data.  Stay tuned!

• 99% of internet users (desktop) have Flash.
• The vast majority of people have an out of date version.  One study claims 80% and mozilla.com’s own traffic stats show about 75% of visitors on a non current version.

So, what has transpired since last Wednesday?

In one week, 10,000,000 people have clicked on the “flash update” link below.

Taking this analysis one step further, we wanted to gain a better sense for users’ interaction with this page.  Breaking down the data by day, we looked more carefully at the en-US version of the 3.5.3 whatsnew (or update) page and pulled the following numbers:

1. How many total people hit the whatsnew page?
2. Of this cohort, how many had an out of date Flash version, and hence, saw the message above?
3. And of this smaller cohort, how many people actually clicked on the flash update link?

Beyond the total impact of 10,000,000 clicks, the most impressive pattern that stands out is the click through rate.  While the Firefox whatsnew page generally sees a click through rate below 5%, the flash update link alone has generated a click through rate north of 30%.  Phenomenal!

(Please ignore the Fx version numbers in this screenshot.  Unfortunately, we don’t have an exact copy on hand.)

For people interested in leaving their feedback, here is the survey they saw:

In total, nearly 5,000 users provided feedback.  Similar to our other recent feedback efforts, we decided to manually read through comments from the second question above in order to extract the most insight and best understand the true pain points and thoughts of these Firefox users.  Sorting through the comments, I noticed about fourteen different types of comments/issues that were cited multiple times… a distribution breakdown and example comments are below.

Location Bar (clearing history) = “There are several issues with Firefox 3, but the browser is not very secure if browsing history cannot be deleted from the location bar. The Tools->Clear Private Data command was no help.”

Speed/Memory Leak Concerns (real or perceived) = “I had heard it made your computer run slowly when it first came out….but after a few months i had forgotten about this tid bit and installed ‘3′. It did indeed bring my computer to a near standstill. My friend did this as well. Another friend then jarred my memory as to why and so…i then un-installed it and re-installed ‘2′ and since then my computer has been running great….so….would you take the chance again?”

Add-ons Compatibility = “There are a number of add-ons which will not work with Firefox 3, Getmail, Silverscreen, Feebe and others.”

Look & Feel (general UI) = “Compared to v2, v3 has an interface that sucks, sucks, sucks big time.  Graphically, it’s much worse: harder to read (too dark, type too small), icons lack color, etc.”

Bookmarks (UI) = “Getting to alphabetized bookmarks with the latest version is a serious pain in the fundament and my wife and I continue to be amazed as to why you decided to take a huge step  backwards with what was supposed to be an advance. Cure this problem and we’ll move up. Thank you.”

Crashes = “Firefox 3 Crashes every time I install it.  Even with a clean upgrade.  It won’t load at all it just crashes restarts and crashes again.”

Lost Stuff when Upgrading = “Last time I took a Firefox upgrade, my entire Favorites list was wiped out.  It would take me more than a year of frustration and work if that happened again.  So, I’m afraid to take another upgrade.”

Developer for Testing = “I am Web developer and have to test my page in all browser. So I am keeping ff2 alongwith ff3 and other browser .”

Don’t Fix What’s not Broken = “I need more time to figure out what to do.  I am happy with all the stuff I have and the status quo.  My computer works well as it is.  Do I really need this update? ”

Fonts (size) = “The window is huge.  the writing is huge.  when I go searching only 2 or 3 websites fit in the window.”

U3 Compatibility = “Firefox 3 does not work on U3 devices in fact it makes the device unusable and so does the latest version of Thunderbird. The U3 device only accepts the mobile versions although I keep getting upgrade versions popping up.”

Cost? = “If you say this is free. I have always heard there is really nothing free in this world.  Times are tough all over, especially for us seniors living on social security.”

Printing Problems = “When printing from FF3, text comes out as scrambled ASCII. doesn’t work on Win(XP/Vista), Linux, etc. Big pain in the butt. I love FF3 otherwise, but as a network admin I just can’t make the switch.”

Enterprise Support = “We can use only software expressly approved and that works together with our network and other older software.  We can’t download or install anything on our own.”

In our next post, we’ll provide some interpretation around these findings and discuss possible implications from a product standpoint.  Stay tuned for Part II!

Lastly, thanks to Sam Sidler for implementing this survey and to Eric Hergenrader for providing much of the analysis.

The metrics and SUMO teams have been working hard to better understand the traffic at support.mozilla.com and users’ experiences there.  One critical item we’ve had trouble wrapping our heads around has been the unusually high bounce rate on the SUMO home page.  Looking strictly at the web analytic numbers, roughly 86% of visitors immediately exit the site.

As a multi-pronged approach to figure out what the heck is going on here, we considered a couple options:

1. Implement a feedback button/form on the home page, allowing users to tell us what is on their mind – more on this below
2. Modify our tracking so that we can see the percentage of visits that are derived from people hitting the F1 key on their keyboard – the SUMO team will soon be discussing the details of this change

For approach #1, we initially implemented a button in the bottom corner of the page (button and feedback form are pictured below).  This page sees nearly 10,000,000 visitors monthly, so we were fairly sure that we would be able to understand some key insights rather quickly.  Unfortunately, events did not play out as expected.  Over the course of nearly a month, we had fewer than 1,000 total responses, which translates to a click-through rate of roughly 0.0001% (ouch!).  To boot, there didn’t appear to be any revealing patterns or insights within the responses.

At this point, we felt bewildered and didn’t know where else to turn.  So, what did we do?  We implemented a pop-up.

Let me stress that I hate pop-ups, hate them.  Given our circumstances, we decided to run the feedback pop-up question for just a 24-hour period and show it to 60% of visitors to the SUMO home page (they saw it when trying to leave the page).

The findings were amazing. Firstly, out of 700 total comments, only one or two people said something negative about the pop-up itself.  Many comments were along the lines of, “Firefox is great” (someone even pointed out that we should have included a positive category within the survey form).  Here’s one example:

“i admire this site and this browser. it is faster, and safer,…….. so many add ons that can use and its fun…… i think this is the best internet browser that i have…. i using ubuntu 9.04 and its work fine…tnx more power to you…. and to your team….”

Secondly, we were able to confirm a hypothesis – nearly half of the traffic to the SUMO home page appears to be accidental (e.g., a user accidentally hitting F1 on his/her keyboard).  The screenshot below comes from Kampyle’s dashboard (they’re the feedback analytics firm who helped us with this project) and it shows a breakdown of people’s feedback by category.  That blue slice of the pie is “Arrived by Mistake”… that’s huge!  Even though it shows as 40%, we’re guessing that the true percentage is closer to 50%, as some folks arriving by mistake likely closed their tab/window within a fraction of a second and missed our feedback question.

[As a side note, this is a perfect example of why pure web analytics – alone – can fail us (analysts, marketers, business managers, etc.).  Without hearing real comments from real people, we could have easily been led down the wrong path by relying solely on the data.]

Lastly, we wanted to look at the exact feedback left by people visiting support.mozilla.com by mistake.  The vast majority indicated that they indeed realized that they had accidentally hit the F1 key.  Diving in a little further, we wanted to share a small sample of actual comments.  Enjoy!

“actually the cat walked on the keyboard”

“bent over to pick something up and accidentally leaned on the f1 key”

“Cat jumped onto laptop”

“CHILD GOT A HOLD OF THE KEYBOARD AND STARTED PUSHING BUTTONS.”

“finger too big.  finger hit f1 by mistake.  stupid bigfinger.”

“i was dusting, its nice to know this site exists.”

“I was stretching in my chair, and next thing I know, I’m on the FF help site. OOPS!!”

“I was using a netbook in the dark and thought I was pressing 1 but I hit F1 by mistake. Aside from that I’ve usually found the firefox support site quite helpful on the rare occasion that I’ve needed it.”

“my 2 year old was playing with the keyboard”

“My 9 month old was playing with my keyboard.”

“My baby likes to grab my computer and press buttons.  She grabbed my keyboard and held the F1 button down so the Support site opened in about nine tabs.”

“My cat stepped on my laptop. ”

“My daughter is always pressing it. I never knew which button it was before now.”

“my fat finger syndrome is acting up again”

“My nails are long and I constantly hit the F1 button.”

“now I know which key my baby hit when he was banging on the keyboard.”

“Oops. Accidentally hit F1, but wow! The support site is great! Lots of info. I may have to come back intentionally!  Thanks for a great product!”

“the key was hit by the telephone”

“Yes, I arrived at this site by pressing F1 key. No matter how I came here but would like to let you know – I love Firefox and thanks for provinding us this great product.”

“cat hit keys”

“Cat on keyboard”

“Dropped food on keyboard”

“my cat hit the help key….lol”

“My cat pressed F1, but it’s cool”

“my pet rat walked over my keyboard and hit the F1 Key LOL – Sorry!”

“No clue–my cat was wandering around and probably stepped on something but it was nice ending up here because I’m seeing all kinds of things that look very interesting.”

“my son was hitting some random keys on my laptop and suddenly I get this.”

For some additional context, the average was around 11% and the U.S. was at 9.6%.

Could state/enterprise adoption be driving what we’re seeing in Vietnam?  Lars Adermalm and several other community members in Vietnam recently pointed us to these two posts (from Jean Chrisophe André and David Trembly’s blog).  Full text in Vietnamese is available here.

By the way, I know how much Chofmann and other folks like this type of report (e.g., we previously looked at Fx2 usage by country).  We hope to have this type of data more easily accessible in the future.

Mozilla has traditionally been very much in-tune with the temperature and feelings of its users, much more so than I’ve seen at other organizations.  However, we’ve sometimes noticed our own blind spots, and over the past year or so, we’ve been working to get on top of our “Voice of the User.”

What do I mean by “Voice of the User”?

Every day, across many avenues, tens of thousands of Firefox users describe a pain point they’ve encountered or comment about their experience with Firefox or make a suggestion.  The sum of all of these voices constitutes what I’m referring to as “Voice of the User”.  To provide more specificity, here is a list of some channels through which Firefox users are speaking to us every day:

• Hendrix
• Crash reporting
• Uninstall Survey
• Report Broken Web Site
• Major Update Survey
• Firefox Support (search, live chat, etc.)
• Bugzilla
• Customer Satisfaction Survey
• Installer Survey
• Web site testing
• webmaster@mozilla.com, webmaster@mozilla.org
• Facebook
• Blogs
• Twitter

This list is not fully exhaustive, but it provides a good overview for how we can think about “Voice of the User” at a high-level.

So, what strides have we made in the past year?  And where do we go from here?  First, let’s look at some of the ground we’ve covered so far (this is in no way meant to capture everything):

• Uninstall Survey – data was analyzed for the first time, a new version was launched, and the current data is now more accessible and usable
• Firefox Support — understands its role in quickly discovering emerging issues affecting users and escalating them, and more work is currently being done along these lines
• Installer Survey — was conceived and launched
• Major Update Survey — was conceived and launched (details coming soon!)
• Report Broken Web Site — we’ll be publishing the first breakdown and analysis of Reporter data this summer
• Bugzilla – there’s a current project underway aimed at answering some key analytical questions related to Bugzilla and providing folks with a bunch more information (e.g., bug burn down rate by product, issues by status and product, average days to resolution by priority and product, open vs close trend by product, etc.)
• Customer Satisfaction Survey – was conceived and launched
• Social media – we’ve started looking at tools allowing us to synthesize user feedback and comments

But we’re not done yet.  We have more work ahead of us and I’m sure we’ll come across more stones unturned.

Taking a step back from all the specific examples just highlighted, how do you think we should be approaching “Voice of the User” more generally?  How can we optimally stay on top of feedback from Firefox users on a daily basis, considering that that feedback comes from a wide range of channels and that it numbers in the tens of thousands (daily)?  Are there broad organizational/community structures or strategies that can help us, i.e., structure/routines/culture that can transform the “Voice of the User” into decisions and actions?

(images from www.journalist.co.uk)

The world of Mozilla includes many Firefox related feedback mechanisms.  Crash reporting, bugzilla and spreadfirefox.com represent a few such examples.  Another feedback mechanism is the Firefox uninstall survey.  When a Firefox user (Windows XP only) decides to uninstall their browser, they’re asked if they want to complete a survey during the uninstall process.

Often a user is merely temporarily uninstalling before immediately reinstalling.  However, in some cases a user enjoys a seemingly less than optimal experience with Firefox, and in these cases, it behooves us a community to figure out what’s going on and if there are true pain points that the user is experiencing (and perhaps that we’re not aware of).

From there, we eventually highlighted some key steps moving forward, including revised questions and design, along with more usable/insightful/actionable data and reporting.  Fast forwarding many months, we finally implemented the new survey this past Friday.

The old survey:

The new survey (with thank-you page):

Here are a few implementation details:

• We’ve embedded a SurveyGizmo survey within a www.mozilla.com page.  This allows us to do two things – (1) update the look and feel of the survey, and (2) have a reporting engine that will be useful and usable.
• Since the survey first came into existence back in 2005 or 2006, all of the data and responses have always been available publicly (at survey.mozilla.com).  The problem was the data wasn’t always fully up-to-date and it wasn’t available in a friendly format.  We’re currently looking at options for making the new reporting from SurveyGizmo both user friendly and publicly available.  More details to come very soon!
• Thanks to Fred Wenzel, Blake Cutler, among many others, for making this happen.

Lastly, let’s not forget why this initiative is so critical.  With tens of thousands of users telling us why they’re uninstalling Firefox, the Firefox team and the broader community should now be able to arrive at several key insights around usage pain points… insights that will ultimately drive user experience improvements… improvements that will eventually impact tens of millions of Firefox users.