A few months ago, we laid out the following:
“Installing a piece of software can be a challenging process for the typical person to traverse. For example, with Firefox, we know that each day there are more than 50,000 people who fall into the following group – they complete the download process (receive the full data/file), but don’t then complete their installation of Firefox.”
To help solve this problem, we integrated the following feedback mechanism into the Firefox installer for a single day in late March.
As a result, we arrived at two critical issues that were preventing users from installing Firefox:
In response, the Firefox team integrated a fix for Problem #1 into the Fx3.5 installer (released last month). In short, the installer now requests elevation for non-admin users on Vista and above when UAC is turned on (see bug 404541 for details) and fallback to non-privileged install locations when the user doesn’t have write access to the Program Files directory.
Did the fix work? Did the modified Firefox installer actually lead to an improved user experience?
Yes.
This past Tuesday, we re-ran the feedback mechanism for a 24-hour period, so that any user canceling out of the Firefox installer was asked if they wanted to leave feedback. And what did we find? With the chart below, of the 400+ responses we received, I went through the individual comments left by users and grouped them into 5 different categories. Problem #1 (noted above) virtually disappeared. Problem #2 grew its overall share of the pie (compared with last time).
How did we arrive at our estimated impact of 2,000,000 users? We estimate that roughly 5,000 people were running into Problem #1 – daily. Multiplied by 365 days, that equates to roughly 2 million people who will be aided by an improved experience over the next year.
Up next on the horizon… the Firefox team is looking into implementing some workaround solutions for Firefox users falling into the blue pie slice above (the timeframe is a couple months out). Also, with our next post, I’ll dive into the second most popular comment – selecting Firefox as your default browser during the installation process. Stay tuned!
In Part I of our Broken Web Site Reporter analysis here, we looked at some very general data patterns concerning usage of our Report Broken Web Site tool in the Firefox Help menu. Here I will attempt to dive a little further into users’ responses, examining the Problem Type category from the data. When the user reports a broken website, he/she is given nine different problems to choose from:
These categories help us determine what pain points Firefox users experience most often while surfing the web. Let’s take a look at the distribution of of these since the reporter was launched three years ago:
A first observation — Disability Access has consistently been our top reported website problem. Browsing the comments associated with this category finds many of these reports refer to “Server not found” or “Connection Timed Out” errors. The Disability Access category is meant for websites incompatible with some sort of physical disability, like blindness. It seems from these comments that the name of this category may be confusing or might overlap with some of our other categories.
Taking a look at the Appearance Wrong category, we can see that it has slowly descended from the top reported problem to the fourth from the top. Along with the work done in Firefox 3.0 and 3.5 to display graphics correctly, this decrease has followed a strong increase in the Printed Output is Wrong category introduced in 3.0. It is possible that this newer category is cannibalizing reports from the Appearance Wrong problem type due to their similar connotations. If the printed output in a website is wrong, many users may choose the Appearance Wrong option and vice versa.
Over this time period we can see a relatively linear increase in the Can’t Log In category. Whereas most problem types saw slower growth after the 3.0 launch (as evidenced by their increased flatness), Can’t Log In looks to be growing at about the same rate. It remains to be seen whether or not this trend will continue with 3.5.
What conclusions can we glean from the problem type trends? The Can’t Log In category seems to be the most worrisome. It has been growing steadily since Firefox 2.0, and there is little reason to believe users are confusing Can’t Log In with any of the other categories. Taking a look at specific user comments yields the same results – 35% of en-US users had the word “log in” or a variation in their report. We now know our naming scheme for these reporter problems is flawed; Disability Access sees far too many problems reported, and Appearance Wrong and Printed Output is Wrong are probably feeding off each others’ reports.
Next up in Part III we’ll be looking at the reporter data by locale. Stay tuned!
One of the ways Firefox users can easily and directly contribute to making the browser a better product is through our Help->Report Broken Web Site feature:
This allows users to let us know if they experience problems with Firefox on specific websites. Let’s take a high-level look at this “Broken Site Reporter” data over the past few years:
These numbers are significant. Each day, over a thousand Firefox users are showing us pain points they have experienced while surfing the web. This sort of feedback can be wildly impactful for internet users everywhere.
A few things to note:
We can see a BIG spike for the 3.0 launch in June ’08 and a less pronounced jump for the 3.5 release in June ’09. The reports are trending upwards as our daily usage grew over this period, but it would be more telling to look at a direct comparison between daily users and daily broken site reports:
This view of the data conveys some good news. Just looking at the chart, it seems our daily usage and daily broken site reports are diverging – meaning we are getting fewer reports per user on average. Here are two more views revealing this information:
The chart above shows two Ordinary Least Squares (OLS) regression lines tacked onto our original graph. The linear regression lines are clearly diverging – more evidence that we are getting fewer reports per user.
Since July 1st, 2007 our daily reports per 100,000 users figure has decreased from about 2.5 to 2 (a fall of 20%).
I think it is safe to conclude that daily usage is pulling away from daily broken site reports for two main reasons. The first is that the web has improved – websites users navigate to are more likely to function correctly in any browser. The second is that Firefox is a better product today than it was in July 2007. Either way we look at it, the major win is that Firefox users are seeing fewer and fewer broken websites.
In parts II and III of our Reporter analysis (coming soon!), we’ll be looking at the types of problems reported by users, where these problems occur, how problems with websites on Firefox have evolved over time, and what recommendations could be made for improved user experiences moving forward.
How would you devise a marketing plan for Mozilla?
This was the business problem that Jakob Marovt and a group of other students recently tried tackling. Jakob is a student in Slovenia at the University of Ljubljana, Faculty of Computer and Information Science. His Mozilla project came about earlier this year while studying abroad at University of Southern Denmark, Faculty of Engineering, department of manufacturing and management.
I recently came across Jakob’s weekly presentations and final plan and I was impressed. It’s not often that a student project leads to such a thorough and accurate understanding of an organization/community.
I followed up with Jakob and asked him about the biggest challenge they encountered or key insight they arrived at. Given that their assignment called for utilizing frameworks/strategies/tactics within traditional marketing theory (porters five forces, swot analysis, etc.), Jakob noted that one challenge was having to fit Mozilla’s recommended actions within a more mass media and less internet/social media approach.
Great job!
Last week, we looked at which regions have shown the fastest and slowest adoption rates of Firefox 3.5. Asa had a great idea to expand on this analysis – why not visualize it through a global map to more easily see if there are any overarching patterns across particular regions?
So, we took our Firefox usage data from the past week, uploaded it to IBM’s manyeyes project (which we previously mentioned here), and created the following visualization. The image below shows Fx3.5 adoption by region. You’ll notice that there seems to be relatively more dark shaded areas around Russia and Asia.
If you want to explore this visualization further, you can visit the manyeyes page or simply click on the “interact” button below. You’ll notice that with the drop-down menu near the upper left-hand corner, you can select the Firefox version — Fx2, Fx3.0, or Fx3.5 — you’re interested in viewing stats for. This should also come in handy for detecting regions where Fx2 usage is still significant (which we previously explored here).
Would you like to see this type of data updated regularly? Would seeing it trended over time be helpful? Let us know.
With the recent launch of Fx3.5, we wanted to take a quick look at adoption patterns across the globe. With the chart below, we took the number of Fx3.5 users in each region over the past week and divided it by the total number of Firefox users (all versions) in each region over the past week.
For some additional context, the average was around 11% and the U.S. was at 9.6%.
Could state/enterprise adoption be driving what we’re seeing in Vietnam? Lars Adermalm and several other community members in Vietnam recently pointed us to these two posts (from Jean Chrisophe André and David Trembly’s blog). Full text in Vietnamese is available here.
By the way, I know how much Chofmann and other folks like this type of report (e.g., we previously looked at Fx2 usage by country). We hope to have this type of data more easily accessible in the future.
While presenting at and attending the eMetrics conference a couple months back, I noticed one concept heavily emphasized by several presenters — “Voice of the Customer.” Several leaders within the analytics world were able to show how they’ve transformed their organizations and industries by allowing “Voice of the Customer” to drive most actions and decision making across their respective enterprises.
Mozilla has traditionally been very much in-tune with the temperature and feelings of its users, much more so than I’ve seen at other organizations. However, we’ve sometimes noticed our own blind spots, and over the past year or so, we’ve been working to get on top of our “Voice of the User.”
What do I mean by “Voice of the User”?
Every day, across many avenues, tens of thousands of Firefox users describe a pain point they’ve encountered or comment about their experience with Firefox or make a suggestion. The sum of all of these voices constitutes what I’m referring to as “Voice of the User”. To provide more specificity, here is a list of some channels through which Firefox users are speaking to us every day:
This list is not fully exhaustive, but it provides a good overview for how we can think about “Voice of the User” at a high-level.
So, what strides have we made in the past year? And where do we go from here? First, let’s look at some of the ground we’ve covered so far (this is in no way meant to capture everything):
But we’re not done yet. We have more work ahead of us and I’m sure we’ll come across more stones unturned.
Taking a step back from all the specific examples just highlighted, how do you think we should be approaching “Voice of the User” more generally? How can we optimally stay on top of feedback from Firefox users on a daily basis, considering that that feedback comes from a wide range of channels and that it numbers in the tens of thousands (daily)? Are there broad organizational/community structures or strategies that can help us, i.e., structure/routines/culture that can transform the “Voice of the User” into decisions and actions?
(images from www.journalist.co.uk)
Firefox 3.5 launched just one week ago. It looks like it’s been an incredible success thus far, so we thought we’d offer up a few high-level metrics.
Market Share – from a couple different sources, we’ve seen the worldwide market share for Fx3.5 hit the 4% mark. Real-time tracking is still available at whos.amung.us.
Downloads – At the one week point, cumulative downloads for Fx3.5 have surpassed 17,000,000.
So, how are we supposed to interpret this number? I think it’s safe to say that not many pieces of software have ever seen more than 10 million consumers make the conscious decision to download within a single seven-day period. For some historical perspective, I’ve charted the cumulative download counts for Fx1.5, Fx2, Fx3, and Fx3.5 within the first week of their respective releases. The key thing to keep in mind here is that all of these numbers are gigantic in scale (and the Mozilla community has also grown a lot in the past four years).
