Congrats everyone!

For those interested in “before and after” pictures, below is one way to visualize the browser market share landscape (based on Net Applications’ data).

Two months later, Eric’s model does a remarkable job predicting future usage.  Below, I have plotted actual against predicted usage.  The correlation between these two lines is remarkable.  Since September, the average prediction was off by just 1.63%!

To better understand daily growth variations, I created a heat map that visualizes month over month usage growth. Each cell contains one day’s growth rate over the same day of the week, 28 days earlier.

Some highlights:

• Strongest growth occurs during Spring and Fall
• Periods of growth and decline are clustered
• Growth is outperforming predictions by 2.58%

We must be careful not to overreach with our conclusions.  We are working with a limited range of data and have likely missed significant confounding variables.  That said–thanks to Eric’s work–we’re off to a great start.  Please leave your thoughts and any suggestions on how to improve our projections in the comments.

Emerging from the summer months, both reports show Firefox gaining about 1% (over the past couple months).  And looking at September specifically, Net Applications shows Firefox gaining more than 0.75%.

Net Applications:

StatCounter:

Beyond just the past couple months, you’ll notice that overall trending also looks quite good.  Over the past year, Net Applications shows Firefox picking up more than 4 percentage points, and StatCounter shows more than 6 percentage points.

• Which search terms lead to which articles?
• Are visitors ultimately satisfied with the articles they choose to view?

Let’s take a look at the data:

Highlighted in green and red are paths our support site seems to do particularly well and poorly with, respectively. A positive take-away from just glancing over the data is that in four cases (under searches private browsing, clear cache, cache, and export bookmarks) we  have top articles performing really well to visitors’ standards.

Focusing on some of our poorer marks, we see the cookies, enable cookies, bookmarks, and clear history searches are not directing users to the content they want. With help from David Tenser, we were able to come up with some reasoning behind these numbers.

Starting with the clear history search to the “Clear Recent History” article, there seems to be a solid explanation for the poor grade. A continuing problem we have had (documented concretely here) with the Awesome Bar is that users do not realize their bookmarks show up while typing in an address. When this is the case, users that believe they are clearing their history will still see any undesirable bookmarks in the Awesome Bar and hence think the “Clear Recent History” article has not helped their problem. Users need to change the content their Awesome Bar remembers in the Privacy section of Firefox preferences to accomplish this task.

Visitors hitting the “Lost Bookmarks” page from the  bookmarks keyword have given the article a low approval rating. An issue with the “Lost Bookmarks” page is that it does not link to an article explaining how to import bookmarks. Users who have upgraded their browser or have just come from another browser may not know how to import their bookmarks, and they navigate to “Lost Bookmarks” in order to find a solution.

We are looking deeper into users’ experiences with two other articles highlighted above in red — “Websites say cookies are blocked” and “Cannot log into websites” — to see if any content is missing and needs to be added. The problems are straightforward enough — visitors making their way to these pages are probably not clicking on them by accident or confused about their problem. The SUMO team is currently implementing changes across the site to increase user satisfaction in areas like those discussed above. This should provide for a much smoother help site experience across the board for our users.

]]> http://blog.mozilla.com/metrics/2009/09/09/getting-help-when-using-firefox-part-ii/feed/ 4 http://blog.mozilla.com/metrics/2009/09/03/getting-help-when-using-firefox-part-i/ http://blog.mozilla.com/metrics/2009/09/03/getting-help-when-using-firefox-part-i/#comments Fri, 04 Sep 2009 00:47:54 +0000 ehergenrader http://blog.mozilla.com/metrics/?p=1065 A few months ago, David Tenser from our Support team came to the Metrics team with some ideas for analysis concerning user experience with our support.mozilla.com site (SUMO). At the time, we did not have the functionality set up to get the data we wanted, but thanks to some hard work from Jeremy Orem and multiple members of our WebDev team, all of this was made possible about three weeks ago! Now we can see a user’s search term, which article is clicked on after searching, and whether or not the user thought the article was helpful:

The first bit of data we wanted to extract was our Bounce and Refine Search rates for our top search terms in SUMO. For this, I defined a bounce as a SUMO visitor searching a particular term and then immediately leaving the site. Bounces are viewed as failures to give the user results they were looking for. A refined search is defined as a user that searches one term and without clicking an article link or navigating anywhere else searches for a refined version of the same term. Again, we look at a refined search as evidence of a user not receiving the appropriate information with their original search. Here’s what these numbers look like for our top search keywords:

Highlighted in red are the particularly problematic search terms for SUMO in terms of bounce and refine search rates. Private browsing is the most alarming — not only is it our third most popular search term, but it also is in the highest bounce rate and refine search rate groups. From a user standpoint, searching private browsing is most likely an attempt to find out how to turn on private browsing. We can see this in the data; the top refined search for this phrase is start private browsing. Looking at the search results a visitor sees, the only one relevant to private browsing is the top result:

This article does explain how to start private browsing, but the description gives off more of a “what is private browsing” vibe. Either a “Start Private Browsing” article should be started or the phrase should be included in this article’s description.

The bounce rate of home page is a little more perplexing. It seems the main motivation behind this search is to change the default home page in Firefox. Fortunately, the top search result is:

So no issues right? Looking at the refined searches reveals a surprising anomaly: the top two refined searches are “set home page” and “change home page”. I’m baffled. Could users just be skipping past this first result without realizing it? Not likely, but possible. We need to explore other reasons for the high bounce rate — maybe users are searching home page for other reasons.

Next up is history. Users searching this term are (on assumption) likely to be looking to clear their history. Unfortunately, for Firefox 3.0 users there is not a clear result to navigate to. “Clearing Location bar History”, “How to clear Search bar History”, and “Clearing Private Data” are the appropriate articles to continue with, but for inexperienced users who might not know what these terms mean, there is not a definitive how-to-clear-your-history result.

For the mass migration from the downloads search results page (44% bounce rate) I think an article on where to find downloaded files would work well. Right now, there is no result pointing users in this direction, which is probably contributing to such a high bounce rate.

Import bookmarks is up last. Looking at the refined searches, import bookmarks from internet explorer tops the list. Comparing this with the search results, this article is not listed on the front page. I believe moving it up in the line of results could quell the refine search rate.

Each of these changes represents a small change to our database of support articles that could taken together greatly increase the effectiveness of search in the Knowledge Base. In part II of the SUMO analysis, I’ll be taking a look at pathing after searches — the articles visitors are choosing to view after a specific search and their satisfaction with each one.

]]> http://blog.mozilla.com/metrics/2009/09/03/getting-help-when-using-firefox-part-i/feed/ 7 http://blog.mozilla.com/metrics/2009/09/02/changing-how-users-report-a-broken-web-site-in-firefox/ http://blog.mozilla.com/metrics/2009/09/02/changing-how-users-report-a-broken-web-site-in-firefox/#comments Wed, 02 Sep 2009 16:05:56 +0000 ehergenrader http://blog.mozilla.com/metrics/?p=898

When I began my analysis of the Broken Site Reporter (seen above) in Firefox (first post can be read here and second here), I imagined a series of 4 or 5 posts with a concluding post on changes I thought would be beneficial to the tool. For now, the end is near for the Reporter series. Why did I cut the analysis short? In my mind, the Broken Site Reporter in its current form leads to too many data issues to draw a whole lot more meaningful analysis from it. The subsequent changes I propose in this post should alleviate these concerns, and after getting fresh data we should be able to breath life back into the series.

So what are the problems we’ve seen data-wise with the Reporter? The first issue that comes to mind is category-confusion. By this I mean certain categories are underrepresented/overrepresented because users are getting confused about where their individual problem fits with respect to the categories we let them choose from. Here are the category types:

Three categories that stand out are Behavior wrong, Appearance wrong, and Printed output is wrong. Both Appearance wrong and Printed output is wrong seem to be subsets of Behavior wrong (for that matter, most reported problems technically belong to subsets of Behavior wrong). It is also unclear what meaning is assigned to Printed output is wrong — is it output printed to the screen (in which case the category is dangerously close to Appearance wrong) or is it output physically printed on paper? In this case, some sort of example next to the category could help my confusion.

Another problem is that Other content missing borders close enough to Other problem that the latter is going to cannibalize reports meant for the former.

With these issues at hand I have come up with a very rough sketch of changes I’d like to see made to the Reporter (Keep in mind that I know absolutely nothing about UI — these pictures are more intended to outline my ideas in a concise way than to provide any sort of finalized draft of what the Reporter should look like; criticism is welcome).

The biggest change in this new Reporter is the narrowing of the problem types. I have drilled them down to five distinct problems (Behavior wrong, Disability access, Printed output is wrong, and Other content missing have all been eliminated). In this way, much of the old Behavior wrong and Printed output is wrong responses can be absorbed into the more concise Appearance Wrong category; reports from these categories that do not get redirected to Appearance Wrong will hopefully be sent to Other Problem. The question mark next to the categories is meant to be a link to a site with descriptions of each problem type and examples of reports and their corresponding appropriate category.

Disability access represents a major hurdle for data analysis in the current Reporter implementation. The category is meant for human-disabilities, but most of the reports it receives are from users’ connections timing out or “Server not Found” errors. These are not the types of errors the Reporter is built for, and they end up skewing our data. In light of this, I am proposing that the Reporter detect these errors when present, and when a user tries to report a broken website, give the user the option of reporting the error in a way like this:

With an error checking page such as this we can both clean up our data and get new useful information about the state of different websites (which sites are down at what time, how often certain sites are reported as down, etc.).

With modifications to the Reporter like these we can get out much cleaner data that will help us better respond to problems in Firefox. Hopefully we’ll see some changes to the tool soon so we can get more actionable data!

]]> http://blog.mozilla.com/metrics/2009/09/02/changing-how-users-report-a-broken-web-site-in-firefox/feed/ 9 http://blog.mozilla.com/metrics/2009/08/31/firefox-add-ons-are-statastic/ http://blog.mozilla.com/metrics/2009/08/31/firefox-add-ons-are-statastic/#comments Mon, 31 Aug 2009 18:51:37 +0000 Ken Kovash http://blog.mozilla.com/metrics/?p=1010 In case you missed last week’s announcement, the AMO team recently re-launched its stats dashboard.  If you want to know how the universe of Firefox add-ons has been growing over time (e.g., via downloads or usage), you can easily adjust the chart to see the visualization you’re interested in.  You can also easily export the full data set.  Check it out!

Thanks to some recent work by Simon Krueger, our metrics intern, we’ve been able to arrive at a rough approximation.  Here’s my thought process for the important data points Simon’s been able to extract:

• We know that there are roughly 300 Million Firefox users, and on a given day, we see about 100 Million active users of Firefox
• Narrowing our focus to just a single day’s worth of data (we simply picked June 22nd), we turned our attention to estimating add-ons usage for just that one day, so that we could compare something on a relative basis to our overall Fx metric of 100 Million active daily users
• We saw Firefox add-on pings from 32.8 Million unique IP addresses
• There are a couple issues at this point: (1) There could be multiple users from the same IP address, and (2) There could be a handful of add-ons where adoption isn’t fully user-initiated (e.g., people happen to have it, but don’t know what it is)
• Balancing these two issues (i.e., they may come close to offsetting), I think it’s safe to say that roughly 33% of Firefox users (32.8/100) have at least one add-on
• So, of our user base (roughly 300 Million), perhaps about 100 Million people use at least one add-on

I want to conclude with a few words of caution.  This methodology is far from perfect.  It’s conservative and there are also potential holes in it (some of which we haven’t yet thought of).  It also remains unclear as to how useable this slice of data will be as we see it trended in the future.  Moreover, the data above is wide open to interpretation.  The add-ons team and other folks across the community may arrive at a completely different interpretation (read here for Justin Scott’s point of view).  Regardless, a fruitful discussion around this topic should hopefully ensue.

(photo from mynameisideal.blogspot.com)

In just over 30 days, Firefox 3.5 has been downloaded 47.6 million times! This does not include downloads from users who go to (in Firefox) Help->Check for Updates. Looking at the conversion rate from user download to capturing that download as a daily user, we seem to have almost regained the conversion of Firefox 2.0:

This is especially impressive when noting that we are in the middle of July in which ADU numbers are most likely depressed from events like schools on summer vacation. Firefox 2.0 was released in October, traditionally a stronger month than July for Firefox.