Now that this new program has been live for about a week, we thought we’d check the numbers (cumulative since last week):

Considering that this page hasn’t been directly pushed to users, these numbers are significant.  There are a couple spots within mozilla.com where users can click to find this page, but otherwise, our web analytics reporting tells us that about 90% of the traffic has been coming from news and technology related web sites.

Next, you may be wondering where people ventured upon clicking to “Update” or “Research”.  Here’s a breakdown:

We’ll continue to keep an eye on these numbers, and perhaps more excitedly, there are plans to soon integrate this feature into the browser itself.

Two months later, Eric’s model does a remarkable job predicting future usage.  Below, I have plotted actual against predicted usage.  The correlation between these two lines is remarkable.  Since September, the average prediction was off by just 1.63%!

To better understand daily growth variations, I created a heat map that visualizes month over month usage growth. Each cell contains one day’s growth rate over the same day of the week, 28 days earlier.

Some highlights:

• Strongest growth occurs during Spring and Fall
• Periods of growth and decline are clustered
• Growth is outperforming predictions by 2.58%

We must be careful not to overreach with our conclusions.  We are working with a limited range of data and have likely missed significant confounding variables.  That said–thanks to Eric’s work–we’re off to a great start.  Please leave your thoughts and any suggestions on how to improve our projections in the comments.

Why is this important?

• It provides users with an opportunity to share their praise/criticism or any other thoughts
• It strongly complements our web analytics efforts (i.e., we’re able to answer the “why” questions in terms of user experiences)
• It ultimately allows the Fx team, marketing team, and many other groups across Mozilla to gain valuable insights and improve the Fx user experience

To accomplish this, we’ve implemented a feedback button/form on three critical touch points at mozilla.com:

Getting Started page

First Run page

Whatsnew page

The feedback form on each page is slightly different (looking at the “categories” and “sub-categories”), depending on a user’s expectation of that page and the typical point in a user’s experience when he/she hits that page.  As an example, below is the form you will see upon hitting the feedback button on the whatsnew/update page.  And like our previous implementations, we’re not 100% sure what feedback to expect, so we’re going to closely monitor user comments and tweak the feedback forms accordingly.

Stay tuned!

Emerging from the summer months, both reports show Firefox gaining about 1% (over the past couple months).  And looking at September specifically, Net Applications shows Firefox gaining more than 0.75%.

Net Applications:

StatCounter:

Beyond just the past couple months, you’ll notice that overall trending also looks quite good.  Over the past year, Net Applications shows Firefox picking up more than 4 percentage points, and StatCounter shows more than 6 percentage points.

You may have noticed that Mozilla Labs implemented its very first experiment for Test Pilot earlier this month.  The goal is to answer the following two questions (or at least put some quantitative framework around the questions):

1. What would be the best default tab behavior after users close a tab?
2. What should be default placement when users open a new tab?

As someone who downloaded Test Pilot and participated in this study, I was impressed by a couple things.  First, the introductory page did a nice job of summarizing what was going on (first image below).  And second, at any time I could view what data was being collected about me (latter image).  At the end of the experiment, it was then my decision whether or not to submit my data.

Moving forward, the metrics team looks forward to partnering the Test Pilot team in making an impact with the resulting data.  Stay tuned!

• 99% of internet users (desktop) have Flash.
• The vast majority of people have an out of date version.  One study claims 80% and mozilla.com’s own traffic stats show about 75% of visitors on a non current version.

So, what has transpired since last Wednesday?

In one week, 10,000,000 people have clicked on the “flash update” link below.

Taking this analysis one step further, we wanted to gain a better sense for users’ interaction with this page.  Breaking down the data by day, we looked more carefully at the en-US version of the 3.5.3 whatsnew (or update) page and pulled the following numbers:

1. How many total people hit the whatsnew page?
2. Of this cohort, how many had an out of date Flash version, and hence, saw the message above?
3. And of this smaller cohort, how many people actually clicked on the flash update link?

Beyond the total impact of 10,000,000 clicks, the most impressive pattern that stands out is the click through rate.  While the Firefox whatsnew page generally sees a click through rate below 5%, the flash update link alone has generated a click through rate north of 30%.  Phenomenal!

• Which search terms lead to which articles?
• Are visitors ultimately satisfied with the articles they choose to view?

Let’s take a look at the data:

Highlighted in green and red are paths our support site seems to do particularly well and poorly with, respectively. A positive take-away from just glancing over the data is that in four cases (under searches private browsing, clear cache, cache, and export bookmarks) we  have top articles performing really well to visitors’ standards.

Focusing on some of our poorer marks, we see the cookies, enable cookies, bookmarks, and clear history searches are not directing users to the content they want. With help from David Tenser, we were able to come up with some reasoning behind these numbers.

Starting with the clear history search to the “Clear Recent History” article, there seems to be a solid explanation for the poor grade. A continuing problem we have had (documented concretely here) with the Awesome Bar is that users do not realize their bookmarks show up while typing in an address. When this is the case, users that believe they are clearing their history will still see any undesirable bookmarks in the Awesome Bar and hence think the “Clear Recent History” article has not helped their problem. Users need to change the content their Awesome Bar remembers in the Privacy section of Firefox preferences to accomplish this task.

Visitors hitting the “Lost Bookmarks” page from the  bookmarks keyword have given the article a low approval rating. An issue with the “Lost Bookmarks” page is that it does not link to an article explaining how to import bookmarks. Users who have upgraded their browser or have just come from another browser may not know how to import their bookmarks, and they navigate to “Lost Bookmarks” in order to find a solution.

We are looking deeper into users’ experiences with two other articles highlighted above in red — “Websites say cookies are blocked” and “Cannot log into websites” — to see if any content is missing and needs to be added. The problems are straightforward enough — visitors making their way to these pages are probably not clicking on them by accident or confused about their problem. The SUMO team is currently implementing changes across the site to increase user satisfaction in areas like those discussed above. This should provide for a much smoother help site experience across the board for our users.

]]> http://blog.mozilla.com/metrics/2009/09/09/getting-help-when-using-firefox-part-ii/feed/ 4 http://blog.mozilla.com/metrics/2009/09/08/helping-people-upgrade-to-the-latest-version-of-firefox/ http://blog.mozilla.com/metrics/2009/09/08/helping-people-upgrade-to-the-latest-version-of-firefox/#comments Tue, 08 Sep 2009 16:04:42 +0000 Ken Kovash http://blog.mozilla.com/metrics/?p=1092 We discussed a couple more prominent issues at the time, but when we last analyzed why people cancel out of the Firefox installation process (the actual installer), we found one other significant group of people who could use some assistance – a cohort of existing Firefox users who were confused about what they were doing (in this case, a fresh install) and what they were optimally supposed to be doing (e.g., going to Help -> Check for Updates).

This insight brought me back in time.

When I first started talking with Mozilla folks a few years ago (before joining the org), one of the first things I wanted to do was make sure I was using the latest version of Firefox.  At the time, I went to mozilla.com and tried looking for information about how to do so.  I didn’t find anything and then searched on Google (again, unsuccessfully).  Eventually, I gave up and just did a fresh download and install, all the while not feeling totally sure I was doing the right thing.

Now, armed with our new insight, a couple changes are either on the way or have just been implemented:

• The Firefox team is making one small UI change (mentioned in Rob Strong’s blog post)
• The marketing team changed the main Firefox product page at www.mozilla.com, as seen by existing Firefox users.  Below are the two page variations, and I’ve highlighted the new additions to the pages.  These features point to this newly created FAQ page answering many key questions related to… how do I update/upgrade?  do I need to do a fresh install?  do I need to uninstall first?  what will happen to my bookmarks?  etc.

Thanks to Laura Mesa, John Slater, among many others, for their efforts.  It’s always nice to see our metrics activities come full circle – data -> insights -> actions/changes -> improved user experience!

The first bit of data we wanted to extract was our Bounce and Refine Search rates for our top search terms in SUMO. For this, I defined a bounce as a SUMO visitor searching a particular term and then immediately leaving the site. Bounces are viewed as failures to give the user results they were looking for. A refined search is defined as a user that searches one term and without clicking an article link or navigating anywhere else searches for a refined version of the same term. Again, we look at a refined search as evidence of a user not receiving the appropriate information with their original search. Here’s what these numbers look like for our top search keywords:

Highlighted in red are the particularly problematic search terms for SUMO in terms of bounce and refine search rates. Private browsing is the most alarming — not only is it our third most popular search term, but it also is in the highest bounce rate and refine search rate groups. From a user standpoint, searching private browsing is most likely an attempt to find out how to turn on private browsing. We can see this in the data; the top refined search for this phrase is start private browsing. Looking at the search results a visitor sees, the only one relevant to private browsing is the top result:

This article does explain how to start private browsing, but the description gives off more of a “what is private browsing” vibe. Either a “Start Private Browsing” article should be started or the phrase should be included in this article’s description.

The bounce rate of home page is a little more perplexing. It seems the main motivation behind this search is to change the default home page in Firefox. Fortunately, the top search result is:

So no issues right? Looking at the refined searches reveals a surprising anomaly: the top two refined searches are “set home page” and “change home page”. I’m baffled. Could users just be skipping past this first result without realizing it? Not likely, but possible. We need to explore other reasons for the high bounce rate — maybe users are searching home page for other reasons.

Next up is history. Users searching this term are (on assumption) likely to be looking to clear their history. Unfortunately, for Firefox 3.0 users there is not a clear result to navigate to. “Clearing Location bar History”, “How to clear Search bar History”, and “Clearing Private Data” are the appropriate articles to continue with, but for inexperienced users who might not know what these terms mean, there is not a definitive how-to-clear-your-history result.

For the mass migration from the downloads search results page (44% bounce rate) I think an article on where to find downloaded files would work well. Right now, there is no result pointing users in this direction, which is probably contributing to such a high bounce rate.

Import bookmarks is up last. Looking at the refined searches, import bookmarks from internet explorer tops the list. Comparing this with the search results, this article is not listed on the front page. I believe moving it up in the line of results could quell the refine search rate.

Each of these changes represents a small change to our database of support articles that could taken together greatly increase the effectiveness of search in the Knowledge Base. In part II of the SUMO analysis, I’ll be taking a look at pathing after searches — the articles visitors are choosing to view after a specific search and their satisfaction with each one.

]]> http://blog.mozilla.com/metrics/2009/09/03/getting-help-when-using-firefox-part-i/feed/ 7 http://blog.mozilla.com/metrics/2009/09/02/changing-how-users-report-a-broken-web-site-in-firefox/ http://blog.mozilla.com/metrics/2009/09/02/changing-how-users-report-a-broken-web-site-in-firefox/#comments Wed, 02 Sep 2009 16:05:56 +0000 ehergenrader http://blog.mozilla.com/metrics/?p=898

When I began my analysis of the Broken Site Reporter (seen above) in Firefox (first post can be read here and second here), I imagined a series of 4 or 5 posts with a concluding post on changes I thought would be beneficial to the tool. For now, the end is near for the Reporter series. Why did I cut the analysis short? In my mind, the Broken Site Reporter in its current form leads to too many data issues to draw a whole lot more meaningful analysis from it. The subsequent changes I propose in this post should alleviate these concerns, and after getting fresh data we should be able to breath life back into the series.

So what are the problems we’ve seen data-wise with the Reporter? The first issue that comes to mind is category-confusion. By this I mean certain categories are underrepresented/overrepresented because users are getting confused about where their individual problem fits with respect to the categories we let them choose from. Here are the category types:

Three categories that stand out are Behavior wrong, Appearance wrong, and Printed output is wrong. Both Appearance wrong and Printed output is wrong seem to be subsets of Behavior wrong (for that matter, most reported problems technically belong to subsets of Behavior wrong). It is also unclear what meaning is assigned to Printed output is wrong — is it output printed to the screen (in which case the category is dangerously close to Appearance wrong) or is it output physically printed on paper? In this case, some sort of example next to the category could help my confusion.

Another problem is that Other content missing borders close enough to Other problem that the latter is going to cannibalize reports meant for the former.

With these issues at hand I have come up with a very rough sketch of changes I’d like to see made to the Reporter (Keep in mind that I know absolutely nothing about UI — these pictures are more intended to outline my ideas in a concise way than to provide any sort of finalized draft of what the Reporter should look like; criticism is welcome).

The biggest change in this new Reporter is the narrowing of the problem types. I have drilled them down to five distinct problems (Behavior wrong, Disability access, Printed output is wrong, and Other content missing have all been eliminated). In this way, much of the old Behavior wrong and Printed output is wrong responses can be absorbed into the more concise Appearance Wrong category; reports from these categories that do not get redirected to Appearance Wrong will hopefully be sent to Other Problem. The question mark next to the categories is meant to be a link to a site with descriptions of each problem type and examples of reports and their corresponding appropriate category.

Disability access represents a major hurdle for data analysis in the current Reporter implementation. The category is meant for human-disabilities, but most of the reports it receives are from users’ connections timing out or “Server not Found” errors. These are not the types of errors the Reporter is built for, and they end up skewing our data. In light of this, I am proposing that the Reporter detect these errors when present, and when a user tries to report a broken website, give the user the option of reporting the error in a way like this:

With an error checking page such as this we can both clean up our data and get new useful information about the state of different websites (which sites are down at what time, how often certain sites are reported as down, etc.).

With modifications to the Reporter like these we can get out much cleaner data that will help us better respond to problems in Firefox. Hopefully we’ll see some changes to the tool soon so we can get more actionable data!

]]> http://blog.mozilla.com/metrics/2009/09/02/changing-how-users-report-a-broken-web-site-in-firefox/feed/ 9