Blog of Metrics

A few months ago, David Tenser from our Support team came to the Metrics team with some ideas for analysis concerning user experience with our support.mozilla.com site (SUMO). At the time, we did not have the functionality set up to get the data we wanted, but thanks to some hard work from Jeremy Orem and multiple members of our WebDev team, all of this was made possible about three weeks ago! Now we can see a user’s search term, which article is clicked on after searching, and whether or not the user thought the article was helpful:

The first bit of data we wanted to extract was our Bounce and Refine Search rates for our top search terms in SUMO. For this, I defined a bounce as a SUMO visitor searching a particular term and then immediately leaving the site. Bounces are viewed as failures to give the user results they were looking for. A refined search is defined as a user that searches one term and without clicking an article link or navigating anywhere else searches for a refined version of the same term. Again, we look at a refined search as evidence of a user not receiving the appropriate information with their original search. Here’s what these numbers look like for our top search keywords:

Highlighted in red are the particularly problematic search terms for SUMO in terms of bounce and refine search rates. Private browsing is the most alarming — not only is it our third most popular search term, but it also is in the highest bounce rate and refine search rate groups. From a user standpoint, searching private browsing is most likely an attempt to find out how to turn on private browsing. We can see this in the data; the top refined search for this phrase is start private browsing. Looking at the search results a visitor sees, the only one relevant to private browsing is the top result:

This article does explain how to start private browsing, but the description gives off more of a “what is private browsing” vibe. Either a “Start Private Browsing” article should be started or the phrase should be included in this article’s description.

The bounce rate of home page is a little more perplexing. It seems the main motivation behind this search is to change the default home page in Firefox. Fortunately, the top search result is:

So no issues right? Looking at the refined searches reveals a surprising anomaly: the top two refined searches are “set home page” and “change home page”. I’m baffled. Could users just be skipping past this first result without realizing it? Not likely, but possible. We need to explore other reasons for the high bounce rate — maybe users are searching home page for other reasons.

Next up is history. Users searching this term are (on assumption) likely to be looking to clear their history. Unfortunately, for Firefox 3.0 users there is not a clear result to navigate to. “Clearing Location bar History”, “How to clear Search bar History”, and “Clearing Private Data” are the appropriate articles to continue with, but for inexperienced users who might not know what these terms mean, there is not a definitive how-to-clear-your-history result.

For the mass migration from the downloads search results page (44% bounce rate) I think an article on where to find downloaded files would work well. Right now, there is no result pointing users in this direction, which is probably contributing to such a high bounce rate.

Import bookmarks is up last. Looking at the refined searches, import bookmarks from internet explorer tops the list. Comparing this with the search results, this article is not listed on the front page. I believe moving it up in the line of results could quell the refine search rate.

Each of these changes represents a small change to our database of support articles that could taken together greatly increase the effectiveness of search in the Knowledge Base. In part II of the SUMO analysis, I’ll be taking a look at pathing after searches — the articles visitors are choosing to view after a specific search and their satisfaction with each one.

When I began my analysis of the Broken Site Reporter (seen above) in Firefox (first post can be read here and second here), I imagined a series of 4 or 5 posts with a concluding post on changes I thought would be beneficial to the tool. For now, the end is near for the Reporter series. Why did I cut the analysis short? In my mind, the Broken Site Reporter in its current form leads to too many data issues to draw a whole lot more meaningful analysis from it. The subsequent changes I propose in this post should alleviate these concerns, and after getting fresh data we should be able to breath life back into the series.

So what are the problems we’ve seen data-wise with the Reporter? The first issue that comes to mind is category-confusion. By this I mean certain categories are underrepresented/overrepresented because users are getting confused about where their individual problem fits with respect to the categories we let them choose from. Here are the category types:

Three categories that stand out are Behavior wrong, Appearance wrong, and Printed output is wrong. Both Appearance wrong and Printed output is wrong seem to be subsets of Behavior wrong (for that matter, most reported problems technically belong to subsets of Behavior wrong). It is also unclear what meaning is assigned to Printed output is wrong — is it output printed to the screen (in which case the category is dangerously close to Appearance wrong) or is it output physically printed on paper? In this case, some sort of example next to the category could help my confusion.

Another problem is that Other content missing borders close enough to Other problem that the latter is going to cannibalize reports meant for the former.

With these issues at hand I have come up with a very rough sketch of changes I’d like to see made to the Reporter (Keep in mind that I know absolutely nothing about UI — these pictures are more intended to outline my ideas in a concise way than to provide any sort of finalized draft of what the Reporter should look like; criticism is welcome).

The biggest change in this new Reporter is the narrowing of the problem types. I have drilled them down to five distinct problems (Behavior wrong, Disability access, Printed output is wrong, and Other content missing have all been eliminated). In this way, much of the old Behavior wrong and Printed output is wrong responses can be absorbed into the more concise Appearance Wrong category; reports from these categories that do not get redirected to Appearance Wrong will hopefully be sent to Other Problem. The question mark next to the categories is meant to be a link to a site with descriptions of each problem type and examples of reports and their corresponding appropriate category.

Disability access represents a major hurdle for data analysis in the current Reporter implementation. The category is meant for human-disabilities, but most of the reports it receives are from users’ connections timing out or “Server not Found” errors. These are not the types of errors the Reporter is built for, and they end up skewing our data. In light of this, I am proposing that the Reporter detect these errors when present, and when a user tries to report a broken website, give the user the option of reporting the error in a way like this:

With an error checking page such as this we can both clean up our data and get new useful information about the state of different websites (which sites are down at what time, how often certain sites are reported as down, etc.).

With modifications to the Reporter like these we can get out much cleaner data that will help us better respond to problems in Firefox. Hopefully we’ll see some changes to the tool soon so we can get more actionable data!

In case you missed last week’s announcement, the AMO team recently re-launched its stats dashboard.  If you want to know how the universe of Firefox add-ons has been growing over time (e.g., via downloads or usage), you can easily adjust the chart to see the visualization you’re interested in.  You can also easily export the full data set.  Check it out!

It has now been a full month since the launch of Firefox 3.5 – what impact has the new browser had?

In just over 30 days, Firefox 3.5 has been downloaded 47.6 million times! This does not include downloads from users who go to (in Firefox) Help->Check for Updates. Looking at the conversion rate from user download to capturing that download as a daily user, we seem to have almost regained the conversion of Firefox 2.0:

This is especially impressive when noting that we are in the middle of July in which ADU numbers are most likely depressed from events like schools on summer vacation. Firefox 2.0 was released in October, traditionally a stronger month than July for Firefox.

In Part I of our Broken Web Site Reporter analysis here, we looked at some very general data patterns concerning usage of our Report Broken Web Site tool in the Firefox Help menu. Here I will attempt to dive a little further into users’ responses, examining the Problem Type category from the data. When the user reports a broken website, he/she is given nine different problems to choose from:

These categories help us determine what pain points Firefox users experience most often while surfing the web. Let’s take a look at the distribution of of these since the reporter was launched three years ago:

A first observation — Disability Access has consistently been our top reported website problem. Browsing the comments associated with this category finds many of these reports refer to “Server not found” or “Connection Timed Out” errors. The Disability Access category is meant for websites incompatible with some sort of physical disability, like blindness. It seems from these comments that the name of this category may be confusing or might overlap with some of our other categories.

Taking a look at the Appearance Wrong category, we can see that it has slowly descended from the top reported problem to the fourth from the top. Along with the work done in Firefox 3.0 and 3.5 to display graphics correctly, this decrease has followed a strong increase in the Printed Output is Wrong category introduced in 3.0. It is possible that this newer category is cannibalizing reports from the Appearance Wrong problem type due to their similar connotations. If the printed output in a website is wrong, many users may choose the Appearance Wrong option and vice versa.

Over this time period we can see a relatively linear increase in the Can’t Log In category. Whereas most problem types saw slower growth after the 3.0 launch (as evidenced by their increased flatness), Can’t Log In looks to be growing at about the same rate. It remains to be seen whether or not this trend will continue with 3.5.

What conclusions can we glean from the problem type trends? The Can’t Log In category seems to be the most worrisome. It has been growing steadily since Firefox 2.0, and there is little reason to believe users are confusing Can’t Log In with any of the other categories. Taking a look at specific user comments yields the same results – 35% of en-US users had the word “log in” or a variation in their report. We now know our naming scheme for these reporter problems is flawed; Disability Access sees far too many problems reported, and Appearance Wrong and Printed Output is Wrong are probably feeding off each others’ reports.

Next up in Part III we’ll be looking at the reporter data by locale. Stay tuned!

How would you devise a marketing plan for Mozilla?

This was the business problem that Jakob Marovt and a group of other students recently tried tackling.  Jakob is a student in Slovenia at the University of Ljubljana, Faculty of Computer and Information Science.  His Mozilla project came about earlier this year while studying abroad at University of Southern Denmark, Faculty of Engineering, department of manufacturing and management.

I recently came across Jakob’s weekly presentations and final plan and I was impressed.  It’s not often that a student project leads to such a thorough and accurate understanding of an organization/community.

I followed up with Jakob and asked him about the biggest challenge they encountered or key insight they arrived at.  Given that their assignment called for utilizing frameworks/strategies/tactics within traditional marketing theory (porters five forces, swot analysis, etc.), Jakob noted that one challenge was having to fit Mozilla’s recommended actions within a more mass media and less internet/social media approach.

Great job!

Last week, we looked at which regions have shown the fastest and slowest adoption rates of Firefox 3.5.  Asa had a great idea to expand on this analysis – why not visualize it through a global map to more easily see if there are any overarching patterns across particular regions?

So, we took our Firefox usage data from the past week, uploaded it to IBM’s manyeyes project (which we previously mentioned here), and created the following visualization.  The image below shows Fx3.5 adoption by region.  You’ll notice that there seems to be relatively more dark shaded areas around Russia and Asia.

If you want to explore this visualization further, you can visit the manyeyes page or simply click on the “interact” button below.  You’ll notice that with the drop-down menu near the upper left-hand corner, you can select the Firefox version — Fx2, Fx3.0, or Fx3.5 — you’re interested in viewing stats for.  This should also come in handy for detecting regions where Fx2 usage is still significant (which we previously explored here).

Would you like to see this type of data updated regularly? Would seeing it trended over time be helpful? Let us know.

Earlier today, we highlighted some real-time tracking by whos.amung.us.  Next up, we have another visualization engine for today’s Firefox 3.5 launch – downloadstats.mozilla.com.

This was a somewhat stealth (i.e., surprise) project brought about by our very own Daniel Einspanjer (in collaboration with the folks at SQLstream).  (Thanks to the Mozilla web dev team as well.)

Similar to the real-time tracking presented at Spread Firefox last year, we set out to provide a similar visualization whereby everyone could see downloads happening in real-time – across every corner of the globe.

You’ll notice several cool things within this new visualization – a lit dot for every actual download of Fx3.5;  a “current”, “minimum”, and “maximum” per second download rate for every country (along with a trend over the past minute);  and a “total” or cumulative number of downloads since Fx3.5 was launched this morning (PDT).

In case you’re really curious, here are a couple comparable numbers from Firefox 3′s wildly successful Download Day last June… in its first 24-hours after launch, the “max” per second download rate was approx. 283 and the “average” was approx. 95.  Please keep in mind that the two product launches are not quite apples-to-apples, but these numbers are still interesting to know from simply a context perspective.

Enjoy the live, interactive map!