Blog of Metrics

In part I of this two-part series (found here), I discussed the first level of my analysis of our support site. Next, we asked ourselves two questions –

• Which search terms lead to which articles?
• Are visitors ultimately satisfied with the articles they choose to view?

Let’s take a look at the data:

Highlighted in green and red are paths our support site seems to do particularly well and poorly with, respectively. A positive take-away from just glancing over the data is that in four cases (under searches private browsing, clear cache, cache, and export bookmarks) we  have top articles performing really well to visitors’ standards.

Focusing on some of our poorer marks, we see the cookies, enable cookies, bookmarks, and clear history searches are not directing users to the content they want. With help from David Tenser, we were able to come up with some reasoning behind these numbers.

Starting with the clear history search to the “Clear Recent History” article, there seems to be a solid explanation for the poor grade. A continuing problem we have had (documented concretely here) with the Awesome Bar is that users do not realize their bookmarks show up while typing in an address. When this is the case, users that believe they are clearing their history will still see any undesirable bookmarks in the Awesome Bar and hence think the “Clear Recent History” article has not helped their problem. Users need to change the content their Awesome Bar remembers in the Privacy section of Firefox preferences to accomplish this task.

Visitors hitting the “Lost Bookmarks” page from the  bookmarks keyword have given the article a low approval rating. An issue with the “Lost Bookmarks” page is that it does not link to an article explaining how to import bookmarks. Users who have upgraded their browser or have just come from another browser may not know how to import their bookmarks, and they navigate to “Lost Bookmarks” in order to find a solution.

We are looking deeper into users’ experiences with two other articles highlighted above in red — “Websites say cookies are blocked” and “Cannot log into websites” — to see if any content is missing and needs to be added. The problems are straightforward enough — visitors making their way to these pages are probably not clicking on them by accident or confused about their problem. The SUMO team is currently implementing changes across the site to increase user satisfaction in areas like those discussed above. This should provide for a much smoother help site experience across the board for our users.

A few months ago, David Tenser from our Support team came to the Metrics team with some ideas for analysis concerning user experience with our support.mozilla.com site (SUMO). At the time, we did not have the functionality set up to get the data we wanted, but thanks to some hard work from Jeremy Orem and multiple members of our WebDev team, all of this was made possible about three weeks ago! Now we can see a user’s search term, which article is clicked on after searching, and whether or not the user thought the article was helpful:

The first bit of data we wanted to extract was our Bounce and Refine Search rates for our top search terms in SUMO. For this, I defined a bounce as a SUMO visitor searching a particular term and then immediately leaving the site. Bounces are viewed as failures to give the user results they were looking for. A refined search is defined as a user that searches one term and without clicking an article link or navigating anywhere else searches for a refined version of the same term. Again, we look at a refined search as evidence of a user not receiving the appropriate information with their original search. Here’s what these numbers look like for our top search keywords:

Highlighted in red are the particularly problematic search terms for SUMO in terms of bounce and refine search rates. Private browsing is the most alarming — not only is it our third most popular search term, but it also is in the highest bounce rate and refine search rate groups. From a user standpoint, searching private browsing is most likely an attempt to find out how to turn on private browsing. We can see this in the data; the top refined search for this phrase is start private browsing. Looking at the search results a visitor sees, the only one relevant to private browsing is the top result:

This article does explain how to start private browsing, but the description gives off more of a “what is private browsing” vibe. Either a “Start Private Browsing” article should be started or the phrase should be included in this article’s description.

The bounce rate of home page is a little more perplexing. It seems the main motivation behind this search is to change the default home page in Firefox. Fortunately, the top search result is:

So no issues right? Looking at the refined searches reveals a surprising anomaly: the top two refined searches are “set home page” and “change home page”. I’m baffled. Could users just be skipping past this first result without realizing it? Not likely, but possible. We need to explore other reasons for the high bounce rate — maybe users are searching home page for other reasons.

Next up is history. Users searching this term are (on assumption) likely to be looking to clear their history. Unfortunately, for Firefox 3.0 users there is not a clear result to navigate to. “Clearing Location bar History”, “How to clear Search bar History”, and “Clearing Private Data” are the appropriate articles to continue with, but for inexperienced users who might not know what these terms mean, there is not a definitive how-to-clear-your-history result.

For the mass migration from the downloads search results page (44% bounce rate) I think an article on where to find downloaded files would work well. Right now, there is no result pointing users in this direction, which is probably contributing to such a high bounce rate.

Import bookmarks is up last. Looking at the refined searches, import bookmarks from internet explorer tops the list. Comparing this with the search results, this article is not listed on the front page. I believe moving it up in the line of results could quell the refine search rate.

Each of these changes represents a small change to our database of support articles that could taken together greatly increase the effectiveness of search in the Knowledge Base. In part II of the SUMO analysis, I’ll be taking a look at pathing after searches — the articles visitors are choosing to view after a specific search and their satisfaction with each one.

When I began my analysis of the Broken Site Reporter (seen above) in Firefox (first post can be read here and second here), I imagined a series of 4 or 5 posts with a concluding post on changes I thought would be beneficial to the tool. For now, the end is near for the Reporter series. Why did I cut the analysis short? In my mind, the Broken Site Reporter in its current form leads to too many data issues to draw a whole lot more meaningful analysis from it. The subsequent changes I propose in this post should alleviate these concerns, and after getting fresh data we should be able to breath life back into the series.

So what are the problems we’ve seen data-wise with the Reporter? The first issue that comes to mind is category-confusion. By this I mean certain categories are underrepresented/overrepresented because users are getting confused about where their individual problem fits with respect to the categories we let them choose from. Here are the category types:

Three categories that stand out are Behavior wrong, Appearance wrong, and Printed output is wrong. Both Appearance wrong and Printed output is wrong seem to be subsets of Behavior wrong (for that matter, most reported problems technically belong to subsets of Behavior wrong). It is also unclear what meaning is assigned to Printed output is wrong — is it output printed to the screen (in which case the category is dangerously close to Appearance wrong) or is it output physically printed on paper? In this case, some sort of example next to the category could help my confusion.

Another problem is that Other content missing borders close enough to Other problem that the latter is going to cannibalize reports meant for the former.

With these issues at hand I have come up with a very rough sketch of changes I’d like to see made to the Reporter (Keep in mind that I know absolutely nothing about UI — these pictures are more intended to outline my ideas in a concise way than to provide any sort of finalized draft of what the Reporter should look like; criticism is welcome).

The biggest change in this new Reporter is the narrowing of the problem types. I have drilled them down to five distinct problems (Behavior wrong, Disability access, Printed output is wrong, and Other content missing have all been eliminated). In this way, much of the old Behavior wrong and Printed output is wrong responses can be absorbed into the more concise Appearance Wrong category; reports from these categories that do not get redirected to Appearance Wrong will hopefully be sent to Other Problem. The question mark next to the categories is meant to be a link to a site with descriptions of each problem type and examples of reports and their corresponding appropriate category.

Disability access represents a major hurdle for data analysis in the current Reporter implementation. The category is meant for human-disabilities, but most of the reports it receives are from users’ connections timing out or “Server not Found” errors. These are not the types of errors the Reporter is built for, and they end up skewing our data. In light of this, I am proposing that the Reporter detect these errors when present, and when a user tries to report a broken website, give the user the option of reporting the error in a way like this:

With an error checking page such as this we can both clean up our data and get new useful information about the state of different websites (which sites are down at what time, how often certain sites are reported as down, etc.).

With modifications to the Reporter like these we can get out much cleaner data that will help us better respond to problems in Firefox. Hopefully we’ll see some changes to the tool soon so we can get more actionable data!