To help get the goal discussion started, it’s obviously helpful to know why SUMO exists. In my opinion, there are three main reasons:

1. To help people have a great Firefox (and by extension web) experience
2. To provide key user and product insights to the Mozilla community
3. To strengthen and grow Mozilla’s community

Based on this list, we can create three focus areas, or “buckets” for our 2010 goals:

• Improve the support experience for users
• Provide better/more accurate/more detailed metrics and insights for other Mozilla teams and the entire Mozilla community
• Make the SUMO experience more enjoyable for contributors

In today’s SUMO meeting, we’ll kick off the discussion by spending 15-30 minutes brainstorming ideas. You’re very welcome to call in! That said, if you don’t have time to call in, or would rather share your ideas in writing, there is an active thread in the SUMO contributor forum dedicated for this. Please feel free to post there with thoughts, ideas, or, if you’re feeling particuarly creative, screenshots, mockups or screencasts of what you have in mind.

Of course, if you don’t have a SUMO account and for some reason don’t want to create it, you are welcome to participate by commenting on this blog post too.

The problem is that if Firefox crashes, it could be for any number of reasons. The range of causes and volume of troubleshooting is so great that we end up doing more to try help the user navigate crash documentation, than offer a solution. In most cases the solution is vague and not very helpful, which confuses users even more.

Each Firefox crash reported to Mozilla using the Mozilla Crash Reporter has a crash ID and lists the type of crash, called the crash signature. Usually, each crash signature has a much more specific cause/solution. Instead of asking users to define each circumstance of their crash, they can get to the solution more quickly if we ask for the crash signature, then provide a document for each crash sig.

What we’ve done is turned our Firefox crashes article into a tutorial on accessing your crash report via about:crashes. At the top right corner of each crash report on crash-stats.mozilla.com, you’ll notice that there is a [Get Help] button. What that does is search support.mozilla.com for the crash signature from the report.

By creating an article for each crash signature, and putting the crash signature in the article content, the Get Help button on crash-stats.mozilla.com will provide the user a link to the article that addresses their specific type of crash.

Naturally, there are a lot of crash signatures. We can’t provide an article for every crash. However, we can get a list of the most common crash signatures, and try to make sure the 10 most common crashes have articles in the knowledge base. Ever since Firefox 3.5 was released, we have been keeping an eye on the top Firefox 3.5.x crashes, and adding them to a list here.

What we need now is people to draft an article for each crash sig. There is a bug link for each crash sig on the list, that contains details about known causes/solutions for each crash. If you need help creating articles, we have a contributor page about creating articles. For any further help, just ask in the Contributors forum.

There are few sources of information to we draw from:

• Top searches: The most common search terms in the SUMO Weekly metrics document.
• Weekly common issues: Our Weekly Common Issues page tracks the most common support issues each week.
• Article polls: At the bottom of each article, there are poll questions: “Did this article solve a problem you had with Firefox?“, “Was this article easy to understand?“, and “Please rate your experience with solving your problem on support.mozilla.com from 1 to 5” (For more precise data there’s the PageView Data.)
• And of course, Article comments: There is a text field on each article for users to provide feedback about the article. When logged in as a contributor, that feedback is displayed at the bottom of the article.

Here’s how that data is utilized to measure the quality of the knowledge base, and make it better:

The top search terms are tested to find out if the first search results contain the article the user is most likely searching for.
If they don’t:

• The correct article may need to be renamed to match the search term.
• The top article in search results may be mistaken for a different issue; so a link to the correct article is added in the intro of the first search result. If users are being redirected to the correct article, the poll data should improve.
• Keywords that match the search terms are added to the correct article.

For generic search terms the article comments for each result may clarify what users are asking about.

The weekly common issues page is checked for any items that need documentation in the knowledge base. If enough information is available to create documentation, the relevant articles are updated or a new article is created.

The comments in articles with the lowest understandability score are checked to get details on what is not understandable in the article, so we can assess what can be done to eliminate that confusion. Sometimes that means rewording or reformatting the article. In some cases it is a matter of adding screenshots. In other cases, it’s a matter of streamlining or purging the article to simplify it for users.

In the end, it’s about taking the data, analyzing why the data is what it is, and what we can do to improve each issue. As a result, the article poll scores should go up, and users will get answers to their questions about using Firefox. We’ve outlined these tests in a contributor page, so everyone as a community can be most affective in making the knowledge base better each week. You can post any suggestions for improvement in the Contributors forum.

There can be many different reasons why people leave a website without interacting with it. When it comes to a support website, one of those reasons can be that the website isn’t helpful enough, or doesn’t provide sufficient instructions on how it should be used. This is something we are trying to minimize on SUMO so the support platform becomes as easy to understand as possible.

In order to improve the current start page, our first step was to figure out how people are using it today and identify areas where we could improve it. chofmann dug up a lot of the initial research about common web design mistakes which our start page was suffering from, and proposed some ideas on how we could use those insights when redesigning the page. chofmann and I then sat down and brainstormed about how we could improve the page, after which I created a simple mockup of our ideas.

Before we could actually test our ideas, we needed to turn the mockup into a polished web page that we would feel comfortable showing to our users, so we turned to Mozilla’s master of design and creativity, John Slater, who connected us with web designer Naz Hamid. The result of our collaboration can be seen below:

The new start page. Click on the image to see a version of it with notes explaining the differences between the current start page.

The test turned out to be successful. With the new start page:

• More people used the search box (+1.3%), which is the best way to use SUMO to find the solution to your problem.
• Fewer people left the site immediately without interacting with it (-0.5%), which means that more people are able to get their problem solved.

For the full report and many interesting insights about how people interact with this new page versus the current page, read the original blog post based on the full analysis of the A/B test by Ken Kovash and Mozilla intern Eric Hergenrader: Improving a User’s Experience with Firefox Support (part II).

Our effort to improve the support experience for our users will of course not end with this test. It’s an ongoing process and a continued focus of the SUMO team to make our support platform as easy to understand and use for as many users as possible.

When looking at the results of our test, it should be noted that the bounce rate is still very high (86%). As I mentioned earlier, there could be many other reasons why people quickly leave the website. One reason, that I suspect plays an important role here, is the fact that you can reach Firefox Support simply by pressing F1 on your keyboard. My theory is that many people accidentally do this when typing on a web page, leading to many unwanted visits to Firefox Support.

The most common SUMO bookmark?

That is one of our next things to test on SUMO: among the people that visit Firefox Support by pressing F1 on the keyboard, how many people close the website right away? Are the people that visit the site by selecting the Help option in the menu more interactive?

We will have the answers to these questions soon.

Our great friend Ken Kovash helped us with analyzing the results of this first test, so please visit the Blog of Metrics for all the details.

We are preparing to launch the second A/B test with a brand new start page design; more on that on this blog soon!

• The Localization Dashboard helps localizers track how much of the site is localized in their language, and view which pages are the most important to translate.
• It is now possible to get notifications when a article edit in their locale is need of review.
• We fixed many website software bugs that were making it difficult to localize.

There is still more we can do, which we plan on addressing soon, such as:

• Listing the differences between Contributors, Approvers/Reviewers, and Locale Leaders, and publishing what permissions each group has, as well as listing who is in each group. This helps contributors identify who the leaders of each community are, and who to contact if they have questions or requests from the leaders of their locale.
• Making it clear what is different in SUMO as opposed to other wikis. Some communities have their own sites, and many contributors are already familiar with other systems. This creates an expectation of how SUMO works, and confusion when SUMO does not work they way they expect. Examples of common causes of confusion from community members include tikiwiki markup, the staging and review system, how article translations work, and how to create/remove the “Content may be out of date” warning.

We’ve been keeping a summary of all l10n feedback on wiki.mozilla.org, so you can take a look if you’re interested. If you are a SUMO localizer, and would like to meet with us, just contact us on this blog or post in the Contributors forum. We’re always willing to meet with you!

We haven’t forgotten! In the next three months, we’d like to test out
the idea, and see how it goes. The purpose of the newsletter would be to
provide a digest of news from the SUMO world for those who are not
actively involved, but still interested in knowing what we’re working on.

Over the next three months, we’ll publish a newsletter each month, and
depending on the feedback from the community, we’ll know whether or not
to make the newsletter a regular thing. Stay tuned for more information.

Monday Jan 12th 2 PM PST/5 PM EST/10 PM GMT:

• California: 650-903-0800 then extension 92, conference number 280#
• Toronto: 416-848-3114 then extension 92, conference number 280#
• Toll-free (US): 800-707-2533 then password 369, conference number 280#
• Skype (free worldwide): +18007072533 then password 369, conference number 280#

The backchannel (where we post links and share text) is #sumo on irc.mozilla.org and our agenda and notes is on the Mozilla wiki.

If you can’t make the time, never fear, just tell us your thoughts in IRC at any time or come to one of our weekly phone meetings Mondays at 10 AM PST/1 PM EST/6 PM GMT.

• It ensures that the solutions to the most common problems are written in a clear, concise, and straightforward language that is easy for our users to understand.
• It reduces the pressure of our community of volunteers since most users are self-served.
• It gives us a powerful way of tracking which problems are the most common with the help of metrics.

The Support Funnel. For more information, see The vision for SUMO – Part 2: Understanding the bigger picture.

So, how do we ensure that people find the solution to their problem in the Knowledge Base? Well, one way of finding the solution is by searching for it — something we try to make very obvious on the start page. The reason why we think searching is the best way of finding the solution is because the Knowledge Base is big. Really big. While we have a list of the most popular support articles right there on the start page, it’s hard to make it obvious that there is a lot more content in the Knowledge Base than what is shown on the start page.

The start page of Firefox Support, clearly emphasizing on the search function.

So, is everyone really comfortable searching? Actually, the almighty and ever so wise chofmann and I have started to see evidence that some people prefer to browse for the solution rather than searching for it. Among the people that visits the Firefox Support start page and doesn’t instantly leaves the page, only roughly half of them actually searches. The other half either clicks on one of the hand-picked popular support articles, or clicks on some other link on the page.

We’re not sure what the reason for that is, or if there are several reasons. It could be that people are unable to accurately describe the problem they’re seeing — considering how many people that are using Firefox today, this is not surprising. Even my older brother, who has been using computers for at least ten years, has problems describing some of the problems he has with his computer, and usually I have to pay him a visit, have a nice cup of coffee, and fix the problem myself.

Another reason could be that people simply prefer to browse a categorized list of articles instead of searching — essentially just clicking on a few links instead of actually typing. I talked to our creative genius John Slater a few weeks ago and he said that he’s usually a little skeptic about internal search engines and that he prefers to just browse.

Francisco Picolinni from the Mozilla Hispano community provided a third possible reason why people are unwilling to search — because they might not think anyone else has the same problem.

Regardless of why not everyone searches, it seems like we should work on providing a good way of browsing the Knowledge Base. We currently have a link at the bottom of the list of popular support articles saying “Browse all Knowledge Base topics.” However, the page that link takes you to is just a long list of all articles ordered by hit count — not exactly easy to navigate.

chofmann, John Slater and I recently brainstormed around how we could ensure that as many people as possible find the solution to their problem in the Knowledge Base with minimum effort. Since the Knowledge Base articles are loosely connected with tags like “bookmarks,” “location bar,” etc., one interesting possibility would be to show these tags in a tag cloud that would provide a better sense of the diversity of the content while still pointing to specific, popular topics. Clicking on a tag would filter the list to only show the articles with that particular tag.

A tag cloud around the topic Web 2.0.

Another problem to solve is making sure that people really understand that they can browse for solutions as an alternative to searching. We want people to understand that Firefox Support has the answer to their problems and that they should stay on the site until the problem is solved. This probably means we have to take a closer look at how the start page is designed to see how we can better communicate this. If you have ideas on how we can achieve this, we are interested in hearing them!

What we really want a user with a problem with Firefox to feel when visiting Firefox Support is: “These people are here for me, and they won’t give up until my problem is solved.”

We just started to work on this, so stay tuned for more…

SUMO contributor myles7897 originally proposed the idea, and more importantly, created a pretty fantastic mockup of how such a newsletter could look like and include:

We’d like to hear what you think of this idea and whether or not this would be useful to you. The idea is that we would blog about various SUMO related things as we already do, and then collect and package the most interesting bits every month and turn that into this newsletter format.

Thanks to myles7897 for creating the mockup, and to Bubo for proof-reading!