SUMO Blog

Following up on my post last week about how the SUMO project developed in 2009, it’s time to repeat the circle for 2010! It’s time to start thinking about where to take the project, which areas to focus on, and ultimately which goals to define for 2010.

To help get the goal discussion started, it’s obviously helpful to know why SUMO exists. In my opinion, there are three main reasons:

1. To help people have a great Firefox (and by extension web) experience
2. To provide key user and product insights to the Mozilla community
3. To strengthen and grow Mozilla’s community

Based on this list, we can create three focus areas, or “buckets” for our 2010 goals:

• Improve the support experience for users
• Provide better/more accurate/more detailed metrics and insights for other Mozilla teams and the entire Mozilla community
• Make the SUMO experience more enjoyable for contributors

In today’s SUMO meeting, we’ll kick off the discussion by spending 15-30 minutes brainstorming ideas. You’re very welcome to call in! That said, if you don’t have time to call in, or would rather share your ideas in writing, there is an active thread in the SUMO contributor forum dedicated for this. Please feel free to post there with thoughts, ideas, or, if you’re feeling particuarly creative, screenshots, mockups or screencasts of what you have in mind.

Of course, if you don’t have a SUMO account and for some reason don’t want to create it, you are welcome to participate by commenting on this blog post too.

Here’s one of those things that make me feel good about being a part of SUMO and Mozilla.

In late 2008, the SUMO team started to collect feedback from the community about what we should focus on in order to make the platform and Firefox Support website a more exciting place to collaborate on. We also added our own ideas about how to improve the experience for both Firefox users visiting the site and contributors helping out. The result of this work is something I called the Vision for SUMO followed by the SUMO 2009 roadmap.

Since then, we’ve worked to make the vision a reality. We’ve improved the quality of our support. We’ve improved our ability to provide user insights and track user trends. We’ve managed to implement many cool and useful features (with lots of help from the amazing web development team at Mozilla) which really made it easier and more fun to contribute on SUMO as well as improved the experience on the website for users. Perhaps most importantly, we’ve had a good time.

All these amazing achievements aside, this is what makes me the most proud:

Growth of SUMO community since October 2008

In a little less than a year, we’ve managed to double the number of active locales, doubled the number of translated articles, and most importantly, more than doubled the number of active SUMO contributors!

As a side note, the survey that was sent out after MozCamp 09 in Prague showed that 30% of the attendees were involved with user support. That’s an amazingly large proportion of our European community actively involved with helping our users having a great experience on the web!

30% of Ludovic Hirlimann's photo of the European community, used under CC-BY-NC license

Even though it’s just October, I’m already blown away by the achievements by everyone in the SUMO community in 2009. And by the way, I’m glad it’s just October: it’s time to start thinking about where to take SUMO in 2010! More on that tomorrow very soon…

Running the latest version of Firefox with outdated plugins is like having a modern burglar alarm while leaving the balcony door wide open. In order to be safe on the web, your plugins must be updated too.

As an example of the real threats of using outdated plugins, consider the popular Flash plugin created by Adobe Systems. It is used to play videos on YouTube and many other sites, and has a history of serious security issues. What this means is that if you haven’t upgraded to the latest version of Flash, it doesn’t matter if you’re using the latest and most secure version of Firefox because plugins are independent pieces of programs running inside the web browser, which Mozilla doesn’t control (and didn’t create). Of course, this is true for every web browser out there, including Internet Explorer.

In other words, no matter how safe your web browser is, you’re still vulnerable to security exploits if your plugins are not up to date, because they open up security holes beyond the web browser’s control.

So how do you ensure that your plugins are up to date? Mozilla’s new Plugin Check page comes to the rescue! The page went live today on mozilla.com after a few weeks of testing and is a very important step in making the web safer. To see if you’re running any outdated and insecure plugins, just visit the Plugin Check page and update any plugins marked as out of date.

For Firefox Support and the SUMO community, this is an important milestone since 30% of the reported crashes are caused by plugins. By increasing the number of users running updated plugins, the number of reported problems on SUMO should drop significantly. Also, giving users access to a page that allows them to take control of their security is a good example of how important web safety is for Mozilla.

This Plugin Check page is just the first step in making it easier to keep your plugins safe. Future versions of Firefox will have this functionality built in, eventually making the experience as seamless as keeping Firefox itself up to date. In the meantime, we’ll make sure to link to the Plugin Check page wherever it makes sense to do so on SUMO.

Sumo

• Weekly metrics
• Last week’s weekly support issues
  • The “AVG blocks Firefox as a trojan” is now fixed!

Knowledge Base

• EU Ballot
  • Finalize list of articles for KB audit by tomorrow?
  • Thoughts on SHOWFOR for Windows 7 based on article list
    • Is it really needed, or can we get away with just creating a “How to install Firefox on Windows 7″ article and simplifying the other articles?
  • Improve Ask a question page?

Forum

• Proposal: Implement checking for tags to predict user’s question (mockup) – Milos
  • Actually already part of the original Ask a Question redesign, but had to be pushed out because an outdated version of Sphinx. Upgrading Sphinx and fixing this will happen in Q4 2009!

Live Chat

• Tags and issues from last week
  • Add-on problems were the most common issue
  • AVG was misidentifying the Firefox installer as a trojan, should be fixed in latest AVG update
  • zzxc to try different graph designs next week to make it easier to understand

Roundtable

• Blog post about Live Chat community on Wednesday (zzxc)
• Effort to get feedback from Firefox beta users in a series of opt-in surveys. What kind of information do we need? comment here
• We have a new system that allows us to do A/B testing on a lot more stuff. Check out the current list of tests and let us know your ideas

Over the past quarter, there has been an effort with Firefox Support around working closer with everyone in the Mozilla community and acting as a bridge between the Mozilla community and the more mainstream segment of Firefox users.  We blogged earlier about some of the ways that we can provide feedback to Firefox developers, QA and other members of the Mozilla community. This blog post will focus on some concrete examples of how this close relationship worked in the wake of Firefox 3.5.

Norton toolbar disabled in Firefox 3.5

Symantec did not have the Norton toolbar updated in advance of the Firefox 3.5 push so many users who updated found themselves without a key part of their browser.  This understandably upset many Firefox users who interpreted the “Norton toolbar is disabled” message as that they were no longer safe online.

In this instance, a close collaboration with the QA, Add-ons and Firefox team alerted us that this may be a big deal so we got a knowledge base article written right away in advance of the push.  Furthermore, with a downloadable patch from Norton, we were able to give these Firefox users a workaround and help make sure that their Firefox 3.5 upgrade went as smoothly as possible.

Lost bookmarks when upgrading

When the upgrade to 3.5 offer was popped up to all 3.0 users, we suddenly noticed a significant increase in the number of users reporting that some or all of their bookmarks were missing.  In addition to helping users restore their bookmarks and updating KB articles, we worked closely with the Firefox team to figure out the cause.  After collecting files from willing users and doing some detective work in conjunction with affected users, we managed to figure out that these users were actually getting reverted to bookmarks from Firefox 2.  This led to a quick patch that rolled out in the next update of Firefox.

What’s really great is how quickly we went from discovering a problem through our forums and live chat into putting out a patch that potentially helped millions of Firefox users who’ve updated since.  This loop of getting information from users and turning that into fixes for users is what makes collaboration with the greater Mozilla community so rewarding and why it’s so important.

Problems caused by extensions, plugins and third party software

Being able to work with users experiencing a problem, SUMO also plays a role in identifying extensions, plugins and third party programs that can issues for users.  For example, when a number of users complained about not being able to use the right mouse button in Firefox 3.5, we narrowed down that Yahoo Toolbar versions older than 2.0 were the cause.  Just this week, SUMO was one of the first places that users reported that AVG was marking Firefox as a trojan.  Very quickly, we got a KB article up and within 24 hours, Tomcat in Mozilla’s QA group was able to work with AVG and get updated virus definitions pushed.

Crashes

In addition to the above, we’ve been working on incorporating information about top crashes into the knowledge base and helping connect the developers working on crashes with users experiencing them.  As part of that we had a recent sprint to write KB articles for the 30 top crashes and are constantly monitoring new topcrashes.  In the upcoming quarter, we hope to really ramp up this effort and make it even faster to get crashed diagnosed or reproduced so they can be fixed.

In general, working closely with support, not only benefits the 4 million SUMO visitors a week, it can also make the Firefox experience better for the 300 million Firefox users out there.

What you can do to help

As a member of the Mozilla community, there are a couple things that you can do to help us work better together.

1. Work on, triage or help reproduce common-issue+ bugs. We’re using the common-issue+ keyword to mark issues that we see a lot on support.  Getting activity on those bugs helps complete the loop from user via SUMO back to improving the experience for users.
2. Mark commonly-duped bugs with the common-issue? keyword. We may not spot every issue or in some cases, the symptoms of a bug are generic and we may not realize they’re distinct from another bug.  Marking a bug with the common-issue? keyword lets us know to pay specific attention to an issue and see if it’s actually happening to users who show up on SUMO.
3. Help with user support.  Working with users relies on having community members answer questions on our support forums and live chat service.  It’s really easy to get started and the more user questions we answer, the more information we can feed back to the larger Mozilla community.

The full slides for a brownbag Cheng hosted on this topic can be found here.  If you have more ideas for how SUMO can work better with you or other feedback, please let us know in the comments below, in the #sumo IRC channel or via email.

Sumo

• Weekly metrics
  • cilias will work with metrics team to figure out why our bounce rate is increasing
• Last week’s weekly support issues
  • Fewer repeated issues
  • Dramatic increase in searches related to “cookies”
    • cww to work with metrics to figure out what we can tell about the people who search for “cookies”, e.g. OS, browser, country, etc.

Knowledge Base

• New articles: Using Firefox with a 64-bit Operating System, Crash signature – nsEventListenerManager::Release(), Bookmarks toolbar, Crash signature – js_LockGCThingRT
• CSAT has taken another dive. I still need to assess the cause.
• Had meeting with Metrics team last week about prioritizing ideas for improving the CSAT score in Q4.
  • We need to do some A/B tests to determine the impact each solution, then we can determine which solutions are most important.
  • cilias needs to make sure they have the raw data from the past year, so they can help determine the reasons for fluctuations in the CSAT score.
  • Will try to figure out why the CSAT is dropping in non-en-US locales (which locales are causing it?)

Forum

• Participation fell. Need to investigate if it’s related to new forum AAQ
• [1] New forum start page (temp)
  • Cww to post in Contributor forum once it’s back up and ask for feedback

Live Chat

• Tags and issues from last week
  • Problems caused by add-ons now #2 issue
• Web client status
  • Behind schedule, but will be online to meet Q3 goal
    • As soon as it’s up on prod, will announce
  • Will be on production server before or shortly after this call (link)
  • Future development will focus first on integrating canned responses and SUMO search/login into the client, then on the remaining features in the PRD.

Sumo

• Weekly metrics
  • Start page bounce rate theory from last week: large increase in referrers from Firefox whatsnew pages.
  • CSAT score on Forum and live chat went up. Not sure why.
  • Top articles block was updated last week, to reflect top questions.
• Last week’s weekly support issues
  • No new issues this week.
  • Something to look out for: 2 new pieces of malware. 1 causes Firefox to crash. 1 tries to steal data entered when you log into a bank site.

Knowledge Base

• New articles: Crash when closing a Gmail tab, Bookmark folders
• Contacted metrics team about what changes make the greatest impact to CSAT score. We may have a phone meeting this week. (Haven’t got to details yet.) [1]
• Article for Facebook issues? [2]
  • What would we say? Clear you cache and cookies? Is that it? One Firewall is known to block a facebook script. If you see any more, post in the Contributors forums thread.
• One article for crashes caused by malware?
  • If we have a more specific answer for any certain crash, in that case we can give it a separate article.
  • It would also be good to add a section to the Firefox crashes article, that says the most common cause of crashes is third party software (extensions, plugins, malware).
  • Might be a good idea to write a blog post helping contributors learn how to analyze crash stats.

Forum

• User traffic in the Contributors forum could be coming from Google.[3] -cilias
  • After we switch to having pretty URLs for forum links, we can make robots.txt tell search engines to ignore the contributors forum.

Live Chat

• Most common chat tags, no major changes in the past two months. (Crashes/hanging is still the most common problem)

Roundtable

Support is one of the top and most discoverable ways that Firefox users can give feedback about their experiences with Firefox. While the primary focus of Firefox Support will always be make sure users have the best possible experience with Firefox (and the web), the SUMO project tracks trends and provides insights about our user base to the greater Mozilla community.

Common user questions

The primary thing we track at SUMO is commonly asked questions and commonly encountered issues. Not only does this help guide the support community, it also helps Firefox developers know what are some of the major pain points for our users. A bug that may not seem like a big deal can actually be a top support issue because it causes confusion or “feels wrong” to our users. Because support works so closely with users in the forum and on live chat, we’re often able to troubleshoot a problem, working with the user to identify the extension, plugin or setting responsible. This can help get traction on bugs where the development or QA teams are unable to reproduce. There will be a followup blog post discussing the feedback that SUMO has provided around common issues during the 3.5 launch.

Improving messaging and websites

Visitors to support are often not the most technically savvy, and in many cases are new to Firefox or the web. Since support volunteers work daily with these users, we often have a good feel for what kinds of messaging works and what can be confusing for these users. Over the past quarter, we’ve been providing feedback to the marketing team at Mozilla working on education efforts. This has helped them figure out which Firefox features to explain or clarify, for example.

A number of Mozilla webpages also link to SUMO. By tracking where these users go on SUMO, we can help figure out what questions people visiting those pages may have and provide feedback about how to improve those pages for better user retention. In particular, we’ve been tracking traffic from the Firefox download page to see if we can help remove roadblocks that new Firefox users may face that may prevent them from getting started successfully with their new browser.

Giving feedback on usage patterns and interface

Whenever there are changes to the Firefox interface, people who are used to a feature working in a certain way or looking like a certain thing get confused and come to support. For example, while most people familiar with computers will use keyboard shortcuts to copy and paste, many visitors to SUMO have trouble with that and will instead use the context menu or the menu bar. Similarly, the Send Link feature is widely used and people often come to SUMO when it doesn’t work as expected rather than just copying and pasting a URL to an email. On the other hand, there are features we don’t hear much about. For example, the Delete option in the Edit menu isn’t much used. SUMO has been able to provide some initial feedback to the developers working on user interface in Firefox 3.7 as they make decisions about how to streamline the future of Firefox.

These are just some of the ways that SUMO can help you learn about the Firefox user base. As we move forward, we’ll be looking for more ways we can use our insights to facilitate a closer working relationship between developers and teams at Mozilla and our users which everyone in the community can benefit from.

More importantly, as a resource for the community, we can help you get answers to any user and usage questions you might have about Firefox. Whether you’re localizing support articles on SUMO, running an independently hosted local support site, or if you work with QA, engineering or anywhere else, we’d love to help you. Just let us know what you need and we’ll do our best to provide answers.

The primary method of navigating the Firefox Support website is through search. For all the documentation we have on the website, it isn’t worth much if users cannot find what they are looking for. Earlier this year, we switched to a new search engine primarily because it gave us greater control of which items appeared in search results for any given search term.

Terms like “bookmarks” and “cookies”, which are too vague for us to know what is being asked, are especially important because they are among our top search terms.  If we know the most frequently asked questions, we can make sure the top search results for generic terms are representative of the most frequently asked questions associated with those terms (e.g. if the most common question about bookmarks is “How do I back up my bookmarks”, we should make sure the ‘Backing up and restoring bookmarks‘ article is the first result when searching for “bookmarks”).

Sometimes users use bad terminology as well. A common issue since Firefox 3 was released is that users want to clear items in the Location bar. After clearing browser history, the Location bar still contains bookmarks, which causes users to think that clearing history does not work.

Figuring out the most frequently asked questions among Firefox users is harder than you would expect. Data from the support forum and live chat is gathered on a weekly basis, and documented in our weekly common issues page. However we funnel users through the knowledge base before they reach the forum/live chat. The issues people are having there are good for finding new issues not covered in the knowledge base, but if the most frequently asked questions are already covered in the knowledge base, the most frequently asked questions are theoretically not being posted in the forum/live chat.

A couple months ago, Cheng, Matthew, and I went through our metrics and page view data to try get a good grasp on the most frequently asked questions (keeping all of the above in mind). Most of it was based on the “Top articles searched for” metric and “Top articles by score”, then we used the top searches and data from article feedback comments to fill out the rest of the list and clarify inconsistencies. We came up with this list. We can’t be sure that it is 100% accurate, but we think it is reliable enough to use for manipulating search results.

1. How do I use Private Browsing?
2. How do I enable cookies?
3. How do I set the homepage?
4. How do I clear my private data?
5. How do I clear the location bar?
6. How do I make Firefox the default browser?
7. Pop-ups are not being blocked.
8. Username and password is not remembered.
9. Firefox takes too long to start up.
10. How do I export bookmarks to an HTML file?

Using that list, we can list common search terms for each question, and make sure the articles that directly address those questions are tagged with those terms and have the right keywords.

Anyone who wants to help can try testing search terms to see which article appear first in the search results. If you don’t think the search results are optimal for your search terms and you know which articles should be more prominent, try editing the article to add the appropriate tags.

Sumo

• Weekly metrics
  • Massive increase in number of new SUMO accounts. Don’t know why yet. Let’s see if it’s the same next week.
  • Two new graphs: en-US KB CSAT and Helpful %
    • Our Helpful % is steadily going up, but the CSAT seems to constantly hover above 85% with a few spikes here and there
  • Bounce rate of http://support.mozilla.com/en-US/kb went up 4% — cilias to look into it (e.g. where do people go from the start page?)
• Last week’s weekly support issues
  • Add column for last week’s numbers and remove superfluos notes

Knowledge Base

• New articles last 2 weeks: Sorting bookmarks , Enabling and disabling cookies , Deleting cookies , Blocking cookies, Firefox crashes when working with PDF files , Crash signature – _eFOGetName, Firefox crashes due to Adware-DoubleD, Configuring McAfee Internet Security, How to find which folder a bookmark is in, plus at least 17 crash articles from the KB crash articles sprint.
• SUMO 1.3 made some notable changes to KB:
  • email notifications when your edit is rejected or approved, with text area for reviewer add review notes (or anything else like a thank you)
  • The ability to mark translations as out of date is now in the hands of the reviewer, not the editor.
  • articles are not allowed to be created with bad characters
  • translations inherit the additional category info from the en-US article
  • edit summaries are displayed when removing an ‘out of date’ warning
  • articles not yet in the KB can be renamed by any contributor
  • Firefox 2 content will not be ignored by SHOWFOR
• CSAT score over time
  • Our goal for this quarter was to get it up to 85% [current csat score]
• Thinking about how to get csat score higher in Q4:
  • List of items in Contributors forum thread.
  • Gather all ideas we’ve collected and talk to metrics team about which items would have the biggest impact
• Other things to think about:
  • What is different in Firefox 3.6, and what will need updating?
    • How much of it can we automate?
    • cilias to lead effort of keeping track of changes and articles affected by 3.6
  • What is different in Windows 7, and what will need updating for Windows 7 users?

Forum

• No updates

Live Chat

• Participation was down last week, with only 12 active helpers. A lot of the most active helpers are back in school/college now.
• Top chat tags from last week, crashes still most common issue
• Web client status: nearly code complete for 1.4
  • Will need help testing after changes land — will post in Contributors forum to ask for help. Cww + cilias to help as well.