SUMO Blog

Attendees: djst, cilias, zzxc, cww, nkoth

Sumo

• Weekly metrics
  • First week of collecting metrics from Omniture instead of Urchin, which explains some differences in the numbers
  • All team members set up with access to Omniture; need to schedule meeting with the one and only (Mr Ken Kovash) to get a good introduction
• Last week’s weekly support issues
  • Got poll count data from the Knowledge Base thanks to Laura. Need to integrate it with the rest of the data. Also, we probably want to look at No votes as well, assuming the majority of the people that vote No do so because they tried the instructions but they didn’t work. Moreover, the data is skewed because of the site traffic (e.g. in-product help articles get higher vote counts).

Knowledge Base

• Bugzilla: 2 new article requests [1][2], 1 article bug verified fixed [3]
• Disable Smart Location Bar article is renamed
• Which in-product articles need to change for Firefox 3.1? cilias to get back with info around this.

Support Forum

• More threads, less answers. We currently have around 55% of threads replied to, but no way of determining whether or not the replies were actually an attempt to solve the problem. For this, we need new performance metrics, which is part of SUMO 0.8. Specifically, we need to be able to tell “me too” replies apart from replies that actually try to help the original poster.

Live Chat

• Traffic down again, answer rate >60% — steadily improving as traffic calms down
• Common issues: in weekly issues thread
• New hours, Helping with Live Chat updated
• About 5 new accounts last week, 0 account approvals
• Need more people to connect after-hours to chat-support.mozilla.com (with any Jabber client)
  • Any instructions on how to connect with Jabber client?–cilias 16:48, 29 September 2008 (UTC)
• Anyone who can help us out in Live Chat this week, please sign up on the live chat coverage page to help us plan it better. Thanks!

The Support Forum is a way for users to ask direct support questions about problems not (yet?) covered in the Knowledge Base, as described previously as the support funnel. It is also an important place to detect new or emerging issues and make sure those issues are dealt with in the best possible way.

The Firefox 3.0.2 release is a very good example of this importance; shortly after the 3.0.2 update started to roll out in Europe, Carsten Book from the QA team noticed early signs of users having problems with the Password Manager from various places such as support forums on Heise, and Hendrix. It was then quickly confirmed to be a frequent issue in the Support Forum of SUMO as well.

In a matter of hours, this went from early signs from our different user channels to a solid plan to release an update of Firefox (due out today or tomorrow) to fix it, as well as a new Knowledge Base support article about the issue. SUMO was just one player in this great and efficient teamwork, but it highlights well our ability to “extend our tentacles” during releases to ensure that we keep doing what we do so well: listening to our users — in this case as an early warning system, if you will.

For contributors, the Support Forum is an exciting place to be, since you can take part of this early warning system, while at the same time making other people happy by helping them with their problems. It’s a good place to start helping since you can simply browse through the posted questions and see if there is anything you know the answer to. There are no obligations to respond if you don’t know the answer.

Reaching out to the people in need

One of our current challenges with the Support Forum is that while we have an incredibly helpful community answering a large number of questions, we don’t have a good way of ensuring that the user asking the question actually comes back to read the answer. This means that we don’t really know if the user’s problem was actually solved or not, only that we at least tried to help.

Question asked in the Support Forum.

We’d like to change this by making it easier for users to find their way back to the forum thread they posted. When a user asks a question in the Support Forum, we should offer the ability to get an e-mail notification when someone responds to the question. This way, people don’t have to worry about how our support site works or what page to bookmark — we will reach out to the user instead of the user finding its way back to us.

Making the forum easier to use for our community

Of course, in order to reach out to our users we need to make sure there are people who want to help others in our forum. We already have a group of really amazing people in the forum who help a lot of the users who turn to the forum with their Firefox support questions. However, we definitely need more contributors to cope with the load.

One important thing to focus on in order to make it more pleasant to hang out in the forum is to make the site easier to use for our contributors. We have a few ideas:

• Give contributors credit for solving problems — Allow the user posting the original question to tell us whether or not an answer from a contributor solved the problem, giving contributors credit for their help. Readers of the earlier parts of this blog series might remember the discussion about a contributor karma system in part 3.
• Highlight the solution to a problem — Some forum threads quickly grow long, making finding the actual solution to the posted problem hard to find. Other people who stumble across the thread in search for a similar (or same) problem typically want to get straight to the solution, which could be buried on page 4 of a very long thread. We should make that solution easily discoverable at the start of the thread, by allowing the original poster of the thread to mark a particular response in it as the solution to the problem. This will benefit not just our users, but our contributors as well.
• RSS feed for incoming support questions — Instead of having to log in to SUMO in order to read the latest incoming support questions, a convenient alternative would be to just subscribe to an RSS feed (also known as Live Bookmark in Firefox). This way, people could set their mail client (e.g. Thunderbird) to notify whenever a new question is posted.
• Smart forum thread filters — When browsing the forum today, all questions are listed, including the ones that are already answered. Many contributors are more interested in looking at threads that haven’t been answered yet. Others might want to see only the threads they’ve responded in. We should allow for an easy way to filter the forum view based on what’s relevant for our visitors.
• Localization — This is a big one. We currently only have an English support forum and we’re starting to get more questions posted in other languages as the visibility of SUMO increases. For the locales that have enough contributors a need for it, we should definitely create a support forum in that language.

What other ideas can you think of that would make the forum more pleasant to use for you? We would love to hear from you about your experiences with the SUMO Support Forum so we could use your feedback to improve the site for everyone.

Attendees: djst, cilias, zzxc, cww, lucy

Sumo

• Localizer feedback
• Weekly metrics
  • Less traffic this week, and less community involvement
• Weekly support issues
  • For Hendrix, we have an experimental database built now that sorts each post into a category so it’ll be easy to pull up the raw data now and collect information with a simple SQL query. Cww will be putting in all the info from the support forums as well pending bug 455790. Once this is done, we’ll have a form where we can look up everyone who has a certain issue.
  • By far the most common issue right now is people being confused about the Awesome Bar not appearing to erase their history
    • Evidence that many people report this in the Forum as well. Many people are asking how to downgrade to Firefox 2.0.0.x
    • We have a downgrading to Firefox 2 article, but we need to add a paragraph explaining why downgrading is a bad idea, and how people can disable the Awesome Bar functionality anyway if they really need to

Knowledge Base

• Bugzilla: no activity (all around pretty quiet week)
  • remember to add user-doc-needed to bugs that require changes in documentation or new documentation
• Add-ons policy
  • Policies page is updated, but could improvement. [1]
  • New content block: label=amo
  • Add-on links have been removed from most articles with some exceptions. Dynamic content still needs to be applied. cilias will work on implementing this for related articles this week.
• List of top articles for translation: [2]
• Looking for suggestions on how to get more approvers to review other people’s edits [3]
  • djst thinks we need to make the review queue more obvious to contributors — need a good contributor start page and make sure we direct people who log in to that page
• We should remove instructions on how to completely disable all history in the Awesome Bar. Very few people actually want that, and we also have evidence of people thinking they broke Firefox when applying that “fix”

Support Forum

• Traffic slightly down, by ~ 15-20%

Live Chat

• Traffic down; chat requests down almost 30%
• 2 new accounts created, 0 account approvals
• New common issues: crashes (crash-stats still down, bug 455999), yahoo toolbar causes window jumping, Firefox 3 “crashes entire computer”, default browser corrupted/wrong browser launches on Windows

A few months ago, we surveyed knowledge base contributors to evaluate the quality of the contributor experience and how we can improve it. From the feedback we got, one common area of concern was in the article editor; so we’re using this as an opportunity to redesign the article editor that will make contributing to the knowledge base faster and less confusing.

I’ve started a discussion in the Support Contributors forum soliciting feedback and general brainstorming criteria for a new article editor. I’ve included some of the criteria based on the survey feedback, such as making the editor WYSIWYG, and tools to make features like SHOWFOR and Dynamic Content easier to use.

Have you been editing or writing new articles in the Knowledge Base? What did you think about the process and was there anything you’d like to change? We’d love to hear your feedback. Please post your ideas in the discussion thread. Thanks!

The Knowledge Base is the heart of SUMO and consists of a large collection of support articles largely written by the community. The key focus and scope of the Knowledge Base is troubleshooting — solving people’s problems with Firefox — as well as teaching people how to use Firefox and, by extension, the web.

The Knowledge Base is meant to be the primary method of getting support, since it already contains the answers to the most common Firefox issues. Because of this, the articles in the Knowledge Base need to be:

1. easy to find,
2. easy to understand, and
3. easy for contributors to edit.

Let’s cover them one by one — easy as 1-2-3!

1 – Better search

Searching is central to the support funnel –- we want people to search for their problems first. Because of this intended flow through the funnel, it’s critical that the search function is working well and serving the right results.

There are some problems with our current search engine. Most importantly, we don’t have fine-grained control over its behavior, and we can’t control the frequency of its reindexing (the process of making sure all articles are searchable). This means that it might take weeks after a new article is created until it appears in the search results, which is really not ideal.

• The search engine should index the Knowledge Base article on a daily basis to ensure up-to-date search results.
• When a new article is published, the search engine should automatically re-index to include it. Being able to quickly ensure new information is accessible to our users is critical if e.g. a new issue is discovered after a new release.
• Search rating should primarily be based on the frequency of the reported problem, the article tags it contains, and its description. It should also support manual tweaking to ensure new and critical issues are getting a top search result listing even before they’re naturally picked up by popularity.

We could also do a better job of integrating the provided solutions to problems from the Support Forum with the search results. For example, when a user searches the Knowledge Base, we could offer a simple way to extend the search to include answered threads in the forum as well.

Another area of improvement is with locale specific searches. Today we include matches in English articles too, but there’s no clear distinction between the content available in the local language and the English (still untranslated) content.

All in all, this should make articles easier for users to find.

2 – Screencasts

Some users resort to the Support Forum or Live Chat simply because they don’t understand the contents of a Knowledge Base article and need some hand-holding to solve their problem. Screencasts (essentially videos) is a powerful visual medium that helps translate and communicate technical concepts to a wider audience. Even for users who are perfectly capable of following written instructions, watching a screencast can speed up the problem solving process significantly.

Frequent readers know that we already have a bunch of screencasts ready to be used in the Knowledge Base — we just need to get the infrastructure up so we can actually provide them to our readers. Specifically, we need:

• an easy way to upload and embed screencasts into Knowledge Base articles, and
• an elegant way of presenting them in the articles themselves, for example a simple “Show me how” link, expanding into a full screen video view upon clicking; no annoying pop-up windows; all embedded into the page à la Web 2.0.

Once we have this up and running, articles should be easier for users to understand.

3 – Streamlined article editing

Since the Knowledge Base is our most powerful tool to provide efficient support for our users, it’s equally important that this tool is as simple for contributors to use as possible. We have some ideas:

• What You See Is What You Get (WYSIWYG) editing — editing a Knowledge Base article should be as simple as writing a blog post. This is easier said than done, but it’s definitely worth investigating the possibilities, as it would dramatically simplify things for casual Knowledge Base contributors that just want to correct a typo without having to learn a whole new wiki syntax language.
• No-nonsense editing process — remove buttons and options that are just confusing. Right now there are a lot of them, and they’re laid out in a somewhat confusing way.
• Other ideas? This is where you come in. Chris Ilias started a thread to discuss this item more thoroughly, and we’d love to hear your feedback!

With these improvements, Knowledge Base articles should be easier for our contributors to edit, allowing more people to participate.

And that ends part six. As simple as Do Re Mi.

Approximately half of our users are running an English (US, British, or other) version of Firefox. This is why localization is such a critical part of Mozilla, because it enables us to reach that other (significant!) half.

Shrep presenting percentage of Firefox locales. Photo (c) Martin Creutziger.

Localizing SUMO is not just about translating English articles to another language — it’s about making the Firefox support experience truly local, while also participating in the mission to further refine the quality of our support.

Let’s take an example: a localizer might start translating an English article into her own language, but while doing that she spots an error in the English article that needs to be corrected. By making that correction in the English article herself, or by notifying an English article editor (e.g. by posting feedback at the bottom of the article), the overall quality of our work improves, which of course everyone benefits from.

Another example is if the localizer is able to express herself in a clearer way than the original English article reads. In this case she could make sure that improvement is shared with the English half of our users in a similar way as described above. The Knowledge Base is far from perfect — there are lots of opportunities to improve the readability of the articles, and localizers have a great opportunity in this ongoing work by either writing a better localized version (and inform English contributors about the improvements), or by simply updating the English article directly.

One aspect of localization in the context of support is that some support topics only really apply to specific locales. For example, a popular German news site (e.g. spiegel.de) might have problems working correctly in Firefox for some reason (highly unlikely, considering the popularity of Firefox in Germany). In this case, a German SUMO contributor could just create a Knowledge Base article for that problem even though it doesn’t yet exist in English. Depending on the nature or severity of the problem, an English article might still be warranted, but the German localizer shouldn’t feel obligated to do that work him/herself.

Of course, for people that are comfortable with just translating an English article to their own language, SUMO can help there as well by providing tools that help determine which articles are the most important to translate. By sharing the weekly stats about the most commonly reported problems or requested features with localizers, we should be able to provide a clear overview of which articles to focus on first.

Clear locale overview

Speaking of a clear overview, here are some specific things we can do to improve SUMO for localizers:

• It should be easy to see the translation status of a SUMO locale. A localizer should be able to log in and be presented with a dashboard summarizing the overall status of the contents available for his or her locale. We should use things like pie charts, colors, tables, etc to paint a clearer picture to everyone participating.
• The priorities should be clear — it should be easy for localizers to see which articles should be translate first. If there are some articles that users read more often than others, these articles should be clearly marked as in need of translation.

Other improvements?

What more can we do to make the localization experience as smooth as possible? Probably a lot of things. We should definitely make sure the UI of the site is fully and easily localizable using established file formats (*.po files anyone?), and more generally, we should connect with localization teams and individuals to gather more feedback.

If you’re a localizer, please help us by providing your feedback; it will really help us verify that we’re on the right track and thus allow us to move forward faster. Thank you in advance!

Sumo

• Weekly metrics
  • Most popular articles list not accurate (see bug)
• This week is the first time we share the weekly common issues list during the SUMO meeting. It’s a chance to make sure everyone can provide their feedback and additional info, and verify that we’re seeing the same things. This way, we have a better list to present to the rest of Mozilla on the Wednesday Firefox 3.1 meeting.

Knowledge Base

• Bugzilla: 2 article bugs filed [1][2], 1 article bug verified fixed [3]
• Still gathering feedback on new editor.
• New start page for Firefox Support localizers [4].
• Policy on linking to add-ons discussion in mozilla.support.planning [5].
  • We don’t want to link directly to add-ons, as that gives a false impression that we’re supporting them
  • We should, however, promote one of the strongest features of Firefox — it’s extensibility. We can do that without endorsing any specific extensions.
• What to do with out of date/obsolete pages — cilias to start a discussion this week

Forums

• Forums are still slow but not throwing database errors.
• Participation is up this week, yay!
  • 775 replies by contribs, this week 1775
  • cww to verify the stats, as it looks like the participation doubled…

Live Chat

• zzxc is without power today, Cww in charge of livechat for the day, pulling stats.

Norton 360

• Main thread: [6]
• Call with Symantec on Wed to finalize suggested workarounds. cww and zzxc to call in.
• cilias to review zzxc’s edit to the Norton 360 article

I’ll start part four of this blog series with an important question: Are our Firefox users happy about their browsing experience? If they are, exactly how happy are they? Can we measure that somehow? If they’re actually not happy, can we figure out why so we can do something about it?

By looking at the right set of metrics, SUMO can really have a finger on the pulse of the large Firefox user base. Analyzing and understanding the data we’re gathering is critical to the continued success of Firefox and will undoubtedly benefit the Mozilla project as a whole.

SUMO is a great source of information here because it’s the obvious channel for mainstream users to look for support and documentation. We are already analyzing the data we’re gathering, but we’re not really there yet. Some information that would help paint a clearer picture:

• Track search terms that have increased in popularity in recent time, rather than just looking at the fairly static list of top search terms as we’re doing today. That could give us early warnings about emerging issues or trends.
• Thorough data mining –- if we dig deeper to understand how our users use the site by tracking data like “Which articles did users read and what did they search for before ending up in the forum/chat?”, we would gain a much better understanding of how well our SUMO website works for our users. This knowledge could give us a clear idea of how to improve Knowledge Base articles and search results.

Today we’re doing a lot of the information gathering manually. To speed this up and allow us to focus on analyzing the data instead, we should automate the information gathering as much as possible:

• Automatically track specific keywords related to “badware” (spyware, adware, trojans, etc.) to ensure we can become proactive against new issues.
• Add alarm triggers when signs of unusual traffic emerges, e.g. excessive reports of tracked keywords.
• Automatically collect metrics to be analyzed daily/weekly.

Measuring SUMO performance

There are many things we can measure to get a better understanding of our strengths and areas of improvement, such as:

• average rating of Knowledge Base article (poll “Was this article easy to understand?”)
• average response waiting time in the Support Forum
• ratio of threads resolved in the Support Forum
• average Live Chat session length and waiting time
• ratio of Live Chat sessions resolved

We are already measuring some of these, but not all of them. Apart from getting all this data in the first place, we should put more focus on presenting it to our contributors, and analyzing it to better understand how we can continuously improve our support and as a result get happier users.

Measuring user happiness

Back to the original question: are our users happy, and how can we measure it? Well, what we can do is measure how well SUMO meets our users’ expectations. If we’re doing well according to our users, that means we have happy users.

To get an overview of our quality of support and whether or not it’s improving over time, we should start measuring customer satisfaction, but since we don’t have customers, let’s just call this user happiness instead! With this data, we will be able to tell how well we meet or surpass our user’s expectations. It can also indicate how well the various components of the site perform.

• The data could show a total average of our user’s happiness level with SUMO as a whole, but it could also show per-component (KB/forum/chat), and even per-article (KB) data, to give us both an overview of, and detailed insights about how well we’re doing.
• The data should be collected automatically and shared quarterly with e.g. marketing team, who are also keen on getting deeper insights about our user satisfaction.

With all of the above, the SUMO community should be able to tell — with confidence — whether or not our users are as happy as we want them to be.

Attendees: djst, cilias, zzxc, cww, nkoth, myles7897

Sumo

• Welcome to the team, Matthew (zzxc) and Cheng (cww)! [1]
• Gathering/reporting top issues
  • cww has been reading forum threads and e-mail
  • zzxc has helped with Live Chat logs
  • Need more people to monitor the data, especially the forum — cww to approach forum moderators about it first
  • We need to get the hard numbers of the poll “Did this article solve a problem?” — current % stats is useless to determine which are our most common problems — cww to ask np about getting a quick query run on a weekly basis until we can get this implemented and run automatically
  • Idea: categorize problems similar to the RRRT did it
• Weekly metrics
  • When will we be using Omniture? Need to get team members set up to access it, then learn the system. Possibly schedule a quick meeting with our metrics guru Ken to get our questions answered. :)
• Norton 360 (bug 452469) – new instructions up in Bookmarks and toolbar buttons not working after upgrading based on info from tomcat
  • 30% of people gets their problem solved with that article.
  • Working with QA to investigate this further. Rebooting works temp. firefox.exe never closes.

Knowledge Base

• Bugzilla: 4 new article bugs [2], 2 article bugs fixed [3][4]
• Need to work on print problems article – cilias & zzxc to work together to make sure we’re covering all issues people are reporting
• In case you missed it We now have notification of when a new translation is created.
• A How to Contribute start page for localizers would be nice, listing things like how to translate the interface, articles, enabling new article notifications, etc — cilias to start working on this
• Gathering feedback and suggestions for new article editor. [5]

Forums

• High traffic
• Really slow performance
• Most Popular Thread list now only includes the last few weeks’ popularity, which actually makes the list useful :) Should be renamed Most visited Threads.
• RSS feed for a whole forum, similar to Mozilla’s intranet forum? Functionality exists in TikiWiki, but we need to test it for db load etc. Cww to file a bug.

Live Chat

• Traffic still high from MU release, helpers get burned out quickly, ca 822 chats
• 56% of chats answered — new record since Firefox 3 release!
• 6 new accounts last week
• Notes/status on a sample of ~70 chats
  • Top issues: bookmarks in location bar, Norton 360 locks places.sqlite, Firefox crashes, Firefox will not start, some printer drivers print garbage text, lost bookmarks
• Contributors conference now proxied to #sumo thanks to mzz’s bot

There has been some changes to the SUMO team recently. Jason Barnabe, who maintains the Support Forum, unfortunately doesn’t have enough time to maintain a full-time job and administrate the Support Forum at the same time. He has made awesome contributions to the project over the year, and we’re incredibly thankful for that. Fortunately, he will remain an active SUMO contributor, so we certainly haven’t seen the last from him!

Also, our Live Chat maintainer Majken Connor (most of the SUMO contributors know her as “Lucy”) is moving on to explore other opportunities. We have all valued her contributions to the project and we wish her every success.

With that said, I’m very excited to announce the new team:

• Chris Ilias remains the “keeper of the Knowledge Base,” as he likes to call it. He is responsible for, among other things, reviewing article edits by the community, training and answering questions from contributors, improving our contributor documentation, gathering article feedback and statistics and ensuring that it is shared with the community.
• Matthew Middleton is the new Live Chat maintainer. Matthew started as a Live Chat volunteer back in January, and since then he’s become more and more active. He has actually been a Live Chat administrator in over a month now, but will now take an even more active role withing the SUMO community. He has impressed a lot of people at Mozilla recently with his incredible dedication, so I’m very happy to have him as part of the SUMO team.
• Cheng Wang is the new Support Forum maintainer, and will work on various things in the project, like administrating the forum, room monitoring Live Chat shifts, and collaborating with the rest of the team and even other parts of Mozilla like the QA team, to figure out our most commonly reported issues with Firefox. Cheng also started contributing back in January and has already made a huge difference by introducing new volunteers and helping the planning and organizing of events like the Support Firefox Days.

This picture was taken from the Mozilla Summit in Whistler, BC. From left to right: Nelson Ko, Jason Barnabe, Laura Thomson, Matthew Middleton, Majken Connor, me, Cheng Wang, Brian Krausz, and Chris Ilias.

Together we will work hard to make our users happy, grow our community, interact with other teams at Mozilla to share insights and information, and shape what we think will become the future definition of Open Source Support.

Stay tuned for part four of The vision for SUMO…