SUMO Blog

Sumo

• Weekly metrics
  • cilias will work with metrics team to figure out why our bounce rate is increasing
• Last week’s weekly support issues
  • Fewer repeated issues
  • Dramatic increase in searches related to “cookies”
    • cww to work with metrics to figure out what we can tell about the people who search for “cookies”, e.g. OS, browser, country, etc.

Knowledge Base

• New articles: Using Firefox with a 64-bit Operating System, Crash signature – nsEventListenerManager::Release(), Bookmarks toolbar, Crash signature – js_LockGCThingRT
• CSAT has taken another dive. I still need to assess the cause.
• Had meeting with Metrics team last week about prioritizing ideas for improving the CSAT score in Q4.
  • We need to do some A/B tests to determine the impact each solution, then we can determine which solutions are most important.
  • cilias needs to make sure they have the raw data from the past year, so they can help determine the reasons for fluctuations in the CSAT score.
  • Will try to figure out why the CSAT is dropping in non-en-US locales (which locales are causing it?)

Forum

• Participation fell. Need to investigate if it’s related to new forum AAQ
• [1] New forum start page (temp)
  • Cww to post in Contributor forum once it’s back up and ask for feedback

Live Chat

• Tags and issues from last week
  • Problems caused by add-ons now #2 issue
• Web client status
  • Behind schedule, but will be online to meet Q3 goal
    • As soon as it’s up on prod, will announce
  • Will be on production server before or shortly after this call (link)
  • Future development will focus first on integrating canned responses and SUMO search/login into the client, then on the remaining features in the PRD.

Sumo

• Weekly metrics
  • Start page bounce rate theory from last week: large increase in referrers from Firefox whatsnew pages.
  • CSAT score on Forum and live chat went up. Not sure why.
  • Top articles block was updated last week, to reflect top questions.
• Last week’s weekly support issues
  • No new issues this week.
  • Something to look out for: 2 new pieces of malware. 1 causes Firefox to crash. 1 tries to steal data entered when you log into a bank site.

Knowledge Base

• New articles: Crash when closing a Gmail tab, Bookmark folders
• Contacted metrics team about what changes make the greatest impact to CSAT score. We may have a phone meeting this week. (Haven’t got to details yet.) [1]
• Article for Facebook issues? [2]
  • What would we say? Clear you cache and cookies? Is that it? One Firewall is known to block a facebook script. If you see any more, post in the Contributors forums thread.
• One article for crashes caused by malware?
  • If we have a more specific answer for any certain crash, in that case we can give it a separate article.
  • It would also be good to add a section to the Firefox crashes article, that says the most common cause of crashes is third party software (extensions, plugins, malware).
  • Might be a good idea to write a blog post helping contributors learn how to analyze crash stats.

Forum

• User traffic in the Contributors forum could be coming from Google.[3] -cilias
  • After we switch to having pretty URLs for forum links, we can make robots.txt tell search engines to ignore the contributors forum.

Live Chat

• Most common chat tags, no major changes in the past two months. (Crashes/hanging is still the most common problem)

Roundtable

Support is one of the top and most discoverable ways that Firefox users can give feedback about their experiences with Firefox. While the primary focus of Firefox Support will always be make sure users have the best possible experience with Firefox (and the web), the SUMO project tracks trends and provides insights about our user base to the greater Mozilla community.

Common user questions

The primary thing we track at SUMO is commonly asked questions and commonly encountered issues. Not only does this help guide the support community, it also helps Firefox developers know what are some of the major pain points for our users. A bug that may not seem like a big deal can actually be a top support issue because it causes confusion or “feels wrong” to our users. Because support works so closely with users in the forum and on live chat, we’re often able to troubleshoot a problem, working with the user to identify the extension, plugin or setting responsible. This can help get traction on bugs where the development or QA teams are unable to reproduce. There will be a followup blog post discussing the feedback that SUMO has provided around common issues during the 3.5 launch.

Improving messaging and websites

Visitors to support are often not the most technically savvy, and in many cases are new to Firefox or the web. Since support volunteers work daily with these users, we often have a good feel for what kinds of messaging works and what can be confusing for these users. Over the past quarter, we’ve been providing feedback to the marketing team at Mozilla working on education efforts. This has helped them figure out which Firefox features to explain or clarify, for example.

A number of Mozilla webpages also link to SUMO. By tracking where these users go on SUMO, we can help figure out what questions people visiting those pages may have and provide feedback about how to improve those pages for better user retention. In particular, we’ve been tracking traffic from the Firefox download page to see if we can help remove roadblocks that new Firefox users may face that may prevent them from getting started successfully with their new browser.

Giving feedback on usage patterns and interface

Whenever there are changes to the Firefox interface, people who are used to a feature working in a certain way or looking like a certain thing get confused and come to support. For example, while most people familiar with computers will use keyboard shortcuts to copy and paste, many visitors to SUMO have trouble with that and will instead use the context menu or the menu bar. Similarly, the Send Link feature is widely used and people often come to SUMO when it doesn’t work as expected rather than just copying and pasting a URL to an email. On the other hand, there are features we don’t hear much about. For example, the Delete option in the Edit menu isn’t much used. SUMO has been able to provide some initial feedback to the developers working on user interface in Firefox 3.7 as they make decisions about how to streamline the future of Firefox.

These are just some of the ways that SUMO can help you learn about the Firefox user base. As we move forward, we’ll be looking for more ways we can use our insights to facilitate a closer working relationship between developers and teams at Mozilla and our users which everyone in the community can benefit from.

More importantly, as a resource for the community, we can help you get answers to any user and usage questions you might have about Firefox. Whether you’re localizing support articles on SUMO, running an independently hosted local support site, or if you work with QA, engineering or anywhere else, we’d love to help you. Just let us know what you need and we’ll do our best to provide answers.

The primary method of navigating the Firefox Support website is through search. For all the documentation we have on the website, it isn’t worth much if users cannot find what they are looking for. Earlier this year, we switched to a new search engine primarily because it gave us greater control of which items appeared in search results for any given search term.

Terms like “bookmarks” and “cookies”, which are too vague for us to know what is being asked, are especially important because they are among our top search terms.  If we know the most frequently asked questions, we can make sure the top search results for generic terms are representative of the most frequently asked questions associated with those terms (e.g. if the most common question about bookmarks is “How do I back up my bookmarks”, we should make sure the ‘Backing up and restoring bookmarks‘ article is the first result when searching for “bookmarks”).

Sometimes users use bad terminology as well. A common issue since Firefox 3 was released is that users want to clear items in the Location bar. After clearing browser history, the Location bar still contains bookmarks, which causes users to think that clearing history does not work.

Figuring out the most frequently asked questions among Firefox users is harder than you would expect. Data from the support forum and live chat is gathered on a weekly basis, and documented in our weekly common issues page. However we funnel users through the knowledge base before they reach the forum/live chat. The issues people are having there are good for finding new issues not covered in the knowledge base, but if the most frequently asked questions are already covered in the knowledge base, the most frequently asked questions are theoretically not being posted in the forum/live chat.

A couple months ago, Cheng, Matthew, and I went through our metrics and page view data to try get a good grasp on the most frequently asked questions (keeping all of the above in mind). Most of it was based on the “Top articles searched for” metric and “Top articles by score”, then we used the top searches and data from article feedback comments to fill out the rest of the list and clarify inconsistencies. We came up with this list. We can’t be sure that it is 100% accurate, but we think it is reliable enough to use for manipulating search results.

1. How do I use Private Browsing?
2. How do I enable cookies?
3. How do I set the homepage?
4. How do I clear my private data?
5. How do I clear the location bar?
6. How do I make Firefox the default browser?
7. Pop-ups are not being blocked.
8. Username and password is not remembered.
9. Firefox takes too long to start up.
10. How do I export bookmarks to an HTML file?

Using that list, we can list common search terms for each question, and make sure the articles that directly address those questions are tagged with those terms and have the right keywords.

Anyone who wants to help can try testing search terms to see which article appear first in the search results. If you don’t think the search results are optimal for your search terms and you know which articles should be more prominent, try editing the article to add the appropriate tags.

Sumo

• Weekly metrics
  • Massive increase in number of new SUMO accounts. Don’t know why yet. Let’s see if it’s the same next week.
  • Two new graphs: en-US KB CSAT and Helpful %
    • Our Helpful % is steadily going up, but the CSAT seems to constantly hover above 85% with a few spikes here and there
  • Bounce rate of http://support.mozilla.com/en-US/kb went up 4% — cilias to look into it (e.g. where do people go from the start page?)
• Last week’s weekly support issues
  • Add column for last week’s numbers and remove superfluos notes

Knowledge Base

• New articles last 2 weeks: Sorting bookmarks , Enabling and disabling cookies , Deleting cookies , Blocking cookies, Firefox crashes when working with PDF files , Crash signature – _eFOGetName, Firefox crashes due to Adware-DoubleD, Configuring McAfee Internet Security, How to find which folder a bookmark is in, plus at least 17 crash articles from the KB crash articles sprint.
• SUMO 1.3 made some notable changes to KB:
  • email notifications when your edit is rejected or approved, with text area for reviewer add review notes (or anything else like a thank you)
  • The ability to mark translations as out of date is now in the hands of the reviewer, not the editor.
  • articles are not allowed to be created with bad characters
  • translations inherit the additional category info from the en-US article
  • edit summaries are displayed when removing an ‘out of date’ warning
  • articles not yet in the KB can be renamed by any contributor
  • Firefox 2 content will not be ignored by SHOWFOR
• CSAT score over time
  • Our goal for this quarter was to get it up to 85% [current csat score]
• Thinking about how to get csat score higher in Q4:
  • List of items in Contributors forum thread.
  • Gather all ideas we’ve collected and talk to metrics team about which items would have the biggest impact
• Other things to think about:
  • What is different in Firefox 3.6, and what will need updating?
    • How much of it can we automate?
    • cilias to lead effort of keeping track of changes and articles affected by 3.6
  • What is different in Windows 7, and what will need updating for Windows 7 users?

Forum

• No updates

Live Chat

• Participation was down last week, with only 12 active helpers. A lot of the most active helpers are back in school/college now.
• Top chat tags from last week, crashes still most common issue
• Web client status: nearly code complete for 1.4
  • Will need help testing after changes land — will post in Contributors forum to ask for help. Cww + cilias to help as well.

Last Thursday, the bug fixes for SUMO 1.3 were pushed to support.mozilla.com. The big focus was on productization, and we also managed to add some other great changes.

So what is productization? From the beginning of the project, the plan was to create a support solution for Firefox, but do it in such a way that makers of other software could use SUMO for their own support sites. In the future, Fennec (the mobile version of Firefox) will have its own support site, using its own implementation of SUMO. In SUMO 1.3, major work was done on separating the code that is specific to Firefox Support from code that can be used for other products. In addition, work was done on turning the code into something you can install yourself. If you would like to try it yourself, we have instructions on wiki.mozilla.org.

Somehow, in all of that work, we managed to add a significant new feature: Article review email notifications. We’ve said in many previous blog posts that our knowledge base works differently from other wikis. All edits go through a review process before being public. For new contributors, who have not visited the Contributors forum, this is a major point of disconnect. When reviewing edits, it is important to state why you are approving or rejecting an edit. Even a simple “Thank you” message can go a long way.

In SUMO 1.3, we take an approach similar to posting to a moderated mailing list. When a contributor edits an article, an email is sent to the contributor, saying that their edit is waiting for review. The reviewer can approve or reject the edit, and a text box is provided for the review to give their feedback to the contributor. When the edit is reviewed, an email is sent to the contributor, telling them the result of the review, including the message from the reviewer. Just as important is the fact that all of these messages include an invitation to the Contributors forum.

This new review notification feature should make a huge difference in growing the SUMO community. Instead of newcomers making one edit and leaving (oblivious to the fact that their edit is not public yet), this gets them thinking about how SUMO works, connects them to the rest of the community, and gives us a chance to personally thank each contributor.

The full list of SUMO 1.3 bugs is on Bugzilla. This release has over 90 bugs, which is the most ever in a SUMO release (around three times more than most SUMO releases). A big thank you goes to Paul Craciunoiu, James Socol, Eric Cooper, Stephen Donner, Vishal Kamdar, Krupa Raj, and SUMOdev lead Laura Thomson for an extraordinary amount of work. Not to mention IT for all their work with the release push. If you have any questions, please feel free to post them in the Contributors forum, or join the #sumodev IRC channel to thank them yourself.

This past Thursday, we had a knowledge base sprint to write articles for the top Firefox crashes. It was a resounding success with 20 articles written on the day, covering the most common crashes experienced by Firefox users. Not only will this help Firefox users searching the knowledge base, this will also help contributors coming across an unfamiliar crash signature in the forums or live chat.

The main sprint wiki page links to all the articles that were written.

A special thanks goes to Jesse Ruderman, a Firefox developer, who took the whole day to work on this sprint, writing articles, analyzing bugs and helping pinpoint the causes of any crashes so we could give up-to-date suggestions in the knowledge base. We also had help from SUMO contributors and people from around the Mozilla project who read drafts and discussed crash reports.

Thank you!

Sumo

• Weekly metrics
  • Lots of activity in the German locale
  • High traffic in the forum due to the major update
  • We should look at search stats for crash signatures.
    • Crash article is rated low in understandability. We should reorganize it and make it clearer that finding crash signatures is fairly advanced, but isn’t the only way to get help.
• Last week’s weekly support issues
• How could Firefox make troubleshooting user problems easier?
  • Discussion/brainstorming that led to this wiki page
• SFD update
  • Went through all top crashers last Friday and made sure bugs were filed for them

Knowledge Base

• Specific articles about Bookmarks and Cookies are being worked on, and hopefully in the KB early this week.
• New articles: Deleting bookmarks, Will Firefox work on Snow Leopard
• Note about Snow Leopard: Screencast recording is fullscreen-only. SUMO has a size limit of 640×480.
• Should we remove the “Print” link in the Actions box? [1]
  • Remove it for now — very few people using it anyway and it’s broken

Forum

• Lots of support threads in the Contributors forum, something happen?
  • cilias updated the How to Contribute document in an attempt to reduce the problem, but it might have had an opposite effect
• Jump in traffic to forums this week

Live Chat

• Top tags from last week
• change diagram to a stacked bar graph [2]
  • Lots of connection problems caused by McAfee last week

Roundtable

• Support common-issues triage tomorrow at 10 AM.
  • This will be a weekly effort to ensure that the keyword is accurate and reliable
  • Sumo team will always call in, others are very welcome too
• New SUMO blog theme proposed by leo.

One of the most frustrating thing for users is to have Firefox crash on them. As we’ve discussed before, we’ve been writing a series of articles to document the top crashes and get them searchable by crash signature.

Just documenting the top 50 crash signatures will greatly improve the experience of users coming to SUMO. It will also help give us somewhere to point users to from the forums or live chat. Towards that end, we’ll be having a one-day sprint to write these articles and get them into the knowledge base and we could use your help!

On September 3rd, we’re going to get together to write as many of these crash articles as we can. We’ll have members from Mozilla’s QA team as well as platform and branch development teams on hand over IRC to help with understanding crash bugs or stacks and answer any questions you may have about the crashes. We’ll be tracking our progress on this page.

We look forward to seeing you there!

Sumo

• Weekly metrics
  • “cookies” is the most popular search term
  • http://support.mozilla.com/kb/Websites+say+cookies+are+blocked
    • On that article, for the question “Did this article solve a problem you had with Firefox?”, 111 people votes yes last week, 160 voted no
  • Not clear why people are having this problem recently — keeping eyes and ears open
  • cilias to parse through the article feedback of Websites say cookies are blocked to see if we can get any clues there
  • Live Chat: 25% of all cookie issues were about Yahoo, all were fixed by clearing the cookies
• Last week’s weekly support issues

Knowledge Base

• Attn: bug 505356 “Open all external links in KB articles in new window/tab” -Milos
  • Annoying when you read a support article, and you want to read a .. right-click
• Bookmarks reorganization — Bo is going to start filing bugs this week for new task-specific articles.
• Thanks to Bo, all top 5+ crash articles are in the KB. One new one this week. Big thanks to Bo!
  • cilias to test searches to make sure they work (including non-en-US locales)
  • cww to see how we can add “crash” to the Crash Report form as an OR search, to ensure that the generic “Firefox Crashes” article is always included in the results
• This week we’ll focus on getting the Bookmarks and Cookies articles split.
• KB CSAT took a dive in March, but I don’t know the cause
  • There were three firedrills in April, which was the second month of drop, but that doesn’t explain March
  • djst to ask Patrick and Ken about this to see if there are other similar drops in March-April other than SUMO CSAT
• PRD draft for KB article requests, tracking, and discussion. (Discussion thread)
• Looking for feedback from localizers: Simplifying how to translate articles.
• Probable change in screencast/screenshot policy. Have a look. Give your feedback. [1]
• Sumodev stuff for anyone who wants to test on support-stage:
  • Email notification when an edit is made/approved/rejected (with comments). bug 446082
  • Reviewers now set “Mark other translations as out of date” instead of editors. bug 469188

Forum

• Writing guidelines for forum moderators (draft)
• Forum search currently filters by locale, which is rather pointless since we only have one locale and it wouldn’t be able to distinguish other languages even if someone e.g. wrote a post in German
  • cww to test and file bug

Live Chat

• Top tags from last week
  • Crash was the top issue all of last week
• Web client demo – currently in testing
  • Goal is to have Milestone 1 complete this week
  • zzxc to make sure the page works and then SUMO team will test. Then we can get more testing from Live Chat people once we know it at least work relatively well.
• Canned responses have been updated with new common issues and troubleshooting steps
  • To update this week http://support.mozilla.com/en-US/kb/Introduction+to+Live+Chat

Roundtable

• SFD will be a KB writing sprint on 9/3
  • Blog post about it to be posted today
• Laura wants to know if we should link to SUMO articles on the Plugin Checkup page.
  • Let’s definitely link to SUMO, but let’s make sure to review the target articles to make sure they’re up to date.