The SUMO team has been dedicated to improving the support.mozilla.com site in many different ways. One such effort has been around website optimization and testing. We’ve previously touched on a broader roadmap for Mozilla, and as part of that vision, the SUMO team is implementing some A/B tests at support.mozilla.com. As a starting point, we launched this super simple A/B test last month. The thought behind it was this – we already have two slightly different designs for the SUMO homepage, so why not test them side-by-side to see if there’s any discernible difference in how users interact with each version (i.e., does one provide a better experience for users?).
First, let’s look at the two page versions…
Version #1 (in-product start page):
Version #2 (support.mozilla.com start page):
You’ll notice that the designs are relatively similar… the only difference being the content below the knowledge base search box. As expected, the results weren’t particularly telling. (Remember – this was a lightweight and easy test to implement, so it behooved us to at least try.)
Version #2 showed a slightly greater bounce rate (+0.2%). This is a borderline case of “statistically significant”, so we shouldn’t read too much into it. Conversely, of the people taking some action on these pages (e.g., clicking on a link/article, searching, etc.), Version #2 had a greater percentage of people utilizing the knowledge base search box (which is generally the desired outcome). Thus, at an aggregate, there doesn’t appear to be a “winning” version of the SUMO start page.
Next up, the SUMO team is planning to soon roll-out a much more substantial A/B test at support.mozilla.com. With a somewhat radically different page design, our expectations for this next test are different. In other words, the more variation that is tested generally leads to bigger differences in the results. Stay tuned!
