We recently discussed an optimization test at the Firefox Support site (support.mozilla.com). At the time, we mentioned that:
Next up, the SUMO team is planning to soon roll-out a much more substantial A/B test at support.mozilla.com. With a somewhat radically different page design, our expectations for this next test are different. In other words, the more variation that is tested generally leads to bigger differences in the results. Stay tuned!
We just ran this “test #2″ over the past week or so and we now have the results to share. For reference, the full project plan is here and the two page versions we tested are below.
Version #1 (Existing SUMO home page):
Version #2 (New design for SUMO home page):
What did the results say? Out of nearly 1.5 million visitors, we saw the following behavior:
What’s the impact of the “new design” page? Decreasing the bounce rate (or exit rate) by 0.5% translates to an improved user experience for approximately 260,000 Firefox users (annually). These are users who would have otherwise immediately exited the SUMO site (some perhaps feeling unsatisfied or unhappy). The one other critical metric for the SUMO team is the rate at which users utilize the knowledge base search box. Again, you’ll see that by this measure, the “new design” wins. 11.9% of visitors searched vs. 10.6% on our control (or existing) page design.
Thus, the SUMO team will now be implementing the “new design” as the permanent new Firefox Support home page.
Moreover, we dug into the data here a little further. We asked ourselves, “of people clicking through or taking some action on either version of the SUMO home page, did one version affect the chances that the user clicked on something beyond their initial action?” The answer appears to be “yes”.
First, let’s identify what are the other common actions taken by users at our two page versions, and then we can talk through what happened to users after this initial action.
The two most frequently clicked links on the existing SUMO home page – “Browsing basics” and “Customizing Firefox” – seem to provide content on their respective pages that is rarely used. During our test, we saw that less than 10% of users who navigated to “Browsing basics” further clicked on its relevant content. “Customizing Firefox” performed even worse, as less than 5% of users navigated to any of its content. Conversely, the two most popular links on our “new design” page – “Browse them all” (Popular Support Articles) and “Cannot clear location bar history” – appear to be clear improvements compared to the existing page. “Cannot clear location bar history” experienced 25% of its users following a suggested link and “Browse them all” saw 40% of its visitors use one of the suggested “Top Articles”.
On many dimensions, this optimization effort at support.mozilla.com looks like a huge win for Firefox users. Please click over to the SUMO blog for more details regarding future plans around further user experience improvements.
