website optimization

speed / website optimization — 21 Comments
5
Apr 10

Firefox & Page Load Speed – Part II

Let’s start with the punchline: By making a few minor tweaks to our top landing pages, we can drive an additional 60,000,000 Firefox downloads per year.

60 million is a big number, so let’s add a little context. Looking back at our record breaking download day during the launch of Fx3 (June 2008), a total of 8,002,530 people downloaded Firefox. Adding 60 million downloads is akin to running 10 Download Day campaigns per year.

What are the tweaks?

It comes down just one factor — speed. As it stands, our landing pages can be painfully slow. Take a look:

The entire Chrome landing page loads before our header appears. Worse yet, our download button doesn’t appear for 7 seconds! While this video does not reflect the experience of every IE.html visitor, nearly a quarter of our visitors wait at least this long.

Now that we know this page is slow, let’s make it faster. To do so, I thought back to the 2 weeks I shopped a class on high performance web sites. But, I couldn’t remember much, so I asked Ryan Doherty for some tips. He had two simple suggestions: combine (or remove) the JavaScript files and inline the CSS.

Results
After implementing these changes in an A/B test, we saw impressive results. Previously, we predicted a 1 second reduction in page load speed would improve download conversions by 2.7%. In reality, our optimized experimental variation shaved 2.2 seconds off the average page load time and increased download conversions by 15.4%!

With 275,000 daily visitors, a 15% improvement on this single English page translates to 10.28 million additional downloads per year. And, if we’re able to achieve a similar performance boost across our other top landing pages, we’ll drive in excess of 60 million yearly Firefox downloads.

Yeah, but…
15% looks great, but how confident are we?

Running a one-sided Student’s t-test with a means difference of 14%, our experimental data yields a P value of 0.000051. This means that there is only a 0.0051% chance that we would obtain a 14% (or greater) improvement if the real effect wasn’t at least this large.

Next Steps
We’re excited by our initial results, but are only just getting started. We have a bunch more optimizations to try and have some ideas of our own. For example, we want to test a landing page that loads the download button before all other page elements. Later on, we plan to test more elements of Firefox’s adoption funnel, including the download and First Run experiences.

Many thanks go out to Ryan, Fred, Laura, and Matthew, who suggested running performance tests and has already made impressive performance improvements on addons.mozilla.org.

website optimization — 8 Comments
31
Mar 10

Firefox & Page Load Speed – Part I

With our last experiment, we found that a simplified download page drove conversions up 2.3%. We hypothesized that much of this improvement was due to improved page load speed. Today, we’ll test this theory — answering not just whether speed matters, but also how much it matters.

Specifically, we want to know: for each second improvement in page load speed, by how much does our conversion rate improve?

Let’s begin by answering a simpler question: was there a difference in page load speed between visitors who downloaded Firefox and visitors who didn’t? We could compare average page load times, but that metric only tells part of the story and can be heavily skewed by outliers. Instead, let’s look at the distribution of page load speed of these two groups (1 = visitors who downloaded, 0 = visitors who did not download):

The difference is surprisingly large. Half of the downloaders loaded our landing page in under 2.10 seconds. For non-downloaders, that figure was 3.67 seconds — 75% slower. Furthermore, a quarter of non-downloaders waited over 7 seconds for our page to load!

Now that we’ve confirmed a difference exists, let’s estimate by how much page load speed improves conversions. To do so, I ran a simple logistic regression, controlling for the experimental variations.

We found that a 1 second increase in page load time decreases conversions by 2.7%! Assuming our model is correctly specified, we have massive room from improvement and can easily increase our conversion rate by 5 – 10%.

We must be careful, however, not to reach too far with our conclusions. There are two factors which complicate our analysis.

First, page load speed does not precisely measure what we care about. What’s important isn’t whether the page fully loads, but rather how quickly our main calls to action appear. Nonetheless, page load time serves as a strong proxy for visible content.

Second, our model suffers from omitted variable bias. We may imagine, for example, that visitors with faster Internet connections are naturally more inclined to download Firefox. In this case, it’s not page load speed that explains the higher conversion rate, but rather an unknown variable that’s correlated with both page load speed and download conversions.

Correlation is not the same thing as causation, but as Tufte said, it’s a darn good hint! And we can verify this hint by running a second experiment.

With the help of Ryan Doherty and Frederic Wenzel, we have already built an optimized version of the IE download page. By randomly assigning users to either this faster variation or the slower control, we will learn whether speed itself drove a higher conversion rate. Look for a discussion of our results later this week.

website optimization — 25 Comments
22
Mar 10

Addition by Subtraction

With our first multivariate test, we set out to answer a simple question: which design elements drive downloads on the IE landing page? We didn’t know which elements were most effective, but expected each would help persuade visitors to ditch IE.

Our results are surprising! But before I share them, let me briefly describe our experiment design.

We focused on the 4 boxes highlighted in red. To test how each element contributes to Firefox downloads, we created 16 variations of this page, each containing a different set of elements. Note: rather than removing the entire footer, we simplified the footer, keeping the Privacy Policy, Legal Notices, and Report Trademark Abuse links.

Below, you can see 3 of the 16 page variations. Which do you think performed best?

The title tipped it off, but only one element positively impacted the download conversion rate–the download stats box. The main features box, the deep footer, and the switching tips all drove conversions down.

The simplest variation (far right image) performed the best, increasing the download conversion rate by 2.3%, at the 99% confidence interval. This improvement translates to 1.7 million additional Firefox downloads per year!

Next up, we will run similar tests on our non-IE download page and on our localized pages. If we hit the same 2.3% conversion improvement across these pages, we’ll drive 5.4 million additional downloads per year! And, if just 25% of those downloads convert into daily users, we’ll have added as many users as we have in all of Argentina.

We’ll also explore why the simpler variation performed better. One hypotheses is that more users converted because the page loaded faster. Another is that Take a Tour and Reasons to Switch elements are more visible and more persuasive. Have other ideas? Please leave them in the comments!