With our last experiment, we found that a simplified download page drove conversions up 2.3%. We hypothesized that much of this improvement was due to improved page load speed. Today, we’ll test this theory — answering not just whether speed matters, but also how much it matters.
Specifically, we want to know: for each second improvement in page load speed, by how much does our conversion rate improve?
Let’s begin by answering a simpler question: was there a difference in page load speed between visitors who downloaded Firefox and visitors who didn’t? We could compare average page load times, but that metric only tells part of the story and can be heavily skewed by outliers. Instead, let’s look at the distribution of page load speed of these two groups (1 = visitors who downloaded, 0 = visitors who did not download):
The difference is surprisingly large. Half of the downloaders loaded our landing page in under 2.10 seconds. For non-downloaders, that figure was 3.67 seconds — 75% slower. Furthermore, a quarter of non-downloaders waited over 7 seconds for our page to load!
Now that we’ve confirmed a difference exists, let’s estimate by how much page load speed improves conversions. To do so, I ran a simple logistic regression, controlling for the experimental variations.
We found that a 1 second increase in page load time decreases conversions by 2.7%! Assuming our model is correctly specified, we have massive room from improvement and can easily increase our conversion rate by 5 – 10%.
We must be careful, however, not to reach too far with our conclusions. There are two factors which complicate our analysis.
First, page load speed does not precisely measure what we care about. What’s important isn’t whether the page fully loads, but rather how quickly our main calls to action appear. Nonetheless, page load time serves as a strong proxy for visible content.
Second, our model suffers from omitted variable bias. We may imagine, for example, that visitors with faster Internet connections are naturally more inclined to download Firefox. In this case, it’s not page load speed that explains the higher conversion rate, but rather an unknown variable that’s correlated with both page load speed and download conversions.
Correlation is not the same thing as causation, but as Tufte said, it’s a darn good hint! And we can verify this hint by running a second experiment.
With the help of Ryan Doherty and Frederic Wenzel, we have already built an optimized version of the IE download page. By randomly assigning users to either this faster variation or the slower control, we will learn whether speed itself drove a higher conversion rate. Look for a discussion of our results later this week.
