Blog of Metrics

We wanted to revisit one of our previous discussions surrounding shifts in worldwide browser market share. In that previous analysis, we considered the universe of new internet users (or those who switched browsers) in 2007, and then asked the question, “who grabbed which slice of the pie?” (in other words, how have the successes of Firefox and other browsers/versions stacked up against one another?). The short answer was that Firefox grabbed the majority of the pie and Opera Mini also enjoyed considerable success.

Revisiting this discussion, we wanted to turn to a different provider of market share data – WebSideStory – to see if the story is similar. The previous study utilized data from Net Applications, where you can drill down into market share trending data for about forty different browser versions. Though not nearly as granular as the Net Applications’ data, the data set from WSS should tell us if our original story was directionally correct.

The pie chart below is derived from the WSS data set. Here are a few things to note:

• WSS’s sample includes approximately 100 million daily unique visitors (across their clients’ sites)
• WSS’s data set breaks down market share data for three browser brands: IE, Firefox and Other (unfortunately, we can’t dive down into browser versions as we could with our previous data set). Our best guess is that “Other” consists almost entirely of Opera and Safari.
• Given that the WSS data requires a subscription, we’ll share our analysis based on their raw numbers, rather than the raw numbers themselves
• The chart below compares an average of Jan-Feb numbers to Nov-Dec numbers. A more thorough explanation of our methodology was described here.

What does this mean for Mozilla?

In short, trending (relative to the competition) is the name of the game. Instead of thinking about browser brand X declining from 75% to 70% or browser brand Y increasing its share from 5% to 10%, it’s important to keep the broader browser landscape in perspective. These assigned percentages are not always accurate and the scale of macro level shifts is generally out of anyone’s hands. What each brand does have control over is their own growth relative to the other players.

I did an admittedly poor job of framing my last post in its broader context. We set out to discover the true effect of paid search advertising on a change (or increase) in number of Firefox downloads, and in the process, both the experiment and analysis were focused narrowly on one channel (US search) through which Firefox downloaders arrive (to the exclusion of all other channels).

In the broadest universe of Firefox downloads, what did our experiment tell us?

• Over any two-day period, Mozilla sees about 1 million downloads of Firefox
• Approximately half of these downloads are for our en-US version and about half are for our localized language versions
• We focus our search marketing efforts primarily on the en-US market
• Within this specific market, about 38% of downloaders come through the search channel (organic + paid)
• Within this narrowly defined download channel – en-US search – about 75% is attributable to organic search and 25% to paid search.
• Over the two-day periods in our experiment, we saw an average of 21K additional downloads as a result of our paid search advertising
• Thus, by participating in paid search advertising, we are currently able to increase the total number of global Firefox downloads by approximately 2% (increasing 1 million by 21K).

A few last notes about this 2% estimate:

1. It’s based on experiment numbers representing 80% of our en-US search traffic, so the true effect is closer to 2.5% (if we extrapolate the 21K increase) before considering the next point.
2. When we have our paid search advertising “turned off,” the “loss” of downloads might be far less than the 21K indicated above. These prospective Firefox downloaders may simply be diverted to the “Other” traffic category (e.g., our affiliates program); unfortunately, we don’t have a good way to detect and measure such an occurrence. So, the true effect, rather than 2.5%, might be closer to 1%.

Summing it all up… we need to do more experiments, and from a cost-benefit standpoint, we need to decide if our marketing spend on paid search advertising is worth the additional 1% of Firefox downloads (alternately, we can work towards more effective strategies and tactics within this channel).

Here’s a quick quiz to start our discussion:

If you acquire three customers through online advertising and your total cost is $3 and your reporting system shows your cost per acquisition as $1, what was your true cost to acquire each customer?

Hint: the correct answer is not $1 to acquire customer A, $1 for customer B, and $1 for customer C. You’ll find our answer at the end of this post.

In our previous post, we discussed an upcoming experiment related to Mozilla’s online advertising efforts. We saw with funnelcake that approximately 38% of Firefox downloaders originate from search engines, so understanding users’ behavior within this channel – along with the full costs and benefits of our paid search advertising – is critical from a marketing perspective.

The question we set out to answer is this: if we turn off our pay-per-click ads, will a user who would have normally been expected to click on our paid ad simply click on our regular (i.e., organic) search listing?

Our problem solving methodology somewhat mirrored the flicking on/off of a light switch, as we conducted the experiment in such a way that we were able to largely isolate the effect of having our search ads on/off on a change in the number of Firefox downloads. Here’s the methodology we used:

• Utilized a Monday through Thursday time period within a particular week. We’ve previously noted that internet traffic appears fairly consistent across these days.
• On alternate days, we turned off our paid ads. For example, one week our ads were active on Tuesday and Thursday and turned off on Monday and Wednesday. In the other week, we reversed the on/off days. We alternated like this to reduce the chance of any biasness or exogenous factors affecting the experiment.
• The audience involved includes only users downloading the en-US version of Fx
• The results from Week 1 and Week 2 are presented below. The pie charts answer the question: Where are downloaders coming from?

Here’s what we found in Week 1:

Here’s what we found in Week 2:

How should we interpret these pie charts?

Approach A – How much of that green pie slice are we still able to capture with zero ads?

• You’ll see in Week 1 that 19.3% of downloaders came through organic search and 7.4% arrived via paid search. When we turned off our paid ads, our organic search did not pick up that entire group of 7.4%, but it did pick up most of that group (24.6-19.3, or 5.3%).
• Taking 5.3/7.4 tells us that we capture 72% of would be paid ad downloaders through our regular (i.e., organic) search listing when we turn off our paid search ads.

Approach B – Total effect on aggregate numbers

• What was the effect on total number of downloads (just the raw number)?
• In Week 1 we saw 142K downloads through organic search when our paid ads were turned off. And when our paid ads were active? We saw 114K downloads via organic search and 44K downloads through the paid search ads.
• So, we captured a total 158K downloads vs. 142K by having our paid search ads active, or an 11% increase when considering just total numbers.

What does this mean for Mozilla marketing and our future efforts related to online advertising? While the findings are both surprising and telling, it’s important to remember that the steps outlined above were only an experiment. We’re not going to dramatically transform our business practices overnight without fully considering all ramifications (i.e., we’re not going to instantly turn off all Firefox advertising tomorrow). That said, these findings do tell us that we need to seriously rethink our existing online advertising strategies and tactics.

David and I look forward to returning to this subject again in the future, and we encourage your thoughts and feedback.

Lastly, returning to our original quiz, Mozilla’s cost of acquiring three new users would look like this (please note: the dollar figures are hypothetical):

customer A = $0, customer B = $0, and customer C = $3

How can we explain this? Marketers have been taught to think of costs associated with acquisition marketing as average costs (e.g, cost per acquisition). Borrowing a term from economics, we would benefit by transforming our thinking to that of marginal costs. In other words, marketers should be asking themselves, “is it worth $3 to acquire customer C?”, not “is it worth $1 to acquire each customer?”.