process

plans / process — 2 Comments
6
Feb 09

Analytics and Firefox Support

Over the past year, SUMO (also known as Firefox Support, Firefox Help, or support.mozilla.com) has grown into one of the most critical activities within the Mozilla community.  David Tenser and the SUMO group have done a stellar job of maintaining a SUMO metrics report, and to complement that, I’ve recently been talking with David about the broader story of a typical user’s experience with SUMO.

There are a bunch of different ways that users can find SUMO and there are many different ways for users to interact with the site and get the help they’re looking for, so I thought it would be interesting to start outlining these user experiences at a high-level.  In this discussion, I’d like to answer two questions:

  1. How does a Firefox user get to SUMO in the first place?
  2. What is a user’s most common interaction/experience on the site?

For question #1, there are two primary ways for a Firefox user to locate and arrive at SUMO.  First, there’s a support option within the browser itself (i.e., inproduct).  Click on “Help” at the top of your browser and you’ll see something called “Help Contents”.  Users clicking here go directly to support.mozilla.com.

The second primary way for users to enter SUMO is through the main navigation bar at www.mozilla.com and localized Mozilla sites.  The navigation bar for the mozilla.com web site includes five menu options – Products, Add-ons, Support, Community, and About.  Users clicking on “Support” (depending on the localized version) go directly to support.mozilla.com.

So, how are Firefox users actually getting to SUMO in the first place?

Last month, SUMO saw 12.9 million visitors.  Of these visitors, more than 71% entered the site via the “inproduct” path (Help -> Help Contents), about 13% entered the site via the main navigation menu on Mozilla’s web sites, and about 16% entered through some other path.

A few things seem surprising here:

  • That 71% slice of the pie is huge!
  • 12.9 million visitors to SUMO (last month) is also a massive number.  To put that in some perspective, last month about 4.3 million visitors entered the www.mozilla.com site through the home page.  It’s not quite a fair apples-to-apples comparison, but still…

Continuing this discussion, we’ll soon post some analysis around question #2 – What is a user’s most common interaction/experience on the site?

process — 20 Comments
6
Feb 09

Some Firefox Users Still on Fx2

While adoption of Firefox 3 continues to grow at a breakneck pace, we have continued to notice one other interesting trend – some Firefox users are still on Fx2.  Fortunately, the vast majority of Firefox users are always on the latest version of Fx3, and we’ve seen some great strides in recent months thanks to major updates helping most remaining Fx2 users upgrade to Fx3.

To help the remaining Fx2 users complete their upgrade, we thought we’d conduct a little investigation.  One relatively easy question to answer is… where in the world are these users?

The chart below looks at usage of Firefox over a recent week-long period and shows the percentage of Firefox users in each continent using Fx2.  The average is pretty close to 15%.  The continents are sorted left-to-right according to the where most Firefox users (regardless of version) are located.  For example, about 46% of the total Firefox user base is located in Europe, about 32% in North America, etc.

Perhaps more interesting and relevant is an analysis by country.  There are a handful of countries showing Fx2 usage levels above 20%:

If anyone has any thoughts as to how we can better help these particular users and make sure they get upgraded to the latest version of Fx3, please chime in with a comment.

plans / process — 2 Comments
20
Nov 08

We shipped funnelcake03

We recently reinitiated our Funnelcake experiment on November 18th from 00:00:00 – 23:59:59 PST.  As you may recall from last year’s effort, we:

Before we soon publish our latest findings, I want to take an opportunity to briefly articulate why Funnelcake is important within the Mozilla community.  Generally, we have just about zero visibility into the experience of new users of Firefox.  For example, did a new user enjoy a happy download and install process? did he/she continue to be satisfied a month later, six months later?

Funnelcake doesn’t specifically relate the attitudes of new Firefox users, but it does start to provide some structure around the answers to those questions.  Having some information around the potential pain points of new Firefox users, along with how such users behave (e.g., are they satisfied?) over a long period of time, is important knowledge not simply for marketers or statisticians, but for the general online population as a whole and for the Mozilla community in particular.

process / Uncategorized — 5 Comments
6
Nov 08

Firefox Usage and Europe

It’s surprising how the simplest things can sometimes be the most revealing…

We’ve previously talked quite a bit about the international landscape that describes Firefox adoption and usage.  See here and here, along with this old post from Schrep.  Today, Jane asked me a relatively simple question related to Firefox usage in Europe.  Within a minute, I found the answer to her question creating the pie chart below.  In the past, I’ve generally focused on usage patterns by country, but I found this simple view of the data much more eye opening than previous analyses.

What is most surprising?

We previously highlighted the interactive Treemap below showcasing internet population and penetration by both region and country.  Asia appears to represent about 40% of the global internet population, but it isn’t as nearly well represented in the Firefox chart above (12%).  Clearly, there’s much more work for us to do here in the future (a good thing!).

process — No Comments
8
Oct 08

Visualizing Data and the Localization Community

Thanks to Seth for highlighting some recent analysis by Blake Cutler.  As previously mentioned, we’re always on the lookout for new, innovative ways for visualizing data, and as an example, Blake was able to create several cool motion charts based on our localization communities and their adoption of Fx3 beta.

Such analysis is useful in that it can help us ask better (or more interesting) questions.  In Seth’s own words, “When I saw these charts, one question came to mind.  How can we grow our localization communities so all localizers can benefit from having thousands of beta users?”

Click here to read more.

process / results — 5 Comments
16
Sep 08

Do Ads Driving Firefox Downloads Affect Firefox Downloads?

We’ve previously talked extensively about one of Mozilla’s core marketing tactics – search engine marketing.  One of the questions we always come back to is: “what macro level effect does this activity have?”  In other words, we know this channel is driving clicks and Firefox downloads every day, but is this effort actually aiding Firefox adoption in some way?

This thought might seem anti-intuitive.  If these marketing campaigns are driving a mass number of clicks and downloads, of course it *seems* as though we’re having some small impact on Firefox adoption.

To test these questions, we recently set-up an experiment much like an experiment we conducted last fall.  Here’s what we did:

  • Starting in July, we alternated our bidding on alternate days on a major search engine over a five week period
  • Some weeks we turned off our branded search campaigns on Mondays and Wednesdays (to compare with Tues/Thurs during those weeks), and during some weeks we turned off our branded search campaigns on Tuesdays and Thursdays (to compare with Mon/Wed within those weeks)
  • For the latter example, a Monday/Wednesday combination with our campaigns left active would have benefited from about 80,000 ad clicks that the Tuesday/Thursday period within that week did not benefit from

Below are the findings when looking at Firefox download numbers on a macro-level, i.e., across all channels, locales, etc.  This is a very large macro-level number, and the traffic seen each day via search engines is only a small part of the overall story, so as one would expect, it’s difficult to see an effect.

Fortunately, we can drill down into the data a little further.  Below are the findings when looking at Firefox download numbers solely through Google.  The numbers represent all downloads through that single channel, i.e., via both organic search results and paid advertisements.

Are these results surprising?

Sort of.

Our previous experiment (with a more limited scale), showed us two things:

  • The vast majority of people – who would normally click on our advertisements – will either click on our organic search result or find Firefox via some other method when we turn off our ads
  • Having our search campaigns turned on increased the overall pie of daily downloads by approximately 1%

This latest experiment suggests that that initial finding could hold true, but it also makes clear the fact that the answer to our original question (“do search campaigns aid Firefox adoption in any way?”) remains somewhat inconclusive.

We want to make sure we’ve done a fully rigorous job here, so please comment with your thoughts if you have a different take on these numbers and analysis.

process / results — 6 Comments
4
Sep 08

Advertising Firefox on Wednesdays

We’ve previously talked about search engine marketing as one of the key marketing efforts here at Mozilla.  Those previous analyses have pointed to experiments we’ve performed that helped increase our efficiency by approximately 50% earlier this year (that’s pairing a reduction in total spend with an increase in total clicks).  While those are some great gains, we’re always considering additional ways to improve our efficiency, and ultimately the new user’s experience, through this marketing channel.

An example of a potentially untapped area for improvement is day/time parting, i.e., better targeting by day of week or hour of day.  Intuitively, it’s easy to imagine that a particular day (e.g., Fridays) or a particular hour (11pm PT) might see a lower price (cost per click) or differing interest among new users (click through rate).

Fortunately, this is a fairly easy hypothesis to test thanks to regression analysis.  With our AdWords account, we can grab hourly data and see what variables (e.g., particular hours of the day) cause the price/interest changes mentioned above.  Continuing our example, perhaps most advertisers run out of their daily ad budget by the end of day, causing late night hours to see a lessened competitive bidding environment, and consequently, a cheaper cost per click.  Regression analysis would be able to tell us exactly that – and tell us that perhaps we should be diverting more resources to such market inefficiencies.

Below is the regression equation and full regression output.  We used a recent data set from our primary campaign containing more than 4,300 hours, which equates to about six months worth of data.  Clearly our equation could be strengthened in different ways (more instances, more control variables).  For example, the results did improve when we substituted the day of week and month variables with a dayofweek*month interaction variable.  For any stats fans reading this, we’re open to other suggestions for improving our work (comments encouraged).

Cost per Click = α + β1hour_of_day + β2day_of_week + β3month + ε

Note: hour of day results are relative to the midnight to 1am hour (pacific time), day of week results are relative to Fridays, and month results are relative to April.

So, how do we interpret these results?

  • For hour of day, it looks like there’s little inefficiency to be found.
  • However, the day of week findings are generally very robust.  Tuesdays, Wednesdays and Thursdays are relatively less expensive, and Saturdays and Sundays are relatively more expensive.
  • At a macro level, these findings tend to point to a generally efficient market, meaning we likely won’t be changing our marketing decision making based on this information alone.
process — 3 Comments
9
Jun 08

Community Driven Stats for Download Day – Part II

Following up on his initial analysis of Download Day pledge data, Ehsan Akhgari created a dashboard to visually interpret the data in real-time.  The example below shows Poland as the tan-orange line and Brazil as the blue-teal line.  I’m impressed!

process — 4 Comments
5
Jun 08

Community Driven Statistics for Download Day

Hats off to community member Ehsan Akhgari for his awesome analysis of our Download Day pledge data (it far exceeds my feeble attempt).  Not only is Ehsan’s analysis updated in real-time, but he’s also made the source data available for other community members to work with and build on his ideas.  Great job!

plans / process — 3 Comments
21
May 08

The Psychology of Downloading Firefox

What attributes of a browser are most important to a prospective user when he/she is considering downloading a new browser?

This is a question we’ve set out to answer.  For marketing purposes, for example, this type of knowledge could be tremendously useful for Mozilla.  One of the key learnings that any student takes away from business school is the power of conjoint analysis, which can precisely answer our question.  However, such market research can be time consuming, expensive and often quite complex.  Fortunately, our question is much simpler than the one facing most consumer marketers for two reasons:

  1. Typically with conjoint analysis, each attribute can be broken down into different levels.  As the Wikipedia example points out, the screen format for TVs includes the levels of CRT, plasma, LCD, etc.  But in the case of web browsers, a browser generally either has a feature or it doesn’t, e.g., there aren’t different levels of tabbed browsing as a feature.
  2. The decision being made by a new, prospective browser user is fairly binary in nature.  In other words, it’s either “yes, I’ll download” or “no, I don’t want to download”.  On the other hand, typically with conjoint analysis, a marketer has to design several prototypes of a product and allow the prospective buyer to assign some dollar amount to each feature-level combination (i.e., the results are along a relatively complex spectrum).

Given the situation outlined above, and given that we frequently look for marketing solutions beyond the traditional (e.g., not “let’s hire a market research firm and let them figure it out”), David Rolnitzky and I had the idea of using search marketing to see if we could find the answers to our original question.

In other words, given a control ad variation, what if we swap out a few words of an existing advertisement with a highlight of a new feature in Firefox 3?  For example, a text ad within AdWords includes four lines (text and URLs).  If our existing (control) ad variation has a line stating, “More Customizable!”, perhaps we could rotate other phrases in its place, e.g., “Now with the Awesome Bar!” or “Now with Improved Support!”

If the results show that simply changing one line within an advertisement has a big impact on whether or not prospective new users click on the ad, then we’ll presumably have an idea as to which attributes of Firefox are most important to the user in their download decision process.  Perhaps there are some flaws in our thinking here or in our methodologies, but this exercise seems worth trying given both our situation and the question we’re attempting to answer.

We’re currently in the very early stages of this experiment, so please let us know if you have any suggestions for what Firefox features/attributes (or associated ad phrases) we should be considering.

← Older Entries
Newer Entries →