Blog of Metrics

As an intern here at Mozilla, I’ve learned that one of the core goals of the organization is to help build a better Internet by creating great software like the Firefox browser.  But like many other endeavors, one of the first steps to developing a browser, or software in general, is to know the end-users.

Consequently, we were excited to examine responses to the Firefox 4 Beta Background Survey. This short survey was pushed to all Beta users through the Feedback add-on and asked a few basic browsing behavior and demographic questions. In all, over 30,000 people, or roughly 7% of the entire Beta user-base, were kind enough to submit their responses.  Analyzing these submissions will not only help us understand the current Beta users, but also reveal the missing user groups we need to acquire to make the Beta sample more representative of the larger, general Firefox population.

So what did we find out? Who are our Firefox 4 Beta users?  Here are our main highlights:

1) The personal computer has been around longer than 75% of our Beta users

Although not surprising, our Beta population is concentrated among the lower age groups – over 75% of the Beta population is younger than 35 years old!  Even though we are skewed towards younger users, the good news is that there was a moderate increase in the participation of older users compared to our previous Test Pilot only sample.

2) Where are our Female Users? 96% of our Beta Population is male!

Evidently, our Beta Population is overwhelmingly skewed in terms of gender as 96% of users submitting feedback were male.

3) Early adopters are highly technical and spend multiple hours on the Internet a day

Due to the nature of betas, we expected the early adopters to be highly technical and spend a significant amount of time online.  Indeed, 69% of our Beta users are on the Web for over 4 hours a day, and 71% of gave themselves an advanced technical rating.

4) Entertainment, Communication and Socializing Drive Web Usage.

One of the most interesting aspects of this demographics survey was the potential to gain insights into the online interests of our Beta users.  We were particularly interested in identifying the main reasons why they use the web, as well as their most frequent visited websites. In particular, we found Beta users have three main reasons or using the web: Entertainment, Communication and Socializing.  We also noticed how these drivers were reflected in the most frequently visited websites as the top sites included News, Webmail, Social Networking, and Video content sites.

So what does all this survey information show? Overall, it clearly indicates that our Beta sample is far from representative of the general Firefox population. And in order to achieve more representative feedback, we need to increase the female, older and less technical user bases; we will definitely coordinate our Marketing, Metrics, and Community efforts to reach out to these users.

Maybe you existing Beta users can do some communicating and socializing to help us out! Tell your friends to download Firefox 4 Beta and help test the future of the Web!  Also, expect a follow-up blog post in the future as the makeup of the Beta sample evolves and (hopefully) becomes more representative.

Private Browsing was introduced in Firefox 3.5, giving users the option of browsing the web without keeping track of their history.  A recent Test Pilot study recorded, among other things, the time users activated Private Browsing, and the time they deactivated it.  Though what happens in Private Browsing stays in Private Private Browsing – that is, neither Firefox nor Test Pilot records anything during the period – we did learn a few things about the timing and duration of of Private Browsing mode sessions.

Here are a few simple insights we’ve gleaned from the data.

Note: Test Pilot is an opt-in service for Firefox 4 Beta users.  We would never record a user’s activity for any reason unless they explicitly signed up for this study.  Again, we do not record any information at all during a user’s Private Browsing session – only when private browsing was entered and exited. You can read over the Test Pilot Privacy Policy here.

Activation spikes at lunch.

Though people switch into Private Browsing mode throughout the day, there are a few periods where activation surges:

1. Lunch: users likely switch into Private Browsing during their lunch breaks.  We see a major spike between 11am and 2pm.
2. After School / Work: users appear to switch on Private Browsing just after they’ve returned from work or school, which is around 5pm.
3. After Dinner: there is another substantial usage peak between nine and ten pm.
4. Late Night: a minor spike exists an hour or two after midnight.

The 10-minute window is the most common.

Now that we know when users jump into Private Browsing, the next question involves finding out how long users stay in it.  The 25th percentile stays on for about 4 and a half minutes, the 75th percentile around 22 minutes, and the median stays in for about 10 minutes.

This trend appears to hold over the entire course of the day, with the notable exception of 5pm.  For some reason the median and lower quantiles are lower than the other hours.

We’ve more insights to glean from the Week-In-The-Life study.  Stay tuned for more.

Last month, Mike Beltzner shared an early product plan for the next version of Firefox with the Mozilla community and presented our vision for Firefox 4. A big upgrade coming with Firefox 4 is the new theme project: creating a new, sleek, and simpler default theme for the browser. The UX team has been working tirelessly on this – streamlining and modernizing the user interface, particularly on Windows, given the aesthetic changes introduced by Vista and Windows 7.

Note: Plans May Change.

Metrics and Test Pilot have been doing what we can to help guide the UX team as they create an optimized design. A few months ago, Test Pilot released the Menu Item Usage Study, from which we gained valuable insight into how users interact with the menu bar. And a few weeks ago, we released the Main Window Usage Study to capture the remaining major UI elements of the browser, and provide more data-driven recommendations.

Today, we’d like to present some initial findings from this study. Like previous Test Pilot studies, the main window study ran for 5 days and covered Firefox versions 3.5 and up. The sample consists of 9,667 users who have opted into the Test Pilot program AND opted to submit their data.

We usually display study results in charts and graphs on this blog, but for this study, we were inspired by the work of principal designer, Alex Faaborg, and came up with a slightly different kind of visualization. Back in March, Alex created a heatmap to visualize the menu study data (his post is also a great example of how the UX team is using Test Pilot data to inform design decisions). Overlaying the actual UI and putting the data in context was really useful, so with his help, we expanded on the idea and constructed this interactive, web-based heatmap for the main window study:

(image links to heatmap)

Currently, the heatmap can display the usage data in two ways. Selecting ‘Used by x% of users’ displays the percentage of users who clicked each UI element at least one time, and ‘Clicks per user’ is simply the frequency click count for each item, divided by the number of users in the sample (we normalize for sample size to make comparisons across the different subsets more meaningful). The first approach is useful because it mitigates the effect of outliers, since each person who clicks an item counts only once, whether they clicked the item 1 time or 25,000 times. The obvious drawback is that this metric fails to capture intensity of use, which ‘Clicks per user’ does capture.

You can also choose to examine specific subsets within the sample; there are selectors to display breakdowns by OS and by self-reported computer/web skill level. Unfortunately, you cannot combine these subsets yet; that is, while you can look at data for just Windows users or for just Advanced users, you cannot look at data for the subset of Advanced, Windows users. Our sample is too small for such sub-setting right now, but we’ll add this ability as soon as our sample size increases.

Be sure not to miss the table to the right of the heatmap – it lists all the UI elements in descending order and also outlines the color spectrum. In addition, if you are viewing data for a subset, the table will show the differences between the currently selected subset and the other related subsets to make comparison easy (the differences for the ‘Used by x% of users’ filter are simply one subset minus the other, the differences for the ‘Clicks per user’ filter are in percentage change).

We’ll discuss results more thoroughly in future blog posts, but here are a few findings that immediately jumped out to us. As always, these findings apply to our sample of Test Pilot users only, and we realize this sample is not likely representative of the population of Firefox users as a whole. Furthermore, its important to remember that usage data like this is only one part of a rigorous design process, and that it alone will not dictate any design decisions.

• The Back button is used far more often than any other navigation element (by which we mean the Back, Forward, Reload, Stop, and Home buttons). 93.1% of study participants used the Back button at least once, and on average, each user clicked Back 66.2 times over the 5 days – that’s 3x more clicks than the Reload button, 10x more than the Home button, and over 30x more than the Forward and Stop buttons!
• Of all items that are not in the Firefox main window by default, the New Tab Button (Navigation Toolbar) is the most commonly used, both by % of users and frequency count.  Looking at click counts, the New Tab Button (Navigation Toolbar) is a more popular way to generate new tabs than the New Tab Button (Tab Bar) that is located in the tab strip by default.  The New Tab Button (Navigation Toolbar) is also more often used than the default UI elements Bookmark Star and RSS Icon by click count.
• Another conspicuous result is the disparate use of the scroll arrows in comparison to the scroll slider.  This holds true for both the vertical scroll bar and the horizontal scroll bar; while 89% of people used the vertical scroll slider, only 21.2% and 16.6% of people used the up and down arrows. And although 42.8% of users used the horizontal scroll slider, only 2.1% and 0.9% clicked the left and right arrows to scroll.

Besides putting the data in a spatial context, another benefit of this visualization is that it is easy to reuse, update, and improve. We plan to add more data, more breakdowns, and the ability to synthesize multiple subsets in the near future. We also want to improve the robustness of our findings by filtering out certain extensions (e.g. some users have the Google Toolbar and might be searching through that channel instead of the integrated search box), and by increasing our sample size.

In addition, the heatmap will be updated over the course of the Firefox 4 beta program. Hopefully, this visualization will help us understand how the various UI changes affect user behavior, and ensure that these design decisions are in fact improving the product for our (beloved) end-users. Look for these updates soon!

Thanks again to Alex Faaborg, and to all Test Pilot users for opting-in to provide us with the data. If you’d like to help us understand how people use Firefox and build a better browser, head over to the Test Pilot website to learn more and download the add-on. And as always, please feel free to download the raw data and supplement our analysis with your own!

Last post, we identified the most and least commonly used menu items. Today, we will answer another question: how many different menu items do people actually use? It’s a simple question, but one that could be critical to the proposal to condense the menu bar into a single application button.

The 80-20 rule, or the Pareto principle, suggests that the number is on the low end — around 14 items. This popular rule of thumb states that for many events, 80% of effects come from 20% of the causes. Applied to software, the common saying is that 80% of all end users generally use only 20% of an application’s features.

Let’s see if menu bar interactions follow a similar pattern:

We expected a median around 14, but the actual median is 3 — almost 5x smaller! This means that 50% of all users clicked 3 items or less during the course of the study. Moreover, only 25% of people clicked on more than 4 commands. The rule of thumb states that 80% of users use only 20% of an application’s features, but menu bar behavior is far more limited: 80% of users use only 5 menu items, or just 7% of the total number of commands!

We were also surprised by the number of people who click only one unique menu item. Considering that over 20% of people fall into this category, we want to investigate further and identify which single item these individuals are using. Identifying these items is especially important since these single commands account for 100% of these users’ menu bar interactions — imprudently removing any of these commands could severely damage the browsing experience of these users.

Conveniently, the list roughly matches the ranking of the most commonly used menu items. This means that we can continue to base menu design decisions off of this ranking, without crippling the single-item users. If the orders were radically different, we may have needed to give special consideration to items that are important to single-item users, but not commonly used otherwise.

The presence of items such as “User Bookmark Item” and “Unknown”, however, leads us to some qualification of our findings:

• A few items, “User Bookmark Item”, “Unknown”, and “User History Item”, actually encapsulate more than one unique command.  We either did not find it useful to parse these items further (in the case of “User Bookmark Item” and “User History Item”), or were not able to (in the case of “Unknown”).
• A few items were not tracked during the study due to a bug (Special Characters, Character Encoding/More Encodings/*, Character Encoding/Customize List, Show All History, Organize Bookmarks, Minimize, and Zoom).  This means that the number of unique items for users hitting these commands is slightly downward biased.
• The study ran for 5 days as the goal was to learn about conventional menu bar behavior.  Over the course of a browser’s lifetime, users probably click a few more menu items, but we want to optimize the bar for normal usage, so limiting the scope to a typical work week is fairly reasonable.
• The findings refer solely to mouse interactions — again, keyboard shortcuts will remain consistent under any design.
• And as always, our sample is of Test Pilot users only, and is likely not representative of the population of Firefox users as a whole.

Nevertheless, we are confident in our general point that relatively few menu items account for nearly all the interactions with the menu bar. Given how many items are actually used, the menu bar is too comprehensive and takes up too much valuable real estate — in other words, a condensed menu bar seems well justified.

Thoughts? How many different menu items do you typically use?

Now that we’ve ironed out some data issues and cleaned the data sets, we can get back to our analysis.  If you remember, we were trying to answer 3 questions as the UX team contemplates modifying the design of the menu bar for future Firefox (Windows) versions:

• Which menu items are the most commonly used?
• Which menu items are the least commonly used?
• How long do users spend exploring the menu bar contents before selecting each particular menu item?

We took a stab at answering the first two questions by examining the total number of clicks on each menu item.  Again, all data is for mouse user interactions only–keyboard shortcuts will obviously remain consistent under any design.  The updated chart is shown below:

The most significant change is the usage of “Copy”, “Paste”, “Back”, and “Reload”.  These menu items were previously part of the top 10 most commonly utilized commands, but each item now accounts for less than .4% of total clicks.  This change isn’t too surprising since these commands are all available from context menus, and a major part of our data cleaning was the removal of context menu events that were incorrectly tracked as mouse clicks.

The revised ordering of these items also aligns more closely with how we conventionally believe people interact with the menu bar, especially for “Copy” and “Paste”.  Crtl + C and Ctrl + V are two of the most well known keyboard shortcuts, and intuitively, it seems unlikely that many users would forgo the hot keys and use the menu bar for these commands (especially Test Pilot users).

Besides these items, the relative order is fairly unchanged.  Notably, “User Bookmark Item” still dominants usage of the menu bar: 41% of all interactions are “User Bookmark Item” hits.

In previous posts, we also acknowledged that a major problem with analyzing aggregated data in this way is the potential for outliers to skew our results.  In order to present a more complete picture, we again move beyond looking at aggregated counts to examining how these counts are distributed. The table below presents some key statistics on the distribution of clicks for each menu item.

The distributions are now far less distorted by outliers, although clearly some right-skewness is still present. “User Bookmark Item” and “New Tab” are two of the more skewed distributions as both means are 4x greater than the medians.

Wrap Up
Now that we’ve revised our initial analysis, we will discuss new issues regarding menu usage behavior in the next series of posts. As always, thanks to all Test Pilot users and to everyone for the insightful comments–remember, more information on Test Pilot studies is available here, including the data samples themselves for anyone interested in doing their own analysis!