Firefox Main Window Study: A Heatmap Visualization

Last month, Mike Beltzner shared an early product plan for the next version of Firefox with the Mozilla community and presented our vision for Firefox 4. A big upgrade coming with Firefox 4 is the new theme project: creating a new, sleek, and simpler default theme for the browser. The UX team has been working tirelessly on this – streamlining and modernizing the user interface, particularly on Windows, given the aesthetic changes introduced by Vista and Windows 7.

Note: Plans May Change.

Metrics and Test Pilot have been doing what we can to help guide the UX team as they create an optimized design. A few months ago, Test Pilot released the Menu Item Usage Study, from which we gained valuable insight into how users interact with the menu bar. And a few weeks ago, we released the Main Window Usage Study to capture the remaining major UI elements of the browser, and provide more data-driven recommendations.

Today, we’d like to present some initial findings from this study. Like previous Test Pilot studies, the main window study ran for 5 days and covered Firefox versions 3.5 and up. The sample consists of 9,667 users who have opted into the Test Pilot program AND opted to submit their data.

We usually display study results in charts and graphs on this blog, but for this study, we were inspired by the work of principal designer, Alex Faaborg, and came up with a slightly different kind of visualization. Back in March, Alex created a heatmap to visualize the menu study data (his post is also a great example of how the UX team is using Test Pilot data to inform design decisions). Overlaying the actual UI and putting the data in context was really useful, so with his help, we expanded on the idea and constructed this interactive, web-based heatmap for the main window study:


(image links to heatmap)

Currently, the heatmap can display the usage data in two ways. Selecting ‘Used by x% of users’ displays the percentage of users who clicked each UI element at least one time, and ‘Clicks per user’ is simply the frequency click count for each item, divided by the number of users in the sample (we normalize for sample size to make comparisons across the different subsets more meaningful). The first approach is useful because it mitigates the effect of outliers, since each person who clicks an item counts only once, whether they clicked the item 1 time or 25,000 times. The obvious drawback is that this metric fails to capture intensity of use, which ‘Clicks per user’ does capture.

You can also choose to examine specific subsets within the sample; there are selectors to display breakdowns by OS and by self-reported computer/web skill level. Unfortunately, you cannot combine these subsets yet; that is, while you can look at data for just Windows users or for just Advanced users, you cannot look at data for the subset of Advanced, Windows users. Our sample is too small for such sub-setting right now, but we’ll add this ability as soon as our sample size increases.

Be sure not to miss the table to the right of the heatmap – it lists all the UI elements in descending order and also outlines the color spectrum. In addition, if you are viewing data for a subset, the table will show the differences between the currently selected subset and the other related subsets to make comparison easy (the differences for the ‘Used by x% of users’ filter are simply one subset minus the other, the differences for the ‘Clicks per user’ filter are in percentage change).

We’ll discuss results more thoroughly in future blog posts, but here are a few findings that immediately jumped out to us. As always, these findings apply to our sample of Test Pilot users only, and we realize this sample is not likely representative of the population of Firefox users as a whole. Furthermore, its important to remember that usage data like this is only one part of a rigorous design process, and that it alone will not dictate any design decisions.

  • The Back button is used far more often than any other navigation element (by which we mean the Back, Forward, Reload, Stop, and Home buttons). 93.1% of study participants used the Back button at least once, and on average, each user clicked Back 66.2 times over the 5 days – that’s 3x more clicks than the Reload button, 10x more than the Home button, and over 30x more than the Forward and Stop buttons!
  • Of all items that are not in the Firefox main window by default, the New Tab Button (Navigation Toolbar) is the most commonly used, both by % of users and frequency count.  Looking at click counts, the New Tab Button (Navigation Toolbar) is a more popular way to generate new tabs than the New Tab Button (Tab Bar) that is located in the tab strip by default.  The New Tab Button (Navigation Toolbar) is also more often used than the default UI elements Bookmark Star and RSS Icon by click count.
  • Another conspicuous result is the disparate use of the scroll arrows in comparison to the scroll slider.  This holds true for both the vertical scroll bar and the horizontal scroll bar; while 89% of people used the vertical scroll slider, only 21.2% and 16.6% of people used the up and down arrows. And although 42.8% of users used the horizontal scroll slider, only 2.1% and 0.9% clicked the left and right arrows to scroll.

Besides putting the data in a spatial context, another benefit of this visualization is that it is easy to reuse, update, and improve. We plan to add more data, more breakdowns, and the ability to synthesize multiple subsets in the near future. We also want to improve the robustness of our findings by filtering out certain extensions (e.g. some users have the Google Toolbar and might be searching through that channel instead of the integrated search box), and by increasing our sample size.

In addition, the heatmap will be updated over the course of the Firefox 4 beta program. Hopefully, this visualization will help us understand how the various UI changes affect user behavior, and ensure that these design decisions are in fact improving the product for our (beloved) end-users. Look for these updates soon!

Thanks again to Alex Faaborg, and to all Test Pilot users for opting-in to provide us with the data. If you’d like to help us understand how people use Firefox and build a better browser, head over to the Test Pilot website to learn more and download the add-on. And as always, please feel free to download the raw data and supplement our analysis with your own!

13 responses

  1. Pino wrote on :

    One thing that would be interesting to measure additionally is double click to open a tab. From https://bugzilla.mozilla.org/show_bug.cgi?id=575248 it looks like this feature might be removed, but since only 17.9% of people uses new tab buttons in the main window according to this study, there is still 82.1% of users who either don’t use tabs or use a different way to open them. It is speculative of course, but a large share of those might be used double click for opening tabs.

  2. Tiago Sá wrote on :

    I think a very important conclusion that we can draw from this is that the unification of the search bar with the location bar is probably the worst thing a browser can do…

    Also, the small number of people that clicked the Feed Icon doesn’t necessarily mean few people use Live Bookmarks. I have a few, and I very very rarely click the Feed Icon, since I already have my live bookmarks all set 🙂 It’s the same thing with the bookmark button. But I believe the proposed placement for Firefox 4 (where the favicon is in the location bar now) is a better place to be, I think.

  3. wrote on :

    Re: New Tab Button
    Doesn’t appear like that here?


    New Tab Button (Tab Bar) 12.7%

    New Tab Button (Navigation Toolbar) 5.2%

    New Tab Button (Navigation Toolbar) 1.5

    New Tab Button (Tab Bar) 1.3

    There’s no data for whether a test subject were using a desktop or a portable computer, is there? I guess that would’ve explained OS X?

    Windows | vs. OS X | vs. Linux

    Vertical Scrollbar
    190.9 | 205.4% | 94.4%

    Scroll Vertical Bar
    134.9 | 132.2% | 63.1%

    Scroll Vertical Up
    49.4 | 1200% | 263.2%

    Scroll Vertical Down
    6.6 | 1000% | 247.4%

  4. Christopher Jung wrote on :

    @Té

    Re: New Tab Button – By ‘Used by x% of users’, yes the New Tab Button (Tab Bar) is used by a more people; but this isn’t surprising since the New Tab Button (Tab Bar) is in the UI by default, whereas one has to customize the toolbar to add the New Tab Button (Navigation Toolbar). In our post, we are stating that given that it isn’t a default element, its surprising that New Tab Button (Navigation Toolbar) is clicked more frequently than New Tab Button (Tab Bar) when viewing the “Clicks per user” filter.

    Re: Scroll bar usage – We definitely noticed the considerable disparity between OS X and Windows as well. And your right, this difference has a lot to do with the trackpad on Apple laptops (and its 2-finger scroll ability)

  5. wrote on :

    Hehum… sorry about that. I think my brain had a melt-down or something so the “Of all items that are not in the Firefox main window by default” part never registered.

  6. Anonymous Coward wrote on :

    “unification of the search bar with the location bar is probably the worst thing a browser can do” – On the contrary, this research shows that unification is the best thing to do, since it merges two heavily used click-spots into one bigger click-spot.

  7. jmdesp wrote on :

    As far as I’m concerned, scroll arrows are an important productivity tool. Moving the mouse cursor to the scroll bar is unacceptable, and the people who use scroll bar a lot are only I think those who have a *direct* access to it either in trackpad, or with the mouse wheel.

    My trackpad has a non-visible up/down zone and I don’t use it. Too much time needed to position your hand to be certain you won’t hit the wrong zone, too much distance between up and down, too far away from keyboard if I need to type again.

  8. Bill Burcham wrote on :

    I’d be interested to see the keyboard shortcut data too. No one I know uses the mouse to open a new tab (we all use command-t). Same for close tab, navigate back and forward.

    Similarly, we almost never use the vertical scrolling controls (thumb, up/down arrows) opting instead to “scrub” with two fingers on the track pad.

    Thinking that these studies need to segment the users into noobs and experts and look at the way the two (or more) groups use the interface.

  9. Ubu Walker wrote on :

    Compare the Firefox UI with Chrome’s UI. Chrome is extremely minimalistic, compared to Firefox. There is no search bar, only an address bar, since most people do their search there anyway. The bookmarks bar is hidden by default. There is no status bar on the bottom. So, essentially, all of the lesser used features have been stripped out…but can be added back in.

  10. Christopher Jung wrote on :

    @Bill Burcham

    Some great ideas Bill – we plan to add the ability to track keyboard shortcuts for thing like cmd + t in future iterations of the study.

    Also, you can actually segment the data by self-reported skill level right now (although we’d like to make this data more robust by increasing the sample size and supplementing with non-self-reported metrics). You can see that by “Click per user” beginners do use both New Tab Buttons, Back, and Forward more than intermediate and advanced users (125% and 157.1% more for the Tab Bar New Tab Button), presumably partly because beginning users are less familiar with the keyboard shortcuts.

  11. tomh wrote on :

    Another option in the heat map? The % of users and clicks per user are good, but I think there is another option which could be good to include:

    “Clicks per person who used this feature” It would enable people to see how many times people use a feature when they know about it,

    It might also be interesting to include things like the median number of clicks on a feature:- in some ways I think this is a better measure than a simple mean in this situation

  12. Elvin Elvis wrote on :

    Nice Work

  13. Aidan wrote on :

    Very interesting study/results. A few comments…

    I work for a company that makes computers peripherals and was especially interested in the scrolling data. Interaction with the most popular social networking sites today involves a lot of vertical scrolling, and this is reflected in the data. However, the data collected in the study (I looked at the log files) does not really give a very deep insight to the scroll interaction, e.g., distance of scroll, duration of scroll, direction, use of different methods including mouse/touchpad/arrow keys/page up-down, etc.

    Google Wave had some “interesting” innovations in scroll. Well worth taking a look for anyone interested in that. Also, a paper from a few years ago has a number of interesting points to make on scroll interaction in the context of documents, reference below:

    Alexander, J. and Cockburn, A. 2008. An empirical characterisation of electronic document navigation.