Test Pilot

results / Test Pilot — 20 Comments
25
Aug 11

Do 90% of People Not Use CTRL+F?

According to an article in The Atlantic floating around the internet, 90% of users don’t know how to use CTRL+F or Command+F to search a webpage. We were surprised at that percentage. Fortunately, Mozilla has TestPilot studies with open data, and we can see if Firefox users behave similarly. One relevant 7-day TestPilot study of about 69,000 Windows users focused on Firefox’s user interface. Along with seeing how users interacted with the navigation bar, their bookmarks, etc., the study looked at how often people used keyboard shortcuts.

What we found is that about 81% of TestPilot users didn’t use CTRL+F during the course of the study. While 81% is lower than the 90% in the article, TestPilot users are usually more technologically experienced than the general population, since they are largely Firefox Beta users. When we look at TestPilot users who consider themselves beginners, the percentage goes up to 85%. Therefore, our 81% figure does not belie the Atlantic piece.

In addition, those who use CTRL+F on average use keyboard shortcuts twice as much as those who don’t, even when we ignore those people who don’t use any keyboard shortcuts at all. This implies that people who use CTRL+F are more comfortable with keyboard shortcuts in general. The only keyboard shortcut the users who use CTRL+F lag behind in is Full Screen, or F11.

Feel free to take a look at the data yourself and let us know about any interesting trends you discover!

Test Pilot — 9 Comments
3
Aug 11

Test Pilot New Tab Study Results

[Cross-posted at Mozilla User Research]

The new tab page in Firefox is intentionally left blank, while some browsers present rich information on a newly opened tab.

The decision to leave new tab pages in Firefox blank was driven, in part, by a suspicion that too much information in the new tab may distract users from getting to the destination intended for the new tab. To test whether this suspicion is true and to learn more about user behavior after opening a new tab, Test Pilot recently released the New Tab Study and will soon release a multivariate test on the new tab page. Test Pilot is a platform collecting structured user feedback through Firefox. It currently has about 3 millions users and all the studies are opt in. You can help us better understand how people use their web browser and the Internet in order to build better products by participating studies. Test Pilot add-on is available here.The study ran for 5 days and in all, we collected 256,282 valid submissions.
Results of the study show that on average each user daily:
  • opens 11 new blank tabs
  • loads 7 pages
  • visits 2 unique domains
  • visits 2 pages in a new tab before they leave or close it

Below are details on how a user loads a page in a new tab, their intentions when opening a new tab, and time spent on new tabs below.

How do users load a page in new tabs?

 We detected 11 different methods to load a Web page in a blank tab page. Actions in the Url bar include pressing ENTER through keyboard, clicking the go button on the right side of the bar, clicking the Web page suggestions in the dropdown menu and pressing ENTER key for dropdown suggestions. Similarly 4 actions can be performed in the search bar too. Users can load a previously saved page from the bookmark bar in the toolbar or Bookmark/History in the menu bar.

Note:

  • The URL bar is most used when navigating to new websites.
  • The Search bar is also popular. Users rarely use search bar dropdown to look for old search terms.
  • The Bookmark toolbar is used more often than the bookmark menu button.
  • The History Menu button is seldom used.

We can also classify all methods for loading web pages into either keyboard-based or mouse-based category. Generally speaking, users have a slight preference for mouse usage.

 

Why do users open new tabs?

1.    Are they looking for a specific URL?

13.95% of new tabs (13,941,404) are opened while the text in the clipboard starts with “http” or “www”, which are very likely to be URL strings. The number is surprisingly high, although it may be caused by previous actions rather than by pasting for loading a specific URL.

2.    Users browse a limited set of domains, and only a small proportion of domains attract most visits

If we represent each user as a single point in the plot where x-axis is the number of pageloads, and y-axis is the number of unique domains visited, we can get the following graph. The dash line (diagonal) is what will happen if users always visit a different domain for each page load. When the users are not so active, pageloads less or around a few hundreds, the number of unique domains grows linearly. However, once users get to browse more, distinct domains tend to be stabilized and saturated.

Globally, we check the visit frequencies of all domains, and find that globally only 17.38% domains (461,133 unique domains in total) take 80% of the total page loads (8,291,541 pageloads in total). It verifies the famous “20-80” law of long tail phenomena.

On the individual level, we are interested in whether a single user performs the browsing movements according to the 20-80 law. For each individual, domains taking 80% of the total page visits is defined as “main domains”. A user can confirm the 20-80 law if the ratio of the number of his main domains to the number of distinct domains is around 20%. According to the following fig., active users browse more web pages everyday, but the number of primary sites they go to decreases proportionally. It suggests that when users visit more sites, they prefer to go to the same sites more frequently. The result supports the existence of a speed-dial new tab page to some extent.

 

Time Spent on New Tabs

According to the study results, on average, users open 2 pages in a new tab before they leave or close it. They load the first web page in 6 seconds (median) after they open new tabs, and stay on the tab for 1 minute (median) once they start browsing. The distributions of these two types of reaction timings display broad tails. The actually mean values are much higher than the medians: users load the first web page in 45 seconds (mean) after they open new tabs, and stay on the tab for 7 minute (mean) once they start browsing, since the outliners and expected noises can vary the mean value a lot.
Meanwhile, how users open a new tab can distinguish 2 groups of mouse-based users and keyboard-based users. The tabs invoked by “Plus Button” and “Double Click on TabBar” represent the group of mouse-based users, and the tabs invoked by “Command+T” represent the the group of keyboard-based users. The results turn out that keyboard-based users act slightly faster than mouse-based ones, and they can stay on the same new tab a bit longer.

The study is preliminary study for redesign requirement of the new tab pages in Firefox. We detect user behavior patterns of how they use the new tabs, including how they load a new page, broadness of domain visited, and the timing of different actions. In the following New Tab Multivariate Test, we will do a comparison between several designs of the new tab page, and more research questions will be answered, including whether too much information in the new tab may distract users from the original target or not.

plans / Test Pilot — No Comments
27
Apr 11

Join Mozilla Metrics

The Mozilla Metrics team is expanding to meet the growing data related opportunities and challenges faced by Mozilla and the web as a whole. In addition to open positions for a visualization expert and a metrics software engineer, we are also looking for a data analyst to focus on user experience. The UX data analyst will gather structured user insights and then leverage these insights to inspire and inform the design of our products.

Please reach out to us if you (or someone you know) has a passion for data and building a better internet.

The Mozilla community itself is also growing – so if data isn’t your thing, be sure to check out the other career listings as well!

Test Pilot — No Comments
11
Jan 11

Mozilla Open Data Competition – Announcing The Winners!

[Note: cross-posted on Mozilla Labs]

Back in November, Mozilla Labs and the Metrics Team together launched the first Mozilla Open Data Visualization Competition. While we set out to discover creative visual answers to the open question, “How do people use Firefox,” we really didn’t know what level of participation to expect from the Mozilla and data analysis communities. In fact, we were overwhelmed by both the number and quality of submissions – so much so that we had to give ourselves an extra few days to thoroughly review them all!

In all, we received 32 high-caliber submissions. The “visualizations’ took a number of forms, from tools to easily query the data to interactive web applications. They also covered a broad range of important topics, from plugin memory consumption to user web activities. You can find all 32 submissions here; entrants, if you haven’t already, be sure to check out the page as our panel of judges has left feedback on each and every submission.

Needless to say, we want to thank all the participants – your work has made our initial open data competition an overwhelming success and many of your insights will directly help the Firefox team develop a better web browser. In thanks, we’ll be sending this awesome Firefox t-shirt to each entrant:

We also want to thank our 3 partner judges: David Smith, Revolution Analytics; Andrew Vande Moere, Information Aesthetics; and Brian Suda, author of A Practical Guide to Designing with Data. The success of the competition was largely due to your help in publicizing the event and thoroughly evaluating the entries.

And now…lets get to the winners!

Grand Prize

Survey Participants vs. All Users – Contributed by: James Fiedler

While deciding amongst the 32 entries was difficult, the focus on a single, very relevant and important question distinguished this entry. James focused on contrasting survey participants with all users (critical as we often use survey data for segmentation), then set up a simple and helpful environment for the user to explore and discover interesting conclusions of their own. This submission is exactly the type of work we were hoping for: an elegant visualization that presents data around an important and complex question in a clear and easy-to-understand way. James will receive a $300 Amazon gift card for his excellent work.

Finalists

Test Pilot Explorer – Contributed by: Lon Riesberg

One of the more creative entries, Lon created a custom “explorer” that essentially “plays back” time-ordered events as animated plots and includes filters to customize what data is shown. This explorer really shows how you can “see” user behavior on a mass scale, and while we had some quibbles about some of the details of the visualization itself, we found it to be a powerful and enjoyable data exploration tool. Lon will receive a set of all 4 Edward Tufte books for his work.

Firefox Usage by Age – Contributed by: Tom Haynes (University of Michigan)

Tom’s entry also focused on one particular element of the data. His execution sets this submission apart, as his visualization doesn’t try to encompass everything, but tells a clear, specific story around how Firefox usage times vary across age groups. Tom will receive a set of all 4 Edward Tufte books for his work.

Honorable Mention

Given the number of worthy submissions, we decided to hand out 5 Honorable Mention Awards in addition to the original 3 prizes. For varying reasons, we thought these entries were particularly valuable and each team will receive Tufte’s latest book, Beautiful Evidence, in recognition of their great work. Good Job!

Firefox browser – Event Sequences – Contributed by: Benoît Pointet

Firefox 4 beta UI Component Use vs. User Expertise – Contributed by: Nicolas Garcia Belmonte, Maria Luz Caballero

Browser Usage Over the Course of a Day Contributed by: Christian Kreibich

Bookmark a Lot, Browse a Lot – Contributed by: Eugene Tjoa

Firefox Plugin Memory Consumption- Contributed by: Diederik van Liere and David Eaves

Again, thanks to all the participants, judges, and everyone else who helped make this first open data competition such a success! Participants should receive an email within the week with details on how to receive the prizes and t-shirts.

And keep refining those data hacking skills – there will be more open data competitions in the near future!

Test Pilot — 1 Comment
28
Dec 10

Browsing Sessions II: Extensions, Time of Day, Number of Sessions, and Session Length

In our last post we delved into the rudimentary dynamics of the “browser session,” defined as a continuous period of user activity in the browser, where successive events are separated by no more than 30 minutes.

In this short post we’ll discuss another way of cutting the data. Below is the plot. For reference, each crossbar contains the 1st and 3rd quartile, along with the median.

A few insights regarding the plot:

  • Users with more extensions have longer and more varied session times than those with fewer.
  • Extensions in general do a better job of discriminating user behavior than the number of sessions.
  • These trends tend to hold over the course of the day, with only minor fluctuations.
Test Pilot — 4 Comments
22
Dec 10

Browsing Sessions

We recently provided some simple insights we’ve gleaned from how people use private browsing. In this post we’ll take a higher view, and examine behavior regarding when people generally use their browser.

The tl;dr version: users who have more “sessions” (defined below) tend to browse longer, more diversely, and over a broader swath of the day than more casual users.

Preliminaries

Before we begin, the unit of analysis is the “browser session.” Here is our working definition: a browser session is a continuous period of user activity in the browser, where successive events are separated by no more than 30 minutes.

Despite its rudimentary nature, this definition of a session is still fairly common in the web analytics literature.

The median browser session, median number of sessions

As the graph indicates, the median session is only about 30 minutes long, with a very long tail. The first quartile is about 9 minutes long, while the third is about an hour.

The median number of sessions per user, on the other hand, is about 2 a day. Approximately 25% of users actively use the browser only once a day, while the 75th percentile has around 3 sessions a day.

More sessions ↔ longer sessions

Those users with a larger amount of sessions (say, 20 over the week-long study) tend to spend about 10 minutes more per session than those with around 10 sessions.

More sessions ↔ more varied session lengths

Users with more sessions also tend to have much larger variation in the lengths of their sessions, which suggests that for more frequent users, the use case of the browser is in general much more diverse.

More sessions ↔ wider range of use over the day

More frequent users tend to use the browser over a wider swath of the day as well. This is fairly intuitive – more and longer sessions should span a larger part of the day. It is striking, however, how large the range is for users with many sessions. This might be a consequence of the sample bias inherent in the Beta population. Most of our Test Pilot users are tech-savvy young men, so the wide range in which they browse is a little more understandable.

As you can tell there is a lot we can do with just analyzing sessions. We’ll be rolling out more simple insights like these soon – stick around.

Test Pilot / visualization — No Comments
7
Dec 10

Mozilla Open Data Competition – 10 Days Left!

[Note: cross-posted on Mozilla Labs]

Hello Data Hackers!

We just wanted to remind everyone that the submission deadline for the first Mozilla Open Data Visualization Competition is just 10 days away! Submit your entries by December 17th for a chance at a $300 Amazon gift card and a set of all 4 Edward Tufte books!

We’ve already received some great entries, and our panel of expert judges (Kevin Fox and Jinghua Zhang, Mozilla Labs; Hamilton Ulmer, Chris Jung and Blake Cutler, Mozilla Metrics) along with our partner judges (David Smith, Revolution Analytics; Andrew Vande Moere, Information Aesthetics) look forward to seeing the rest of the submissions!

Remember to visit the Official Competition Page for all the information you need, including how to download the data and enter the competition.

Good Luck!

Test Pilot / visualization — No Comments
17
Nov 10

Mozilla Open Data Visualization Contest – Data is Now Available!

[Note: cross-posted on Mozilla Labs]

Two weeks ago the Mozilla Metrics Team, together with Mozilla Labs and the growing Mozilla Research Initiative, announced our first Open Data Visualization Competition. Today, we are excited to release the data sets for this competition!

These data sets come from Mozilla’s own open data program, Test Pilot. Test Pilot is a user research platform that collects structured user data through Firefox. Currently, over 1 million Firefox users from all over the world participate in Test Pilot studies, which aim to explore how people use their web browser and the Internet in general.

For this challenge, data samples from two recent Test Pilot studies have been made available – check out the data sample pages below for a thorough description:

In addition, please note that the submission deadline has been extended from Dec. 5 to Dec. 17th to encourage more participation. Be sure to visit the official Competition page for more general info on dates, judges, and prizes.

Good luck and start hacking!

Test Pilot — No Comments
12
Nov 10

Bouncing Off of Menus

One of the most significant changes to the user interface for Firefox 4 was the rolling of the old menu items into one button, something we have called on this blog the “combined menu.” We’ve addressed the impact of this change in other posts, such as this one.  We thought of one obvious follow-up: how often do people ‘bounce away’ from the combined menu before clicking something in it?

We define the bounce rate as this: given a user clicking the combined menu button, what is the proportion of instances where users do not click on one of the menu items? We built a small NumPy library to help us analyze the sequence of user-browser interactions.

44% of users clicked something other than a menu item after having clicked the Firefox button, while 56% clicked on a succeeding menu item. These percentages do not change even if we throw out the first few instances of every user clicking on the Firefox button (new Beta users are likely curious about the new menu layout, so try it out).

The bounce rate for this study has gone up since the last version (where about 38% of button clicks led to bounces).  One possible explanation points to the handful new items added to the combined menu since the last beta interface study.  The additional cognitive load might lead to more bounces.

We are currently working on calculating the bounce rate for the pre-combined menu UI.  Stay tuned for that.

Test Pilot / visualization — 1 Comment
4
Nov 10

Mozilla Open Data Visualization Competition – How Do People Use Firefox

The Mozilla Metrics Team, together with Mozilla Labs and the growing Mozilla Research Initiative, is excited to announce our first Open Data Visualization Competition.

Using data from Mozilla’s own open data program, Test Pilot, we’d like to explore creative visual answers to the question: “How do people use Firefox?” We are looking for compelling visualizations that tell detailed, meaningful and yet easy-to-interpret stories about interesting user activities.

Read on for details on the data (accessible Nov. 17), prizes, and judges, including special judge, David Smith from Revolution Analytics. Also, don’t forget to check out the official Competition page and follow @moztestpilot on Twitter for news and updates.

The Data

This competition is based on Mozilla’s own open data program, Test Pilot. Test Pliot is a user research platform that collecting structured user data through Firefox. All data is gathered through pre-defined Test Pilot studies which aim to explore how people use their web browser and the Internet.

Currently, over 1 million Firefox users from all over the world participate in Test Pilot studies. The goal for this platform is to encourage everyone from all skill levels to improve the Web experience by conducting and participating in these studies. Test Pilot study results are made available under open licenses, with the data being anonymized before release. (For more information about the Test Pilot data policy, please check Privacy Policy.)

For this challenge, we will use data from two recent Test Pilot studies:

Partners and Judges

We are honored to have David Smith from Revolution Analytics partnering with us and serving as a special judge. Members from Mozilla Labs and Mozilla Metrics will form the rest of the judging panel.

Prizes

To recognize the awesome work from participants, some great prizes are at stake:

  • Grand prize will be a $300 Amazon gift card
  • Two “Best in Class” teams will receive a set of Edward Tufte’s books

We’ll also present all submissions on the Test Pilot website and Mozilla Metrics blog in a special post to highlight your work.

Join the competition!

You can choose any tools you like for your analysis and visualization, including but not limited to: R, Matlab, Protovis, Processing or IBM many eyes. You can participate solo or team up with other people. You are also welcome to enter as many times as you like. If you are interested to join the competition, please follow the following important dates:

  • Nov. 17th: Check the official Competition page on Nov. 17th to download the data
  • Dec.5th : Go here to submit your results and enter the competition before Dec.5th.
  • Dec. 14th: Winning visualizations will be announced on Dec. 14th.

To facilitate the free exchange of ideas, all visualizations and other contributions you make to this challenge must be contributed under the Creative Commons Attribution-ShareAlike 3.0 license.

← Older Entries