Blog of Metrics

[Note: cross-posted on Mozilla Labs]

Back in November, Mozilla Labs and the Metrics Team together launched the first Mozilla Open Data Visualization Competition. While we set out to discover creative visual answers to the open question, “How do people use Firefox,” we really didn’t know what level of participation to expect from the Mozilla and data analysis communities. In fact, we were overwhelmed by both the number and quality of submissions – so much so that we had to give ourselves an extra few days to thoroughly review them all!

In all, we received 32 high-caliber submissions. The “visualizations’ took a number of forms, from tools to easily query the data to interactive web applications. They also covered a broad range of important topics, from plugin memory consumption to user web activities. You can find all 32 submissions here; entrants, if you haven’t already, be sure to check out the page as our panel of judges has left feedback on each and every submission.

Needless to say, we want to thank all the participants – your work has made our initial open data competition an overwhelming success and many of your insights will directly help the Firefox team develop a better web browser. In thanks, we’ll be sending this awesome Firefox t-shirt to each entrant:

We also want to thank our 3 partner judges: David Smith, Revolution Analytics; Andrew Vande Moere, Information Aesthetics; and Brian Suda, author of A Practical Guide to Designing with Data. The success of the competition was largely due to your help in publicizing the event and thoroughly evaluating the entries.

And now…lets get to the winners!

Grand Prize

Survey Participants vs. All Users – Contributed by: James Fiedler

While deciding amongst the 32 entries was difficult, the focus on a single, very relevant and important question distinguished this entry. James focused on contrasting survey participants with all users (critical as we often use survey data for segmentation), then set up a simple and helpful environment for the user to explore and discover interesting conclusions of their own. This submission is exactly the type of work we were hoping for: an elegant visualization that presents data around an important and complex question in a clear and easy-to-understand way. James will receive a $300 Amazon gift card for his excellent work.

Finalists

Test Pilot Explorer – Contributed by: Lon Riesberg

One of the more creative entries, Lon created a custom “explorer” that essentially “plays back” time-ordered events as animated plots and includes filters to customize what data is shown. This explorer really shows how you can “see” user behavior on a mass scale, and while we had some quibbles about some of the details of the visualization itself, we found it to be a powerful and enjoyable data exploration tool. Lon will receive a set of all 4 Edward Tufte books for his work.

Firefox Usage by Age – Contributed by: Tom Haynes (University of Michigan)

Tom’s entry also focused on one particular element of the data. His execution sets this submission apart, as his visualization doesn’t try to encompass everything, but tells a clear, specific story around how Firefox usage times vary across age groups. Tom will receive a set of all 4 Edward Tufte books for his work.

Honorable Mention

Given the number of worthy submissions, we decided to hand out 5 Honorable Mention Awards in addition to the original 3 prizes. For varying reasons, we thought these entries were particularly valuable and each team will receive Tufte’s latest book, Beautiful Evidence, in recognition of their great work. Good Job!

Firefox browser – Event Sequences – Contributed by: Benoît Pointet

Firefox 4 beta UI Component Use vs. User Expertise – Contributed by: Nicolas Garcia Belmonte, Maria Luz Caballero

Browser Usage Over the Course of a Day Contributed by: Christian Kreibich

Bookmark a Lot, Browse a Lot – Contributed by: Eugene Tjoa

Firefox Plugin Memory Consumption- Contributed by: Diederik van Liere and David Eaves

Again, thanks to all the participants, judges, and everyone else who helped make this first open data competition such a success! Participants should receive an email within the week with details on how to receive the prizes and t-shirts.

And keep refining those data hacking skills – there will be more open data competitions in the near future!

In our last post we delved into the rudimentary dynamics of the “browser session,” defined as a continuous period of user activity in the browser, where successive events are separated by no more than 30 minutes.

In this short post we’ll discuss another way of cutting the data. Below is the plot. For reference, each crossbar contains the 1st and 3rd quartile, along with the median.

A few insights regarding the plot:

• Users with more extensions have longer and more varied session times than those with fewer.
• Extensions in general do a better job of discriminating user behavior than the number of sessions.
• These trends tend to hold over the course of the day, with only minor fluctuations.

We recently provided some simple insights we’ve gleaned from how people use private browsing. In this post we’ll take a higher view, and examine behavior regarding when people generally use their browser.

The tl;dr version: users who have more “sessions” (defined below) tend to browse longer, more diversely, and over a broader swath of the day than more casual users.

Preliminaries

Before we begin, the unit of analysis is the “browser session.” Here is our working definition: a browser session is a continuous period of user activity in the browser, where successive events are separated by no more than 30 minutes.

Despite its rudimentary nature, this definition of a session is still fairly common in the web analytics literature.

The median browser session, median number of sessions

As the graph indicates, the median session is only about 30 minutes long, with a very long tail. The first quartile is about 9 minutes long, while the third is about an hour.

The median number of sessions per user, on the other hand, is about 2 a day. Approximately 25% of users actively use the browser only once a day, while the 75th percentile has around 3 sessions a day.

More sessions ↔ longer sessions

Those users with a larger amount of sessions (say, 20 over the week-long study) tend to spend about 10 minutes more per session than those with around 10 sessions.

More sessions ↔ more varied session lengths

Users with more sessions also tend to have much larger variation in the lengths of their sessions, which suggests that for more frequent users, the use case of the browser is in general much more diverse.

More sessions ↔ wider range of use over the day

More frequent users tend to use the browser over a wider swath of the day as well. This is fairly intuitive – more and longer sessions should span a larger part of the day. It is striking, however, how large the range is for users with many sessions. This might be a consequence of the sample bias inherent in the Beta population. Most of our Test Pilot users are tech-savvy young men, so the wide range in which they browse is a little more understandable.

As you can tell there is a lot we can do with just analyzing sessions. We’ll be rolling out more simple insights like these soon – stick around.

One of the most significant changes to the user interface for Firefox 4 was the rolling of the old menu items into one button, something we have called on this blog the “combined menu.” We’ve addressed the impact of this change in other posts, such as this one.  We thought of one obvious follow-up: how often do people ‘bounce away’ from the combined menu before clicking something in it?

We define the bounce rate as this: given a user clicking the combined menu button, what is the proportion of instances where users do not click on one of the menu items? We built a small NumPy library to help us analyze the sequence of user-browser interactions.

44% of users clicked something other than a menu item after having clicked the Firefox button, while 56% clicked on a succeeding menu item. These percentages do not change even if we throw out the first few instances of every user clicking on the Firefox button (new Beta users are likely curious about the new menu layout, so try it out).

The bounce rate for this study has gone up since the last version (where about 38% of button clicks led to bounces).  One possible explanation points to the handful new items added to the combined menu since the last beta interface study.  The additional cognitive load might lead to more bounces.

We are currently working on calculating the bounce rate for the pre-combined menu UI.  Stay tuned for that.

Earlier this year, Test Pilot ran a Menu Item Usage Study to better understand how users interact with the traditional menu bar as Firefox transitions to a more streamlined, single “application button.” We’ve already shared some analysis from this study: we answered which menu items are the most and least commonly used, explored the distribution of menu interactions, and learned that most users actually use very few commands from the menu bar.

We also examined the sequence of menu bar actions – specifically looking for common sequences in commands. The strongest connection was between the ‘About Firefox’ and ‘Check for Updates’ menu items; after clicking ‘About Firefox,’ 9% of users click ‘Check for Updates’ within 20 seconds.

From this data, we reasonably assume that many users go to “About Firefox” to check whether they should update. Accordingly, three minor changes to these menu items should improve user experience.

1. Move ‘Check for Updates’ from the ‘Help’ header to the ‘Firefox’ header directly under ‘About Firefox’ so these related commands are logically grouped together (on Mac OSX).
2. Add a ‘Check for Updates’ button directly to the ‘About Firefox’ window.
3. If automatic updates are set, initialize a check for updates when ‘About Firefox’ opens, and give the user the option to update directly from the window (if any updates are available).

Luckily, we have some excellent engineers working on Firefox who came to the same conclusions.  Margaret Leibovic, one of these awesome engineers, recently blogged about her first month’s work here at Mozilla -  this work included implementing the first 2 of these changes.  Check out the new ‘About Firefox’ window below:

The relocation of ‘Check for Updates’ and this refreshed window have already landed in Minefield, and will be integrated into Firefox 4 from beta 7. Robert Strong, another Firefox Engineer, is working on a patch to implement the third improvement and initialize a check for update automatically when ‘About Firefox’ opens – look for this addition in the next beta!

Note: Google Chrome already does something similar with its about window.  While these data driven recommendations were reached independently,  Chrome should be given credit for identifying this connection early-on.