Blog of Metrics » results

Comparing the Bias in Telemetry Data vs The Typical Firefox User

Saptarshi Guha — Tue, 13 Dec 2011 19:25:49 +0000

Telemetry is a feature in Firefox that captures performance metrics such as start up time, DNS latency among others. The number of metrics captured is in the order of a couple hundred. The data is sent back to the Mozilla Bagheera servers which is then analyzed by the engineers.

The Telemetry feature asks the Nightly/Aurora (pre-release) users if they would like to submit their anonymized performance data . This resulted in a response rate (number of people who opted in divided by the number of people who were asked) of less than 3%. This led to two concerns: small number of responses (which changed when Telemetry became part of Firefox release) and more importantly representativeness: are the performance measurements as collected from the 3% representative of those of people who chose not to opt in?

Measuring the bias is not easy unless we have measurements about the users who did not opt in. Firefox sends the following pieces of information to the Mozilla servers: operating system, Firefox version, extension identifiers and the time for the session to be restored. This is sent by all Firefox installations unless the distribution or user have the feature turned off (this is called the services AMO ping). The Telemetry data contains the same pieces of information.

What this implies is that we have start up times for i) the users who opted in to Telemetry and ii) everyone. We can now answer the question “Are the startup times for the people who opted into Telemetry representative of the typical Firefox user?”

Note: ‘everyone’ is almost everyone. Very few have this feature turned off.

Data Collection

We collected start up times for Firefox 7,8 and 9 for November, 2011 from the log files of services.addons.mozilla.org (SAMO). We also took the same information for the same period from the Telemetry data contained in HBase ( some code examples can be found at the end of the article).

Objective

Are start up times different by Firefox version and/or Source, where source can be SAMO or Telemetry.

Displays

Figure 1 is boxplot of log of start up time for Telemetry (tele) vs. SAMO (samo) by Firefox version. At first glance it appears the start up times from Telemetry are less than those of SAMO. But the length of the bars makes it difficult to stand by this conclusion.

Figure 1: Boxplot of Log SessionRestored for Telemetry/SAMO by FF Version

Figure 2 is the difference in the deciles of log of start up time. In other words, approximately speaking, the deciles of ratio of Telemetry start up time to SAMO start up time. The medians hover in the 0.8 region, though the bars are very wide and do not support to a the quick conclusion that Telemetry start up time is smaller.

Figure 2: Difference of Deciles of Logs

In Figure 3, we have the mean of medians of 1000 samples: red circles are for telemetry and black for SAMO. The ends of the line segments correspond the sample 95% confidence interval (based on the sample of sample medians). The CI for the SAMO data lies entirely within that of the Telemetry data. This makes one believe that the two groups are not different.

Figure 4: Mean of the medians (circles) with their 95% confidence intervals. Red isTelemetry, Black is SAMO

Analysis of Variance

For a more numerical approach, we can estimate the analayis of variance components. The model is

log(startup time) ~ version + src

(we ignore interaction). Since the data is in the order of billions of rows, I instead take 1000 samples of approximately 20,000 (sampling rate of 0.001%) rows each. Compute ANOVA results of each and then average the summary tables of the lm function in R. In other words we make our conclusions based on the average of the 1000 samples of ~20,000 rows each. ( I should point out that the residuals (as per a quick visual check) were roughly distributed as gaussian and other diagnostics came out clean)

The average ANOVA indicates does not support version effect or source effect (at the 1% level). In other words, the log of start up time is not affected by the version nor is it affected by the source (Telemetry/ SAMO).

               Estimate Std. Error     t value   Pr(>|t|)
(Intercept)  8.62635472 0.01171420 736.4390937 0.00000000
vers8       -0.05995627 0.01928947  -3.1089666 0.02922402
vers9       -0.03382135 0.10466330  -0.3247165 0.48286903
vers10      -0.03862282 0.29308642  -0.1418623 0.48228122
srctele     -0.02290538 0.03946150  -0.5811779 0.45300964

This is good news! Insofar start up time is concerned, Telemetry is representative of SAMO.

A Different Approach and Some Checks

By now, the reader should note that we have answered our question (see last line of previous section). Two questions remain:

1. The samples are representative. We are sampling on 3 dimensions: startup time, src and version. Consider the 1000 quantiles of startup time, the 2 levels of src and 4 levels of version. All in all, we have 1000x2x4 or 8000 cells. Sampling from the population might result in several empty cells, so much so, that the joint distribution of the sample might be very different from that of the population. To confirm that our cell distribution of the samples reflect the cell distribution of the population, we computed Chi Square tests comparing the sample cell counts with that of the parent. All 1000 samples passed!

2. Why use samples? We can do a log linear regression testing on the 8000 cell counts (i.e all the 1.9 BN data points) . This of course loses a lot of power: we are binning the data and all monotonic transformations are equivalent. The model equivalent (using R’s formula language) of the ANOVA described above is

log(cell count) ~ src+ver+binned_startup:(src+ver)

If the effects of binned_startup:src and binned_startup:ver are not significant this corresponds to our conclusion in the previous section. And nicely enough, it does! Output of summary(aov(glm(…))) is

summary(aov(glmout <- glm(n~ver+src+sesscut:(ver+src)
                          , family=poisson
                          , data=cells3.parent))

              Df     Sum Sq    Mean Sq   F value Pr(>F)
ver            3 4.6465e+14 1.5488e+14 1131.8666 <2e-16 ***
src            1 3.2705e+14 3.2705e+14 2390.0704 <2e-16 ***
ver:sesscut 3952 5.4969e+13 1.3909e+10    0.1016      1
src:sesscut  988 2.0009e+13 2.0252e+10    0.1480      1
Residuals   2967 4.0600e+14 1.3684e+11

Some R Code and Data Sizes:

1. The data for SAMO was obtained from Hive, sent to a text file and then imported to blocked R data frames using RHIPE. All subsequent analysis was done using RHIPE.

2. The data for Telemetry, was obtained from Hbase using Pig (RHIPE can read HBase, but I couldn’t install it on this particular cluster). The text data was then imported as blocked R data frames and placed in the same directory as the
imported SAMO data.

3. Data sizes were in the few hundreds of gigabytes. All computations were done using RHIPE (R not on the on the nodes) on a 350TB/33 node Hadoop cluster.

3. I include some sample code to give a flavor of RHIPE.

Importing text data as Data Frames

map         <- expression({
  ln        <- strsplit(unlist(map.values),"\001")
  a         <- do.call("rbind",ln)
  addonping <- data.frame(ds=a[,1]
                         ,vers=a[,3]
                         ,sesssionrestored=as.numeric(a[,6])
                         ,src=rep("samo",length(a[,6]))
                         ,stringsAsFactors=FALSE)
  rhcollect(runif(1),addonping)
})
z <- rhmr(map=map
          ,ifolder="/user/sguha/somequants"
          ,ofolder="/user/sguha/teledf/samo"
          ,zips="/user/sguha/Rfolder.tar.gz"
          ,inout=c("text","seq")
          ,mapred=list(mapred.reduce.tasks=120
             ,rhipe_map_buff_size=5000))
rhstatus(rhex(z,async=TRUE),mon.sec=4)

Creating Random Samples

map         <- expression({
  y         <- do.call('rbind', map.values)
  p         <- 20000/1923725302
  for(i in 1:1000){
    zz      <- runif(nrow(y)) < p
    mu      <- y[zz,,drop=FALSE]
    if(nrow(mu)>0)
      rhcollect(i,mu)
  }
})
reduce      <- expression(
    pre={ x <- NULL}
    ,reduce = {
      x     <- rbind(x,do.call('rbind',reduce.values))
    }
    ,post={ rhcollect(reduce.key,x) }
    )
z <- rhmr(map=map,reduce=reduce
          ,ifolder="/user/sguha/teledfsubs/p*"
          ,ofolder="/user/sguha/televers/dfsample"
          ,inout=c('seq','seq')
          ,orderby='integer'
          ,partition=list(lims=1,type='integer')
          ,zips="/user/sguha/Rfolder.tar.gz"
          ,mapred=list(mapred.reduce.tasks=72
             ,rhipe_map_buff_size=20))
rhstatus(rhex(z,async=TRUE),mon.sec=5)

Run Models Across Samples

map        <- expression({
  cuts     <- unserialize(charToRaw(Sys.getenv("mcuts")))
  lapply(map.values, function(y){
    y$tval <- sapply(y$sesssionrestored
                     ,function(r) {
                       if(is.na(r)) return( r)
                       max(min(r,cuts[2]),cuts[1])
                     })
    mdl    <- lm(log(tval)~vers+src,data=y)
    rhcollect(NULL, summary(mdl))
  })})
z <- rhmr(map=map
          ,ifolder="/user/sguha/televers/dfsample/p*"
          ,ofolder="/user/sguha/televers2",
          ,zips="/user/sguha/Rfolder.tar.gz"
          ,inout=c("seq","seq")
          ,mapred=list(mapred.reduce.tasks=0))
rhstatus(rhex(z,async=TRUE),mon.sec=4)

Computing Cell Counts For A Log Linear Model

cuts2                <- wtd.quantile(tms$x,tms$n,
                                     p=seq(0,1,length=1000))
cuts2[1]             <- cuts[1]
cuts2[length(cuts2)] <- cuts[2]
map.count <- expression({
  cuts       <- unserialize(charToRaw(Sys.getenv("mcuts")))
  z          <- do.call(rbind,map.values)
  z$tval     <- sapply(z$sesssionrestored,function(r)
                  max(min(r,cuts[length(cuts)]),cuts[1]))
  z$sessCuts <-
    factor(findInterval(z$tval,
                        cuts),ordered=TRUE)
  f          <- split(z,list(z$vers,z$sessCuts,z$src),drop=FALSE)
  for(i in seq_along(f)){
    y <-strsplit(names(f)[[i]],"\\.")[[1]]
    rhcollect(y,nrow(f[[i]])) }
})
z <-
  rhmr(map=map.count,reduce=rhoptions()$templates$scalarsummer
       ,combiner=TRUE,
       ifolder="/user/sguha/teledfsubs/p*"
       ,ofolder="/user/sguha/telecells",
       ,zips="/user/sguha/Rfolder.tar.gz"
       ,inout=c("seq","seq") ,mapred=
       list(mapred.task.timeout=0
            ,rhipe_map_buff_size=40
            ,mcuts=rawToChar(serialize(cuts2, NULL,
                                ascii=TRUE))))

Do 90% of People Not Use CTRL+F?

Diyang Tang — Thu, 25 Aug 2011 18:31:06 +0000

According to an article in The Atlantic floating around the internet, 90% of users don’t know how to use CTRL+F or Command+F to search a webpage. We were surprised at that percentage. Fortunately, Mozilla has TestPilot studies with open data, and we can see if Firefox users behave similarly. One relevant 7-day TestPilot study of about 69,000 Windows users focused on Firefox’s user interface. Along with seeing how users interacted with the navigation bar, their bookmarks, etc., the study looked at how often people used keyboard shortcuts.

What we found is that about 81% of TestPilot users didn’t use CTRL+F during the course of the study. While 81% is lower than the 90% in the article, TestPilot users are usually more technologically experienced than the general population, since they are largely Firefox Beta users. When we look at TestPilot users who consider themselves beginners, the percentage goes up to 85%. Therefore, our 81% figure does not belie the Atlantic piece.

In addition, those who use CTRL+F on average use keyboard shortcuts twice as much as those who don’t, even when we ignore those people who don’t use any keyboard shortcuts at all. This implies that people who use CTRL+F are more comfortable with keyboard shortcuts in general. The only keyboard shortcut the users who use CTRL+F lag behind in is Full Screen, or F11.

Feel free to take a look at the data yourself and let us know about any interesting trends you discover!

Text mining users’ definitions of browsing privacy

Rebecca Weiss — Mon, 15 Aug 2011 18:32:11 +0000

One issue that’s been on everyone’s mind lately is privacy. Privacy is extremely important to us at Mozilla, but it isn’t exactly clear how Firefox users define privacy. For example, what do Firefox users consider to be essential privacy issues? What features of a browsing experience lead users to consider a browser to be more or less private?

In order to answer these questions, we asked users to give us their definitions of privacy, specifically privacy while browsing, in order to answer these questions. The assumption was that users will have different definitions, but that there will be enough similarities between groups of responses that we could identify “themes” amongst the responses. By text mining user responses to an open-ended survey question asking for definitions of browsing privacy, we were able to identify themes directly from the users’ mouths:

Regarding privacy issues, people know that tracking and browser history are different issues, validating the need for browser features that address these issues independently (“private browsing” and “do not track”)
People’s definition of personal information vary, but we can group people according to the different ways they refer to personal information (this leads to a natural follow-up question; what makes some information more personal than others?)
Previous focus group research, contracted by Mozilla, showed that users are aware that spam indicates a security risk, but what didn’t come out of the focus group research was that users also also consider spam to be an invasion of their privacy (a follow-up question, what do users define as “spam?” Do they consider targeted ads to be spam?)
There are users who don’t distinguish privacy and security from each other

Some previous research on browsing and privacy

We knew from our own focus group research that users are concerned about viruses, theft of their personal information and passwords, that a website might misuse their information, that someone may track their online “footprint”, or that their browser history is visible to others. Users view things like targeted ads, spam, browser crashes, popups, and windows imploring them to install updates as security risks.

But it’s difficult to broadly generalize findings from focus groups. One group may or may not have the same concerns as the general population. The quality of the discussion moderator, or some unique combination of participants, the moderator, and/or the setting can also influence the findings you get from focus groups.

One way of validating the representativeness of focus group research is to use surveys. But while surveys may increase the representativeness of your findings, they are not as flexible as focus groups. You have to give survey respondents their answer options up front. Therefore, by providing the options that a respondent can endorse, you are limiting their voice.

A typical way to approach this problem in surveys is to use open-ended survey questions. In the pre-data mining days, we would have to manually code each of these survey responses: a first pass of all responses to get an idea of respondent “themes” or “topics” and a second pass to code each response according to those themes. This approach is costly in terms of time and effort, plus it also suffers from the problem of reproducibility; unless themes are extremely obvious, different coders might not classify a response as part of the same theme. But with modern text mining methods, we can simulate this coding process much more quickly and reproducibly.

Text mining open-ended survey questions

Because text mining is growing in popularity primarily due to its computational feasibility , it’s important to review the methods in some detail. Text mining, as with any machine learning-based approach, isn’t magic. There are a number of caveats to make about the text mining approach used. First, the clustering algorithm I chose to use requires an arbitrary and a priori decision regarding the number of clusters. I looked at 4 to 8 clusters and decided that 6 provided the best trade-off between themes expressed and redundancy. Second, there is a random component to clustering, meaning that one clustering of the same set of data may not produce the exact same results as another clustering. Theoretically, there shouldn’t be tremendous differences between the themes expressed in one clustering over another, but it’s important to keep these details in mind.

The general idea of text mining is to assume that you can represent documents as “bags of words”, that bags of words can be represented or coded quantitatively, and that the quantitative representation of text can be projected into a multi-dimensional space. For example, I can represent survey respondents in two dimensions, where each point is a respondent’s answer. Points that are tightly clustered together mean that these responses are theoretically very similar with respect to lexical content (e.g., commonality of words).

I also calculated a score that identifies the relative frequency of each word in a cluster, which is reflected in the size of the word on each cluster’s graph. In essence, the larger the word, the more it “defines” the cluster (i.e. its location and shape in the space).

Higher resolution .pdf files of these graphs can be found here and here.

Cluster summaries

“Privacy and Personal information”: Clusters 1, 4, and 5 are dominated by, unsurprisingly, concerns about information. What’s interesting are the lower-level associations between the clusters and the words. The largest, densest cluster (cluster 4) deals mostly with access to personal information whereas cluster 1 addresses personal information as it relates to identity issues (such as when banking). Cluster 5 is subtly different from both 1 and 4. The extra emphasis on “share” could imply that users have different expectations of privacy with personal information that they explicitly choose to leak onto the web as opposed to personal information that they aren’t aware they are expressing. One area of further investigation would be to seek out user definitions on personal information; what makes some information more “personal” than others?

“Privacy and Tracking”: Cluster 6 clearly shows that people associate being tracked as a privacy issue. The lower-scored words indicate what kind of tracked information concerns them (e.g., keystrokes, cookies, site visits), but in general the notion of “tracking” is paramount to respondents in this cluster. Compare this with cluster 2, which is more strongly defined by the words “look” and “history.” This is obviously a reference to the role that browsing history has in defining privacy. It’s interesting that these clusters are so distinct from each other, because it implies that users are aware there is a difference between their browser history and other behaviors they exhibit that could be tracked. It’s also interesting that users who consider browser history a privacy issue also consider advertising and ads (presumably a reference to targeted ads) as privacy issues as well. We can use this information to extend the focus group research on targeted ads; in addition to a security risk, some users also view targeted ads as an invasion of privacy. One interesting question naturally arises: do users differentiate between spam and targeted advertisements?

“Privacy and Security”: The weakest defined group is cluster 3, which can be interpreted in many ways. The least controversial inference could be that these users simply don’t have a strong definition of privacy aside from a notion that privacy is related to identity and security. This validates a notion from our focus group research that some users really don’t differentiate between privacy and security.

Final thoughts

User privacy and browser security are very important to us at Mozilla, and developing a product that improves on both requires a deep and evolving understanding of what those words mean to people of all communities - our entire user population. In this post, we’ve shown how text mining can enhance our understanding of pre-existing focus group research and generate novel directions for further research. Moreover, we’ve also shown how it can provide insight into users’ perception by looking at the differences in the language they use to define a concept. In the next post, I’ll be using the same text mining approach to evaluate user definitions of security while browsing the web.

Why People Don’t Upgrade Their Browser – Part IV

Christopher Jung — Wed, 21 Apr 2010 19:14:35 +0000

Web technology moves quickly, and we at Mozilla do our best to balance our values of keeping users safe and secure with also ensuring that we are giving users the ability to make choices about the software on their computers. We can’t keep up with the cost of providing security updates to older versions forever, though, so we’ll often try to encourage users to migrate to the latest version. A few weeks ago, we made another such offer to our Firefox 3 users, explaining that we were not planning on supporting that release anymore and asking them to upgrade to Firefox 3.6:

In the past, soliciting feedback from users has helped us understand why users were opting to not upgrade, so this time around, people who clicked on the “No thanks” option above were directed to a survey. Over 40,000 people were kind enough to share their thoughts with us.

So what do the results look like this time?

Let’s start by taking a look at responses to the first question:

Over half of the respondents stated that they were simply content with Firefox 3. And compared to what we saw last survey, far fewer users selected “Other reason.” Still, a quarter of users took the time to give an alternative explanation, so exploring these answers can definitely help us gain additional insight. A list of the most common phrases entered into the “Other reason” text field is presented below:

Two things immediately jump out:

First, confusion over cost has virtually disappeared — this had been a problem previously.
Second, the vast majority of comments are now about a lack of time.

The lack of comments about costs is a pleasant surprise. Acting on feedback from the last survey, we revised the update prompt, highlighting that the upgrade is free in two places. It seems that this simple addition was enough to eliminate most of the confusion.

The update prompt itself may also partly account for the rise in “no time” comments. With prior upgrade initiatives, users saw an advertisement and button similar to what you see below:

Users could then click “Get the new version” to initiate a 30-second update procedure. This time, however, instead of seeing a typical software dialog box, people encountered the update prompt via the Firefox “Whats new” page and were presented with the customary green Firefox download button. Hitting the button also initiated the update, but via the more involved process of downloading and installing a fresh version of the browser.

Pushing out the update in this way was a one-time situation, and we’ll revert to the usual process and software dialog box in the future. To alleviate this issue further, we should also perhaps add to the prompt some indication that updating Firefox is relatively quick and painless.

Next, lets turn to Question 2. This was a free form text box so we manually read through a random sample of 10% of the responses and parsed them into categories.

Firefox 3.6’s compatibility, both with add-ons and with specific websites/applications, remains a key issue (although user perception might be playing a small role as well). On the other hand, general stability and performance issues are cited considerably less often: crash and speed comments together account for 11% of responses, down from the 25% we saw last survey.

Our two main insights from above are also evident here. Cost comments have essentially disappeared from Question 2 as well, although they previously made up 7% of responses. And again, time concerns have become a real sticking point for users.

The rise of these “no time” responses, coupled with the still considerable “too many updates” category, has triggered suggestions that extend beyond prompt and wording revisions. For example, some propose that updates occur automatically in the background unbeknownst to the user (à la chrome), perhaps with an easy option to downgrade (unlike chrome).

Clearly, we are a long ways from making any major changes to the update process, but we’d love to hear your thoughts on any of this!

Dramatic Stability Improvements in Firefox

Ken Kovash — Thu, 08 Apr 2010 16:32:43 +0000

Over the past five months, Firefox has seen a 40% improvement in stability. That’s incredible! While the Firefox development community continues to focus its efforts on stability and performance (i.e., 100% improvement is still being strived for), we’re proud to highlight the most recent numbers.

How did we calculate that 40% improvement? We took a look at the Crash Reports data, along with an estimated ratio of daily crash instances to active daily users of Firefox. With that data in hand, we first wanted to know how the number of daily crash instances has been trending by major version of Firefox, e.g., is Fx3.6 particularly more stable than previous releases and what does its trending look like?

The chart above looks amazing… and it brought us to the next question… rolling up major versions, what does the total universe of Firefox stability look like? Using the same numbers as seen above, we simply added them together to come up with the high-level metric – estimated ratio of daily crash instances to active daily users of Firefox. The trend of the blue line below represents a 40% reduction since early November. Spectacular!

So, how exactly did this happen?

As mentioned previously, improving Firefox stability was a top level objective of the Mozilla community during the last few months of 2009. That effort was led by a variety of folks across a wide breath of the community… people who saw the many complaints about crashes, collected the data, figured out what was causing the most pain, addressed those issues (sometimes in code, sometimes by working with partners) and saw product improvements.

And in addition to hitting a home run with Fx3.6 and reducing the crashiness of Fx3.5, a majority of Firefox users quickly migrated to Fx3.6… adding all that up drove what you see in the charts above.

What’s next?

There are still more improvements to come. For example, with a significant percentage of crashes in Flash, a lot of work around stability improvement for Firefox users is currently going into Flash Player 10.1. If you’re personally experiencing any crashiness issues when using Firefox, please make sure you’re on the latest version of Firefox and feel free to use tools like bugzilla and hendrix to notify the community.

Firefox 3.6 Upgrade Offer – An Early Success

Ken Kovash — Wed, 17 Mar 2010 23:04:18 +0000

Late last week, Mozilla pushed a Firefox 3.6 upgrade offer to people on older versions of Firefox. Here was the actual offer:

What has been the impact in just a few short days?

Looking at the chart below, you’ll notice that the percentage of all Firefox users on Firefox 3.6 increased dramatically in recent days. Late last week, the percentage of users on 3.6 stood at roughly 23%, and as of yesterday that number had climbed to 43% (btw, that 20% pick up translates to roughly 75 million total people who made the switch — that’s huge!). Past major update offers have generally converted in the ballpark of 10% to 20% of users, so this most recent push has definitely had a substantial impact.

And below is one further way to interpret these recent numbers. Looking at the launches of Firefox 3, Firefox 3.5, and Firefox 3.6, we considered how quickly each latest & greatest grabbed usage share from its predecessor. You’ll notice that the shapes of the 3.0 and 3.6 curves (blue and green) look almost identical… the only difference being the timing of the first offer/advertisement being pushed (note: there was an issue with our early 3.5 update offers, hence the red curve seems to be missing the same upward spike).

Why Do Firefox Downloads Spike on Release Days?

Ken Kovash — Fri, 19 Feb 2010 23:22:52 +0000

As Daniel pointed out, there has always been a dramatic increase in fresh downloads/installs of Firefox at the time of each minor version release – separate from people simply being updated. We’ve never entirely understood this user behavior until Daniel started some digging yesterday. Here’s what we know…

Yesterday (just after 3.0.18 and 3.5.8 were released) we saw a spike in fresh downloads/installs of Firefox. The typical daily number is in the ballpark of 2 Million and yesterday it shot up to over 4 Million.
As Daniel highlighted, nearly all of the download activity was for Firefox 3.6.
Digging a little deeper, we also discovered that the entire spike in 3.6 downloads was coming from people on Firefox 3.5.8. This means people successfully got the update yesterday (3.5.8), and then went out of their way to manually do one further update (i.e., get 3.6).

Why or how is this happening?

It turns out that the answer was right under our nose. When people get an update, they see an update page. And if they’re not on the current major version (e.g., 3.6), the page suggests that they go and download the lastest and greatest. So, what happened within this user interaction yesterday?

So, this explains a common experience for millions of users each time a Firefox update is shipped. And it’s good to see the messaging on that 3.5.8 update page (and all older update pages) is paying off. Perhaps we should consider changing the concept of those pages to be even more aggressive in getting people to update to the latest and greatest.

An Improved Experience for New Users of Firefox

Ken Kovash — Tue, 09 Feb 2010 22:50:15 +0000

Over the past year, we set out to identify and solve any possible pain points that might arise during a person’s experience downloading and installing Firefox (previous posts are here, here, here, here, and here). Thanks to feedback from users, and some resulting product changes, we can now safely say that there are no issues confronting new users when installing Firefox for the very first time.

How do we know this?

Last week, we re-ran our installer feedback mechanism for a short period of time. If a user clicked “cancel” while walking through the Firefox installer, they were asked if they wanted to provide feedback.

After making an initial round of product improvements based on our first time feedback (March ’09), here are the transformed feedback results from our more recent efforts (both July 2009 and last week):

While we still have plans to tackle the remaining big slice of the pie (see concluding paragraph), we were able to successfully solve the red and green pie slices from last time. In our latest feedback results (pie on the right), the big pie slice now represents nearly 100% of the total feedback (the previously seen categories virtually evaporated). One way to interpret this is that we’ve now successfully identified and resolved 3 of the top 4 issues originally encountered by users.

Here were the specific actions we took addressing those red and green slices (details are in bug 508684):

Don’t Want Firefox as Default

People indicating this issue were missing the selection option earlier in the installation process, arrived at the end, and mistakenly believed that we were making Firefox their default without being given a choice. So, we added the choice to the final step in the installer:

Confusion About Updating-Upgrading-Installing

We did a few different things to help address this area of confusion. First, we added content to mozilla.com and prominently displayed it on the main Firefox product pages seen by existing users:

Second, within the Firefox installer, we changed the Install button to say “Upgrade” instead of “Install”:

Thanks to Rob Strong, the Firefox team, the Funnelcake team, John Slater, and Laura Mesa, among others, for implementing the changes highlighted above.

Lastly, there remains one outstanding problem for installers of Firefox – “it tells me to close Fx, but it’s not open” (the big pie slice in the charts above). This issue affects people who already have Firefox and are attempting to reinstall it, and as we’ve noted previously, this cohort becomes fairly frustrated during the experience. Some fixes are starting to be contemplated (e.g., bugs 496207, 544356)… and I’ll make sure to talk more here once some progress is made.

Why People Don’t Upgrade Their Browser – Part III

Ken Kovash — Thu, 28 Jan 2010 18:36:36 +0000

Mozilla recently advertised a Firefox 3.5 upgrade to users of Firefox 3 (this is also referred to it as a “major update prompt”) in an effort to migrate people to the latest version of Firefox. As of earlier this month, about 32% of all Firefox users were still on a version of Fx3, and as a result of the mid-January push, that number is now down to about 22%.

As a side benefit to this initiative, we also took the opportunity to see what feedback people had, specifically asking users to tell us what was on their mind if they were choosing not to upgrade. We did this once before when upgrading users from Fx2 to Fx3, and the results were extremely impactful, so we wanted to continue this once again as part of our broader user outreach efforts.

For people interesting in leaving feedback, here is the survey they saw:

A little more than 5,000 people were kind enough to share their thoughts. In turning to the results, let’s start with question #1:

The most surprising insight above is that 53% of respondents selected the “Other reason” check box. That suggests that the proposed answers we listed were found somewhat unsatisfactory and that people had other ideas on their mind. Clearly, understanding “Other reason” and seeing what users said within question #2 should provide us with much more insight than the chart above.

Below is a list of the most common phrases people typed into the “Other reason” box. What’s most surprising here? The vast majority of comments are about cost, i.e., “is this upgrade free?”. Taking this insight and turning it into action, we’re planning to make clear that “Firefox is free” within future upgrade prompts/advertisements.

Next, let’s look at how people responded to question #2. It’s a free form text box, so we manually read through comments, sorting them into different categories:

One easy way to interpret this pie chart is to compare it with what we saw last time (when users were upgrading from Fx2 to Fx3). UI related comments have almost vanished. On the other hand, add-on and extension compatibility comments (Norton was far and away the #1 cited) and crash comments have both risen dramatically as their total share of the pie. For “not compatible with specific website”, Facebook and specific Google pages (e.g., calendar) seemed to be the most frequently mentioned.

Moving forward, it will be critical that we acknowledge and address the concerns faced by these users. We’ve been working hard in recent months to reduce the crashiness of Firefox, and some positive results are already evident. For addressing the add-on and toolbar compatibility issues, the Firefox Support team has been raising the visibility of the top extension issues cited by users (Norton, Roboform, etc.).

And how do we properly communicate all of this the next time we advertise a major upgrade? We currently highlight these three bullet points:

Twice as fast as Firefox 3.
Private browsing, tear-off tabs and more.
The most advanced Firefox yet.

The last two should probably be changed to “This upgrade is free” and “Improved stability, fewer crashes.” Your thoughts?

People in France and Australia Are Also Switching Browsers

Ken Kovash — Fri, 22 Jan 2010 04:02:11 +0000

After last week’s warning from the German government against the use of Internet Explorer, the governments of France and Australia followed suit earlier this week.

Similar to our analysis of the impact in Germany, we wanted to see what happened this week with IE users downloading Firefox in both France and Australia. There are a couple patterns you’ll notice with the pictures below:

The rate at which IE users downloaded Firefox roughly doubled in each country. That’s a huge increase!
As we mentioned last time, the shaded orange areas are meant to represent the incremental number of downloads each day that are above what we would have expected on those days (i.e., they are downloads that can be described as directly attributable to the government warnings).
The cumulative orange area for France equates to about 60,000 downloads and the cumulative area for Australia translates to about 35,000 downloads.

France:

Australia: